ZJONSSON / parquetjs

fully asynchronous, pure JavaScript implementation of the Parquet file format
MIT License
34 stars 61 forks source link

parquet.ParquetReader.openUrl gives error not valid parquet file #52

Open thomasroshin opened 3 years ago

thomasroshin commented 3 years ago

I am using parquetjs-lite to read a parquet file from a url and while testing I am encountering an issue. I have put the file on my local and verified access to the file - http://localhost:8000/stock_plans_v2.parquet. From my JS code when I call

parquet.ParquetReader.openUrl(request,'http://localhost:8000/stock_plans_v2.parquet'); It always fails with 'not valid parquet file'. But the same file has no issues when open as a local file:

reader = await parquet.ParquetReader.openFile('stock_plans_v2.parque'); Reading through some issues on the net, I thought that maybe the reason was because of schema and that writing the file from parquetjs-lite (rather than pyarrow earlier used to generate the parquet), I re-created the parquet from the parquetjs-lite associating the schema.

// declare a schema var schema = new parquet.ParquetSchema({ StockName: { type: 'UTF8' }, CountryofListing: { type: 'UTF8' }, Ticker: { type: 'UTF8' }, MarginRate: { type: 'INT64' }, GoShort: { type: 'BOOLEAN' }, LimitedRiskPremium: { type: 'DOUBLE' }, });

// create new ParquetWriter var writer = await parquet.ParquetWriter.openFile(schema, 'stock_plans_v3.parquet');

// append a few rows to the file await writer.appendRow({StockName: '3i Group Plc', CountryofListing: 'UK', Ticker: 'III.L', MarginRate: 25, GoShort: true, LimitedRiskPremium: 0.7}); : : //close writer await writer.close(); But reading this parquet also has the same issue. Works with parquet.ParquetReader.openFile but fails with parquet.ParquetReader.openUrl as 'not valid parquet file'.

Any pointers would be helpful.