How to specify schema while reading file from S3 bucket

ZJONSSON / parquetjs

fully asynchronous, pure JavaScript implementation of the Parquet file format

MIT License

34 stars 61 forks source link

How to specify schema while reading file from S3 bucket #72

Open dkommineni opened 3 years ago

dkommineni commented 3 years ago

How to specify schema while reading file from S3 bucket? I have S3 object which contains one of the column as DATE, when I read without schema, I get the DATE as integer value?

{ VERSION_KEY: 2, VERSION_DESCRIPTION: 'XXX', EFFECTIVE_DATE: '43200000000000' }

I am using the method const reader = await parquet.ParquetReader.openS3(client, params);

nikolaip commented 2 years ago

@dkommineni ... Did you find a workaround for this? The number isn't a timestamp so unless I am missing something, there is no way to turn the number into a date.

nikolaip commented 2 years ago

Just in case someone else comes upon this .. my issue stemmed from the field being an int96 which is no longer officially part of the Parquet spec and so, not supported by this library which makes sense.