LibertyDSNP / parquetjs

Fully asynchronous, pure JavaScript implementation of the Parquet file format with additional features
MIT License
43 stars 24 forks source link

Ensure Buffer objects are returned by compression functions #88

Closed JasonYeMSFT closed 1 year ago

JasonYeMSFT commented 1 year ago

Problem

compression functions' return values are assumed to be Buffer objects. However, in reality they could be non-Buffer objects and will results in exceptions during reading/writing. https://github.com/LibertyDSNP/parquetjs/issues/72

Solution

Detect the data type of the value from various compression libraries and convert the value to Buffer object if necessary before returning.

Change summary:

Ensure Buffer objects are returned by compression functions

Steps to Verify:

As pointed out in the original issue, the problem can be reproduced using the browser build of parquetjs to read a file with snappy compression.

wilwade commented 1 year ago

@JasonYeMSFT Merged!

I'll cut a release this week or next (might try to get in a few of the other decimal issues first).

You can use the pre-release 0.0.0-fa1865 if you need it sooner. Or if it is really important I can cut a release faster if you need it.