Fix RLE encode/decode - Githubissues

ironSource / parquetjs

fully asynchronous, pure JavaScript implementation of the Parquet file format

MIT License

346 stars 175 forks source link

Fix RLE encode/decode #57

Closed ZJONSSON closed 5 years ago

ZJONSSON commented 6 years ago

bitpacking should work for any length of data, not just multiple of 8 (last packed is padded if less than 8)
Improve runs estimation - only start a new run if we are at a mod 8 === 0, otherwise use bitpacking

Added a test to codec_rle.js that fails on master but works on this branch (encoding of more than 8 values)

This seems to fix some issues related to opening parquet files in parquet-mr and vice-versa

ZJONSSON commented 6 years ago

RLE tests verified by parquet-mr here https://github.com/ironSource/parquetjs/pull/56

aconanlai commented 5 years ago

@kessler bumping this, as this solves a problem for us where nested fields cannot be read by spark - from what i can see in the issues, other people have this issue as well and this PR fixes it