Open ZJONSSON opened 6 years ago
Here is a failing branch: https://github.com/ZJONSSON/parquetjs/tree/parquet-mr-fail Problems with the RLE encoding
This PR has been rebased on https://github.com/ironSource/parquetjs/pull/57 to include fixes for RLE in dlevels and rlevels + more test added to verify that the results are correct as seen from parquet-mr
I seem to be running into this issue as well. Are there any outstanding items on this PR that I might be able to help with to get it merged in?
Do your problems go away when you use this branch? The only outstanding thing here is a code review afaik.
NPM install per this comment does the trick for me: https://github.com/ironSource/parquetjs/issues/29#issuecomment-385808572
Here is a very basic example of how we can use dockerized parquet-tools (from parquet-mr) to test on travis whether files created by parquetjs can be read by parquet-mr (and therefore spark etc)
The basic test succeeds but more advanced tests fail. I will add a failing branch that we can use as a guide for fixing any errors.