jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.07k stars 223 forks source link

feat(parquet):support RLE encoding #1481

Closed jun0315 closed 1 year ago

jun0315 commented 1 year ago

Parquet2 already supports RLE encoding, we need to add it to the can_encode Support for adding RLE encoding to the encode function

ritchie46 commented 1 year ago

Thanks.. does this work as is? Could we maybe add a test for this.

jun0315 commented 1 year ago

Now arrow2 isn't support rle encoding https://github.com/jorgecarleitao/arrow2/blob/b09e580f075293e9af2879dcc7f6b2d5c8fe520e/src/io/parquet/write/primitive/basic.rs#L94-L111

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 95.65% and project coverage change: -0.44 :warning:

Comparison is base (b09e580) 83.73% compared to head (2912565) 83.29%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1481 +/- ## ========================================== - Coverage 83.73% 83.29% -0.44% ========================================== Files 389 389 Lines 41811 41908 +97 ========================================== - Hits 35009 34909 -100 - Misses 6802 6999 +197 ``` | [Impacted Files](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | | |---|---|---| | [src/io/parquet/write/boolean/basic.rs](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvYm9vbGVhbi9iYXNpYy5ycw==) | `95.91% <95.00%> (-2.64%)` | :arrow_down: | | [src/io/parquet/write/mod.rs](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1481?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvbW9kLnJz) | `88.43% <100.00%> (+0.11%)` | :arrow_up: | ... and [23 files with indirect coverage changes](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1481/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.