jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.07k stars 221 forks source link

Fix the inferred nullability when converting a nested parquet schema to arrow #1565

Closed jhorstmann closed 9 months ago

jhorstmann commented 10 months ago

This allows the parquet_read example to correctly read the nested data attached to issue #1556 and also makes several test assertions match the comments above.

jhorstmann commented 9 months ago

@ritchie46 or @sundy-li as the parquet expert, could you take a look at this PR? Or should I rather re-open this against pola-rs/nano-arrow?

codecov[bot] commented 9 months ago

Codecov Report

All modified lines are covered by tests :white_check_mark:

Comparison is base (fb7b5fe) 83.06% compared to head (780b987) 83.08%. Report is 8 commits behind head on main.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1565 +/- ## ========================================== + Coverage 83.06% 83.08% +0.02% ========================================== Files 391 391 Lines 42889 42929 +40 ========================================== + Hits 35626 35668 +42 + Misses 7263 7261 -2 ``` | [Files](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | | |---|---|---| | [src/io/parquet/read/schema/convert.rs](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1565?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9zY2hlbWEvY29udmVydC5ycw==) | `95.03% <100.00%> (+0.35%)` | :arrow_up: | ... and [7 files with indirect coverage changes](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1565/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

ritchie46 commented 9 months ago

Or should I rather re-open this against pola-rs/nano-arrow?

@jhorstmann could you please do so? That would get the patched in polars as well.