jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.06k stars 222 forks source link

Improved support for date64 written by pyarrow to parquet #1402

Closed jorgecarleitao closed 1 year ago

jorgecarleitao commented 1 year ago

Writing date64 to parquet is implementation-defined. This PR adds support for the date64 written by pyarrow.

Also protects us in the future against variations that we do not support with an error.

Closes #1400

codecov[bot] commented 1 year ago

Codecov Report

Base: 83.63% // Head: 83.61% // Decreases project coverage by -0.03% :warning:

Coverage data is based on head (a83d6b0) compared to base (3a8da98). Patch coverage: 77.98% of modified lines in pull request are covered.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1402 +/- ## ========================================== - Coverage 83.63% 83.61% -0.03% ========================================== Files 373 373 Lines 40284 40289 +5 ========================================== - Hits 33692 33687 -5 - Misses 6592 6602 +10 ``` | [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | | |---|---|---| | [src/io/parquet/read/statistics/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9zdGF0aXN0aWNzL21vZC5ycw==) | `87.31% <14.28%> (-1.30%)` | :arrow_down: | | [src/io/parquet/read/deserialize/simple.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9kZXNlcmlhbGl6ZS9zaW1wbGUucnM=) | `83.03% <82.35%> (-2.07%)` | :arrow_down: | | [src/bitmap/utils/slice\_iterator.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2JpdG1hcC91dGlscy9zbGljZV9pdGVyYXRvci5ycw==) | `97.56% <0.00%> (-1.22%)` | :arrow_down: | | [src/offset.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL29mZnNldC5ycw==) | `86.11% <0.00%> (+0.34%)` | :arrow_up: | | [src/io/ipc/read/file.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL2lwYy9yZWFkL2ZpbGUucnM=) | `97.32% <0.00%> (+0.89%)` | :arrow_up: | | [src/array/binary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1402?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2JpbmFyeS9tb2QucnM=) | `94.08% <0.00%> (+1.47%)` | :arrow_up: | Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.