jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.06k stars 223 forks source link

Safe conversion between arrow-rs and arrow2 `Array`s #1446

Closed tustvold closed 1 year ago

tustvold commented 1 year ago

Relates to #1429

This provides safe conversion between arrow2 arrays and arrow-rs ArrayData

codecov[bot] commented 1 year ago

Codecov Report

Patch coverage: 98.73% and project coverage change: +0.45 :tada:

Comparison is base (73ed7c8) 83.45% compared to head (41bd956) 83.90%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1446 +/- ## ========================================== + Coverage 83.45% 83.90% +0.45% ========================================== Files 376 387 +11 Lines 41161 41554 +393 ========================================== + Hits 34350 34867 +517 + Misses 6811 6687 -124 ``` | [Impacted Files](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | | |---|---|---| | [src/array/binary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2JpbmFyeS9tb2QucnM=) | `91.82% <ø> (ø)` | | | [src/array/boolean/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2Jvb2xlYW4vbW9kLnJz) | `79.41% <ø> (ø)` | | | [src/array/dictionary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2RpY3Rpb25hcnkvbW9kLnJz) | `95.32% <ø> (ø)` | | | [src/array/fixed\_size\_binary/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2ZpeGVkX3NpemVfYmluYXJ5L21vZC5ycw==) | `84.24% <ø> (ø)` | | | [src/array/fixed\_size\_list/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2ZpeGVkX3NpemVfbGlzdC9tb2QucnM=) | `85.84% <ø> (ø)` | | | [src/array/list/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L2xpc3QvbW9kLnJz) | `91.74% <ø> (ø)` | | | [src/array/map/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L21hcC9tb2QucnM=) | `72.81% <ø> (+10.67%)` | :arrow_up: | | [src/array/primitive/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L3ByaW1pdGl2ZS9tb2QucnM=) | `80.55% <ø> (ø)` | | | [src/array/struct\_/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L3N0cnVjdF8vbW9kLnJz) | `67.44% <ø> (ø)` | | | [src/array/union/mod.rs](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2FycmF5L3VuaW9uL21vZC5ycw==) | `89.71% <ø> (+2.33%)` | :arrow_up: | | ... and [15 more](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | | ... and [7 files with indirect coverage changes](https://codecov.io/gh/jorgecarleitao/arrow2/pull/1446/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) Help us with your feedback. Take ten seconds to tell us [how you rate us](https://about.codecov.io/nps?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao). Have a feature suggestion? [Share it here.](https://app.codecov.io/gh/feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

alamb commented 1 year ago

https://github.com/jorgecarleitao/arrow2/actions/runs/4659165653/jobs/8245723577?pr=1446 is failing on master and I think unrelated to this PR

jorgecarleitao commented 1 year ago

Thank you @tustvold for this beautiful PR 🙇

sundy-li commented 1 year ago

@tustvold is arrow-rs37.0.0 ready to bump?

tustvold commented 1 year ago

@tustvold is arrow-rs37.0.0 ready to bump?

Yes, I can prepare a PR for this. It has some non-trivial breaking changes to schema representation