search

jorgecarleitao / arrow2

Transmute-free Rust library to work with the Arrow format
Apache License 2.0
1.07k stars 221 forks source link

Move parquet async functionality behind feature flag #1586

Closed MostlyAmiable closed 8 months ago

MostlyAmiable commented 8 months ago

Addresses #1296 with a new io_parquet_async feature flag.

Since this is a breaking change for anyone using the async functions through the io_parquet flag, let me know if you want me to bump the version or just add io_parquet_async as a dependency of io_parquet (are circular feature dependencies allowed?).

cargo check output:

$ cargo clean
$ cargo check -F io_parquet
   Compiling proc-macro2 v1.0.69
   Compiling unicode-ident v1.0.12
   Compiling libc v0.2.149
   Compiling semver v1.0.20
   Compiling version_check v0.9.4
   Compiling autocfg v1.1.0
    Checking cfg-if v1.0.0
   Compiling serde v1.0.189
    Checking zerocopy v0.7.15
    Checking array-init-cursor v0.2.0
    Checking fallible-streaming-iterator v0.1.9
    Checking once_cell v1.18.0
    Checking streaming-decompression v0.1.2
   Compiling ahash v0.8.6
    Checking planus v0.3.1
    Checking parquet-format-safe v0.2.4
   Compiling num-traits v0.2.17
   Compiling seq-macro v0.3.5
    Checking base64 v0.21.5
    Checking simdutf8 v0.1.4
    Checking foreign_vec v0.1.0
    Checking streaming-iterator v0.1.9
    Checking either v1.9.0
    Checking hash_hasher v2.0.3
   Compiling quote v1.0.33
    Checking parquet2 v0.17.2
    Checking dyn-clone v1.0.14
   Compiling rustc_version v0.4.0
    Checking ethnum v1.4.0
   Compiling syn v2.0.38
    Checking getrandom v0.2.10
   Compiling arrow2 v0.17.4 (/Users/blevin/dev/lib/arrow2)
    Checking hashbrown v0.14.2
    Checking chrono v0.4.31
   Compiling serde_derive v1.0.189
   Compiling bytemuck_derive v1.5.0
    Checking bytemuck v1.14.0
    Checking arrow-format v0.8.1
    Finished dev [unoptimized + debuginfo] target(s) in 7.99s
$ cargo check -F io_parquet_async
   Compiling futures-core v0.3.28
   Compiling futures-channel v0.3.28
   Compiling futures-task v0.3.28
    Checking pin-project-lite v0.2.13
    Checking futures-sink v0.3.28
   Compiling slab v0.4.9
   Compiling futures-util v0.3.28
   Compiling syn v2.0.38
    Checking pin-utils v0.1.0
    Checking futures-io v0.3.28
    Checking memchr v2.6.4
   Compiling async-trait v0.1.74
   Compiling arrow2 v0.17.4 (/Users/blevin/dev/lib/arrow2)
   Compiling futures-macro v0.3.28
   Compiling async-stream-impl v0.3.5
   Compiling serde_derive v1.0.189
   Compiling bytemuck_derive v1.5.0
    Checking async-stream v0.3.5
    Checking bytemuck v1.14.0
    Checking serde v1.0.189
    Checking futures-executor v0.3.28
    Checking futures v0.3.28
    Checking parquet-format-safe v0.2.4
    Checking arrow-format v0.8.1
    Checking parquet2 v0.17.2
    Finished dev [unoptimized + debuginfo] target(s) in 8.41s
codecov[bot] commented 8 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Comparison is base (b073454) 83.39% compared to head (7b82982) 83.40%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #1586 +/- ## ========================================== + Coverage 83.39% 83.40% +0.01% ========================================== Files 391 391 Lines 43008 43008 ========================================== + Hits 35866 35871 +5 + Misses 7142 7137 -5 ``` | [Files](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1586?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao) | Coverage Δ | | |---|---|---| | [src/io/parquet/read/mod.rs](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1586?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9tb2QucnM=) | `100.00% <ø> (ø)` | | | [src/io/parquet/read/row\_group.rs](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1586?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvcmVhZC9yb3dfZ3JvdXAucnM=) | `96.53% <ø> (ø)` | | | [src/io/parquet/write/mod.rs](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1586?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao#diff-c3JjL2lvL3BhcnF1ZXQvd3JpdGUvbW9kLnJz) | `90.09% <ø> (ø)` | | ... and [3 files with indirect coverage changes](https://app.codecov.io/gh/jorgecarleitao/arrow2/pull/1586/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Jorge+Leitao)

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

MostlyAmiable commented 8 months ago

Looks like the Miri check has been getting cancelled for the past month? 🤔

sundy-li commented 8 months ago

LGTM