search

Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
2.34k stars 164 forks source link

[FEAT] daft-connect range use python generator #3308

Closed andrewgazelka closed 2 days ago

codspeed-hq[bot] commented 3 days ago

CodSpeed Performance Report

Merging #3308 will improve performances by 59.54%

Comparing andrew/improve-range (4f1210c) with main (731a73e)

Summary

⚡ 2 improvements ✅ 15 untouched benchmarks

Benchmarks breakdown

Benchmark main andrew/improve-range Change
⚡ test_iter_rows_first_row[100 Small Files] 320.4 ms 200.9 ms +59.54%
⚡ test_show[100 Small Files] 23 ms 14.9 ms +54.2%
codecov[bot] commented 3 days ago

Codecov Report

Attention: Patch coverage is 60.37736% with 63 lines in your changes missing coverage. Please review.

Project coverage is 77.39%. Comparing base (7922d2d) to head (fb08980).

Files with missing lines Patch % Lines
daft/io/_range.py 0.00% 23 Missing :warning:
src/daft-connect/src/translation/logical_plan.rs 63.79% 21 Missing :warning:
src/daft-connect/src/op/execute.rs 50.00% 12 Missing :warning:
src/daft-connect/src/lib.rs 0.00% 6 Missing :warning:
src/daft-connect/src/op/execute/root.rs 97.91% 1 Missing :warning:
Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308/graphs/tree.svg?width=650&height=150&src=pr&token=J430QVFE89&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) ```diff @@ Coverage Diff @@ ## main #3308 +/- ## ========================================== + Coverage 77.35% 77.39% +0.03% ========================================== Files 678 677 -1 Lines 83119 83114 -5 ========================================== + Hits 64296 64323 +27 + Misses 18823 18791 -32 ``` | [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | | |---|---|---| | [src/daft-connect/src/translation/schema.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Ftranslation%2Fschema.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvdHJhbnNsYXRpb24vc2NoZW1hLnJz) | `0.00% <ø> (ø)` | | | [src/daft-connect/src/op/execute/root.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Fop%2Fexecute%2Froot.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvb3AvZXhlY3V0ZS9yb290LnJz) | `97.91% <97.91%> (ø)` | | | [src/daft-connect/src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvbGliLnJz) | `46.27% <0.00%> (-1.27%)` | :arrow_down: | | [src/daft-connect/src/op/execute.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Fop%2Fexecute.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvb3AvZXhlY3V0ZS5ycw==) | `80.95% <50.00%> (ø)` | | | [src/daft-connect/src/translation/logical\_plan.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?src=pr&el=tree&filepath=src%2Fdaft-connect%2Fsrc%2Ftranslation%2Flogical_plan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29ubmVjdC9zcmMvdHJhbnNsYXRpb24vbG9naWNhbF9wbGFuLnJz) | `63.79% <63.79%> (ø)` | | | [daft/io/\_range.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3308?src=pr&el=tree&filepath=daft%2Fio%2F_range.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9pby9fcmFuZ2UucHk=) | `0.00% <0.00%> (ø)` | |

🚨 Try these New Features:

andrewgazelka commented 2 days ago

This stack of pull requests is managed by Graphite. Learn more about stacking.