search

Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
2.34k stars 164 forks source link

[CHORE]: prepare for nulls first/last kernels #3301

Closed universalmind303 closed 2 days ago

universalmind303 commented 1 week ago

This pr adds nulls_first to all sort functions in preparation for implementing the nulls first/last kernels. This will be followed up by adding the actual nulls first/last implementations.

codspeed-hq[bot] commented 1 week ago

CodSpeed Performance Report

Merging #3301 will improve performances by 67.12%

Comparing universalmind303:nulls-first (017c480) with main (1b84250)

Summary

⚡ 1 improvements ✅ 16 untouched benchmarks

Benchmarks breakdown

Benchmark main universalmind303:nulls-first Change
⚡ test_iter_rows_first_row[100 Small Files] 312.1 ms 186.8 ms +67.12%
codecov[bot] commented 2 days ago

Codecov Report

Attention: Patch coverage is 73.01980% with 109 lines in your changes missing coverage. Please review.

Project coverage is 77.31%. Comparing base (0709691) to head (017c480). Report is 18 commits behind head on main.

Files with missing lines Patch % Lines
src/daft-core/src/array/ops/sort.rs 59.42% 28 Missing :warning:
daft/table/table.py 9.52% 19 Missing :warning:
src/daft-table/src/python.rs 0.00% 12 Missing :warning:
daft/table/micropartition.py 54.54% 10 Missing :warning:
src/daft-sql/src/planner.rs 77.27% 10 Missing :warning:
src/daft-physical-plan/src/ops/sort.rs 20.00% 8 Missing :warning:
src/daft-core/src/utils/mod.rs 64.70% 6 Missing :warning:
src/daft-table/src/ops/groups.rs 0.00% 5 Missing :warning:
daft/expressions/expressions.py 66.66% 2 Missing :warning:
daft/series.py 85.71% 2 Missing :warning:
... and 6 more
Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301/graphs/tree.svg?width=650&height=150&src=pr&token=J430QVFE89&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) ```diff @@ Coverage Diff @@ ## main #3301 +/- ## ========================================== - Coverage 77.59% 77.31% -0.29% ========================================== Files 668 678 +10 Lines 82211 83063 +852 ========================================== + Hits 63793 64218 +425 - Misses 18418 18845 +427 ``` | [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | | |---|---|---| | [daft/dataframe/dataframe.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=daft%2Fdataframe%2Fdataframe.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9kYXRhZnJhbWUvZGF0YWZyYW1lLnB5) | `86.65% <100.00%> (ø)` | | | [daft/execution/execution\_step.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=daft%2Fexecution%2Fexecution_step.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9leGVjdXRpb24vZXhlY3V0aW9uX3N0ZXAucHk=) | `89.47% <100.00%> (+0.03%)` | :arrow_up: | | [daft/execution/physical\_plan.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=daft%2Fexecution%2Fphysical_plan.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9leGVjdXRpb24vcGh5c2ljYWxfcGxhbi5weQ==) | `94.01% <ø> (ø)` | | | [daft/execution/rust\_physical\_plan\_shim.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=daft%2Fexecution%2Frust_physical_plan_shim.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9leGVjdXRpb24vcnVzdF9waHlzaWNhbF9wbGFuX3NoaW0ucHk=) | `95.60% <ø> (ø)` | | | [src/daft-core/src/python/series.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Fpython%2Fseries.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvcHl0aG9uL3Nlcmllcy5ycw==) | `94.70% <100.00%> (+0.02%)` | :arrow_up: | | [src/daft-core/src/series/array\_impl/data\_array.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Fseries%2Farray_impl%2Fdata_array.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvc2VyaWVzL2FycmF5X2ltcGwvZGF0YV9hcnJheS5ycw==) | `95.95% <100.00%> (ø)` | | | [...c/daft-core/src/series/array\_impl/logical\_array.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Fseries%2Farray_impl%2Flogical_array.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvc2VyaWVzL2FycmF5X2ltcGwvbG9naWNhbF9hcnJheS5ycw==) | `94.44% <100.00%> (ø)` | | | [src/daft-core/src/series/ops/list.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=src%2Fdaft-core%2Fsrc%2Fseries%2Fops%2Flist.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtY29yZS9zcmMvc2VyaWVzL29wcy9saXN0LnJz) | `64.49% <100.00%> (+1.05%)` | :arrow_up: | | [src/daft-local-execution/src/pipeline.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=src%2Fdaft-local-execution%2Fsrc%2Fpipeline.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9jYWwtZXhlY3V0aW9uL3NyYy9waXBlbGluZS5ycw==) | `95.43% <100.00%> (+0.01%)` | :arrow_up: | | [src/daft-local-execution/src/sinks/sort.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree&filepath=src%2Fdaft-local-execution%2Fsrc%2Fsinks%2Fsort.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9jYWwtZXhlY3V0aW9uL3NyYy9zaW5rcy9zb3J0LnJz) | `96.15% <100.00%> (+0.40%)` | :arrow_up: | | ... and [31 more](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | | ... and [28 files with indirect coverage changes](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3301/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)

🚨 Try these New Features: