search

Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
2.34k stars 164 forks source link

[FEAT] Support for aggregation expressions that use multiple AggExprs #3296

Closed kevinzwang closed 1 week ago

kevinzwang commented 1 week ago

This enables expressions such as sum("a") + sum("b") or mean("a") / 100 in aggregations. This PR enables Q8 and Q14 of TPC-H and is also necessary for Q17 and Q20 (which are also missing subquery).

codspeed-hq[bot] commented 1 week ago

CodSpeed Performance Report

Merging #3296 will degrade performances by 22.71%

Comparing kevin/multi-aggs (da49fd0) with main (25c3b26)

Summary

❌ 2 regressions ✅ 15 untouched benchmarks

:warning: Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main kevin/multi-aggs Change
❌ test_count[1 Small File] 3.6 ms 4.2 ms -14.78%
❌ test_iter_rows_first_row[100 Small Files] 253.4 ms 327.9 ms -22.71%
codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 92.11823% with 32 lines in your changes missing coverage. Please review.

Project coverage is 77.57%. Comparing base (711e862) to head (da49fd0). Report is 6 commits behind head on main.

Files with missing lines Patch % Lines
...ft-physical-plan/src/physical_planner/translate.rs 68.08% 15 Missing :warning:
src/daft-dsl/src/resolve_expr/mod.rs 83.33% 11 Missing :warning:
src/daft-logical-plan/src/ops/pivot.rs 80.95% 4 Missing :warning:
src/daft-local-execution/src/pipeline.rs 87.50% 2 Missing :warning:
Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296/graphs/tree.svg?width=650&height=150&src=pr&token=J430QVFE89&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) ```diff @@ Coverage Diff @@ ## main #3296 +/- ## ========================================== + Coverage 77.50% 77.57% +0.07% ========================================== Files 666 667 +1 Lines 81335 81685 +350 ========================================== + Hits 63041 63371 +330 - Misses 18294 18314 +20 ``` | [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | | |---|---|---| | [src/daft-dsl/src/lib.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-dsl%2Fsrc%2Flib.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtZHNsL3NyYy9saWIucnM=) | `100.00% <ø> (ø)` | | | [src/daft-local-plan/src/plan.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-local-plan%2Fsrc%2Fplan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9jYWwtcGxhbi9zcmMvcGxhbi5ycw==) | `96.38% <100.00%> (ø)` | | | [src/daft-logical-plan/src/logical\_plan.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Flogical_plan.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9sb2dpY2FsX3BsYW4ucnM=) | `62.72% <100.00%> (ø)` | | | [...rc/daft-logical-plan/src/ops/actor\_pool\_project.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Factor_pool_project.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvYWN0b3JfcG9vbF9wcm9qZWN0LnJz) | `41.89% <100.00%> (+1.61%)` | :arrow_up: | | [src/daft-logical-plan/src/ops/agg.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Fagg.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvYWdnLnJz) | `58.13% <100.00%> (+5.50%)` | :arrow_up: | | [src/daft-logical-plan/src/ops/explode.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Fexplode.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvZXhwbG9kZS5ycw==) | `80.85% <100.00%> (+1.30%)` | :arrow_up: | | [src/daft-logical-plan/src/ops/filter.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Ffilter.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvZmlsdGVyLnJz) | `64.28% <100.00%> (+5.95%)` | :arrow_up: | | [src/daft-logical-plan/src/ops/join.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Fjoin.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvam9pbi5ycw==) | `90.23% <100.00%> (+0.18%)` | :arrow_up: | | [src/daft-logical-plan/src/ops/project.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Fproject.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvcHJvamVjdC5ycw==) | `64.54% <100.00%> (+0.16%)` | :arrow_up: | | [src/daft-logical-plan/src/ops/repartition.rs](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree&filepath=src%2Fdaft-logical-plan%2Fsrc%2Fops%2Frepartition.rs&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-c3JjL2RhZnQtbG9naWNhbC1wbGFuL3NyYy9vcHMvcmVwYXJ0aXRpb24ucnM=) | `66.66% <100.00%> (+1.28%)` | :arrow_up: | | ... and [12 more](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | | ... and [6 files with indirect coverage changes](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3296/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)