search

Eventual-Inc / Daft

Distributed data engine for Python/SQL designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
2.35k stars 166 forks source link

[CHORE] Remove the concept of runner configs #3276

Closed jaychia closed 1 week ago

jaychia commented 1 week ago

Removes the concept of RunnerConfigs.

Our Runners are now materialized in 2 places:

  1. When a user calls daft.context.set_runner_*, eagerly!
  2. When a user first interacts with a Daft API that calls get_context().get_or_create_runner() under the hood
codspeed-hq[bot] commented 1 week ago

CodSpeed Performance Report

Merging #3276 will degrade performances by 25.13%

Comparing jay/remove-runner-configs (2cc0a0f) with main (2c59675)

Summary

❌ 1 regressions ✅ 16 untouched benchmarks

:warning: Please fix the performance issues or acknowledge them on CodSpeed.

Benchmarks breakdown

Benchmark main jay/remove-runner-configs Change
❌ test_show[100 Small Files] 50.5 ms 67.5 ms -25.13%
codecov[bot] commented 1 week ago

Codecov Report

Attention: Patch coverage is 96.29630% with 1 line in your changes missing coverage. Please review.

Project coverage is 76.86%. Comparing base (2c59675) to head (2cc0a0f). Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
daft/context.py 96.15% 1 Missing :warning:
Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3276/graphs/tree.svg?width=650&height=150&src=pr&token=J430QVFE89&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3276?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) ```diff @@ Coverage Diff @@ ## main #3276 +/- ## ========================================== - Coverage 77.03% 76.86% -0.18% ========================================== Files 665 664 -1 Lines 82153 81961 -192 ========================================== - Hits 63288 62996 -292 - Misses 18865 18965 +100 ``` | [Files with missing lines](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3276?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | | |---|---|---| | [daft/runners/ray\_runner.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3276?src=pr&el=tree&filepath=daft%2Frunners%2Fray_runner.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9ydW5uZXJzL3JheV9ydW5uZXIucHk=) | `81.83% <100.00%> (+0.02%)` | :arrow_up: | | [daft/context.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3276?src=pr&el=tree&filepath=daft%2Fcontext.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9jb250ZXh0LnB5) | `90.85% <96.15%> (+17.25%)` | :arrow_up: | ... and [30 files with indirect coverage changes](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/3276/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)