Eventual-Inc / Daft

Distributed DataFrame for Python designed for the cloud, powered by Rust
https://getdaft.io
Apache License 2.0
1.79k stars 108 forks source link

[FEAT] Automatically use Ray Runner if Ray is initialized #2282

Open jaychia opened 2 weeks ago

jaychia commented 2 weeks ago
  1. Automatically switches Daft to use the Ray Runner if a user calls ray.init(...) before running any Daft querying
  2. Also switches behavior to try and deprecate the DAFT_RAY_ADDRESS environment variable so that we can centralize on the normal RAY_ADDRESS behavior

This PR ensures the following behavior:

Ray connection detection, on driver vs on worker:

image

Warning if set_runner_ray is called with an address after Ray is already initialized:

image
codecov[bot] commented 3 days ago

Codecov Report

Attention: Patch coverage is 69.23077% with 8 lines in your changes are missing coverage. Please review.

Please upload report for BASE (main@e836290). Learn more about missing BASE report.

:exclamation: Current head e7ab3d2 differs from pull request most recent head da4e0de

Please upload reports for the commit da4e0de to get more accurate results.

Additional details and impacted files [![Impacted file tree graph](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/2282/graphs/tree.svg?width=650&height=150&src=pr&token=J430QVFE89&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc)](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/2282?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) ```diff @@ Coverage Diff @@ ## main #2282 +/- ## ======================================= Coverage ? 79.12% ======================================= Files ? 471 Lines ? 54404 Branches ? 0 ======================================= Hits ? 43049 Misses ? 11355 Partials ? 0 ``` | [Files](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/2282?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc) | Coverage Δ | | |---|---|---| | [daft/runners/ray\_runner.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/2282?src=pr&el=tree&filepath=daft%2Frunners%2Fray_runner.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9ydW5uZXJzL3JheV9ydW5uZXIucHk=) | `90.35% <50.00%> (ø)` | | | [daft/context.py](https://app.codecov.io/gh/Eventual-Inc/Daft/pull/2282?src=pr&el=tree&filepath=daft%2Fcontext.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=Eventual-Inc#diff-ZGFmdC9jb250ZXh0LnB5) | `76.29% <70.83%> (ø)` | |