search

fugue-project / fugue

A unified interface for distributed computing. Fugue executes SQL, Python, Pandas, and Polars code on Spark, Dask and Ray without any rewrites.
https://fugue-tutorials.readthedocs.io/
Apache License 2.0
1.92k stars 94 forks source link

Fix dependency: Python 3.11.9 breaks Dask #537

Open amotl opened 2 months ago

amotl commented 2 months ago

Dear @goodwanghan and @kvnkho,

first things first: Thanks a stack for conceiving and maintaining this excellent library.

A few days ago, we discovered a flaw with PyCaret and Dask, and finally tracked it back to an incompatibility with Python 3.11.9, which was released on April 2, 2024, and is being rolled out on GHA runners by GitHub starting Apr 8, 2024, when it started to become more visible to downstream users like us.

In order to fix the problem, it is advised to update to dask>=2024.4.1, which includes a relevant fix. Those patches bring in corresponding updates on behalf of downstream applications/frameworks, validating that approach after it tripped us.

In the same spirit, this patch intends to fix Fugue on new and fresh installations, otherwise tripping users into this subtle dependency flaw. [^1]

With kind regards, Andreas.

[^1]: Let us know if you absolutely need to retain compatibility with Python 3.8, which may not be taken for granted after applying this patch as is, so we may look into this detail once again.

amotl commented 2 months ago

Hi again,

this improvement made it into downstream PyCaret on behalf of https://github.com/pycaret/pycaret/commit/fefe8d5fbe3c. Thank you very much, @Yard1. It will be so sweet to also see it integrated here, to provide its value to a wider audience.

With kind regards, Andreas.

goodwanghan commented 2 months ago

@amotl my apology for the delay. And thank you so much for proposing the change.

We can't make the dask lower bound version to be the latest. Because a lot of users are not using the latest Dask. Plus, Dask (as well as Ray) tends to have very aggressive changes, which could break a lot of things.

That being said, I will incorporate your change into another PR I am working on. I will make sure Fugue + Dask works with the latest python version.

And the correspondent change will be released in Fugue 0.9.0 shortly

amotl commented 2 months ago

Dear Han,

thanks for your swift reply, and thanks for taking care. Please feel free to close this PR if it is not applicable. We are looking forward to your other updates. Good luck!

With kind regards, Andreas.

amotl commented 1 month ago

Dear Han,

has this issue been addressed with GH-536 already, and published per version 0.9.0?

If so, thanks a stack, and we may want to close this PR again?

With kind regards, Andreas.