search

dask / dask-expr

BSD 3-Clause "New" or "Revised" License
83 stars 22 forks source link

Global optimization #582

Open mrocklin opened 9 months ago

mrocklin commented 9 months ago

We probably need to modify dask.base to call some dask-expr optimize function. We probably also want to call this on everything we're computing all at once.

mrocklin commented 9 months ago

(my apologies if this has been mentioned / done in some other way and I'm out of date)

phofl commented 9 months ago

This is covered by the scheduler integration basically

mrocklin commented 9 months ago

I'm curious to learn more. Is there a PR I should be aware of? How does this work with the local scheduler?

fjetter commented 9 months ago

No new PR other than https://github.com/dask-contrib/dask-expr/pull/294 where I added a Tuple as you suggested. I still have to update the dask/dask and dask/distributed PRs accordingly

fjetter commented 9 months ago

this likely requires to build this tuple in collections_to_dsk but I don't expect much more work for this