search

cubed-dev / cubed

Bounded-memory serverless distributed N-dimensional array processing
https://cubed-dev.github.io/cubed/
Apache License 2.0
118 stars 14 forks source link

Query optimization #333

Open tomwhite opened 10 months ago

tomwhite commented 10 months ago

There are a couple of possibilities for doing query optimization that have come up recently.

Dask-expr will support arrays soon (https://github.com/dask-contrib/dask-expr/issues/446). It would be interesting to see if the expression system can be used in Cubed, and if there are any changes we'd need to contribute back.

egglog "is a Python package that provides bindings to the Rust library egglog, allowing you to use e-graphs in Python for optimization". Interestingly, it has a prototype of the Array API, which might make it a good candidate for providing query optimization for Cubed. This tutorial has an example of using the Array API implementation to optimize a scikit-learn function. (@saulshanabrook told us about egglog at yesterday's Pangeo Distributed Computing Working Group.)

dcherian commented 9 months ago

Ah wish I could've made the discussion. egglog looks cool!

saulshanabrook commented 9 months ago

I would be happy to do another call to get into more specifics of trying to implement the types of rewrites cubed needs with egglog. I did something similar a few weeks ago around the PyTensor project, using a concrete example to drive some exploration of how it could be implemented: https://egglog-python.readthedocs.io/latest/explanation/2023_11_17_pytensor.html

EDIT: Both the python bindings and the upstream rust library are in active development, so it's definitely useful to see if anything can be improved to support this kind of use case.

tomwhite commented 9 months ago

@saulshanabrook that would be awesome - thanks for the offer!