search

dask / dask-expr

BSD 3-Clause "New" or "Revised" License
79 stars 18 forks source link

Add cache argument to ``lower_once`` #1059

Closed rjzamora closed 1 month ago

rjzamora commented 1 month ago

Adds simple logic to lower_once so that we avoid re-lowering an expression (and its children) multiple times.

Addresses the Expr.dask/compute component of https://github.com/dask/dask-expr/issues/1055: ddf.dask is ~200 times faster for python-only reproducer.

rjzamora commented 1 month ago

Any objections to this kind of change @fjetter ? We are already doing the same thing in simplify_once.

charlesbluca commented 1 month ago

Opened https://github.com/dask-contrib/dask-sql/pull/1332 to point to this patch to test if this unblocks the hanging tests