Fuse more aggressively if parquet files are tiny - Githubissues

dask / dask-expr

BSD 3-Clause "New" or "Revised" License

83 stars 22 forks source link

Fuse more aggressively if parquet files are tiny #1029

Closed phofl closed 5 months ago

phofl commented 5 months ago

This PR does a few things:

Fuse tiny parquet files more aggressive up to 75MB per partition (in memory)
set split_out in groupby also for length one grouping keys (this is more conservative and helps uf get rid of a few split_out=True optimisations in benchmarks)
make isin work with npartitions=1 Dask Series objects

Here is an AB test for this (fuse tag)

https://github.com/coiled/benchmarks/actions/runs/8709180542