(fix): use `to_delayed` and `from_delayed` to submit gram matrix jobs

scverse / rapids_singlecell

Rapids_singlecell: A GPU-accelerated tool for scRNA analysis. Offers seamless scverse compatibility for efficient single-cell data processing and analysis.

https://rapids-singlecell.readthedocs.io/

MIT License

140 stars 20 forks source link

(fix): use `to_delayed` and `from_delayed` to submit gram matrix jobs #210

Closed ilan-gold closed 2 months ago

ilan-gold commented 3 months ago

This refactors so that there is no need to _extract_partitions and use client.submit which seems to really help with task graph craziness and also doesn't require persisting.

ilan-gold commented 3 months ago

Using map_blocks is blocked by https://github.com/dask/dask/issues/11201