`remove_chunks` slow down on`OrderedSet.dascard` when there are many chunks

mars-project / mars

Mars is a tensor-based unified framework for large-scale data computation which scales numpy, pandas, scikit-learn and Python functions.

https://mars-project.readthedocs.io

Apache License 2.0

2.68k stars 325 forks source link

Closed chaokunyang closed 1 year ago

chaokunyang commented 1 year ago

When we have many chunks running on each band, remove_chunks will become the bottleneck of system: