I'm sold on the idea to explore a feature to throttle something across X dask workers, providing a bit more advanced functionality next to Lock, Semaphore, and Variable.
This idea is the result of collaborative brainstorming in a cryo meeting 7th October 2021 where @espg described having run into rate limitations downloading data from a central location across multiple separate dask workers.
Action points
[ ] Make some research if this is already solved somehow or what tools are available
[ ] Figure out a feature proposal to raise, and raise it
I'm sold on the idea to explore a feature to throttle something across X dask workers, providing a bit more advanced functionality next to Lock, Semaphore, and Variable.
This idea is the result of collaborative brainstorming in a cryo meeting 7th October 2021 where @espg described having run into rate limitations downloading data from a central location across multiple separate dask workers.
Action points