i4Ds / Karabo-Pipeline

The Karabo Pipeline can be used as Digital Twin for SKA
https://i4ds.github.io/Karabo-Pipeline/
MIT License
11 stars 4 forks source link

User can specify how many resources (GPU/Memory) a DASK-Task requires #450

Open sfiruch opened 1 year ago

sfiruch commented 1 year ago

Because RASCIL, as example, needs for one step a lot of memory (~15 GB in my example), which is more than what a worker gets. -> Investigate how those cases should be handled, especially if we design that each core should get one worker.

This is required if we want to use Dask to parallelize single-threaded algorithms on a node.