It4innovations / hyperqueue

Scheduler for sub-node tasks for HPC systems with batch scheduling
https://it4innovations.github.io/hyperqueue
MIT License
275 stars 21 forks source link

Implement data transfers between tasks #669

Open Kobzol opened 8 months ago

Kobzol commented 8 months ago

The core of HyperQueue is partially prepared for direct data transfers between tasks, but this functionality is not exposed to the Python API at the moment, where it would make the most sense, to allow returning data from Python function tasks.

Once implemented, its infrastructure could be unified with I/O streaming, to avoid having two separate channels for data transfers.