Describe the feature you'd like
Use Dagger.jl to provide multi-node and/or hybrid (multi-thread on multiple nodes) parallelism for large batches of tasks.
How would this feature be used? Please describe.
Users would add Dagger workers (using their local HPC cluster or some cloud resource) and use the Dagger scheduler to handle partitioning batch items among them.
Describe alternatives you've considered
For multi-node support, Julia's builtin Distributed stdlib would be an option. Dagger.jl provides a graph-based scheduler and the ability to tag resources needed (for example, GPUs) which are a bit more sophisticated.
Additional context
A Julia package extension would be a natural way to add support for this.
Describe the feature you'd like Use Dagger.jl to provide multi-node and/or hybrid (multi-thread on multiple nodes) parallelism for large batches of tasks.
How would this feature be used? Please describe. Users would add Dagger workers (using their local HPC cluster or some cloud resource) and use the Dagger scheduler to handle partitioning batch items among them.
Describe alternatives you've considered For multi-node support, Julia's builtin
Distributedstdlib would be an option.Dagger.jlprovides a graph-based scheduler and the ability to tag resources needed (for example, GPUs) which are a bit more sophisticated.Additional context A Julia package extension would be a natural way to add support for this.