Yes, but as a tester. I don't have enough knowledge of dask-jobqueue and Prefect to write it from scratch.
What problem are you trying to solve?
With prefect-dask it is possible to use Prefect with an HPC environment that uses a job scheduler like Slurm. However, there is effectively no documentation on how to use Prefect with these kinds of compute environments. It would be great to have some recipes to describe this scenario.
Describe the solution you'd like
An example of how to have an agent set up on an HPC environment that will continually poll for work and submit tasks as Slurm compute jobs when they come in.
💻
Yes, but as a tester. I don't have enough knowledge of
dask-jobqueueand Prefect to write it from scratch.What problem are you trying to solve?
With
prefect-daskit is possible to use Prefect with an HPC environment that uses a job scheduler like Slurm. However, there is effectively no documentation on how to use Prefect with these kinds of compute environments. It would be great to have some recipes to describe this scenario.Describe the solution you'd like
An example of how to have an agent set up on an HPC environment that will continually poll for work and submit tasks as Slurm compute jobs when they come in.
Describe alternatives you've considered
Not using Prefect
Documentation, Adoption, Migration Strategy
No response