Description
In order to scale up the number of parallel jobs and ease the deployment of flows, a worker pool needs to be added. This requires parameterize some of the jobs.
Update: Main goal is to be able to deploy updated flows to a remote prefect instance, i.e from the laptop to scopem-openem.ethz.ch
Solution proposals
Add a docker worker
Definition of Done
Implementation is deployed on Scopem-openem.ethz.ch
workers can be added to prefect on a different physical or the same machine
workers run in a container and span flows in a container
Description In order to scale up the number of parallel jobs and ease the deployment of flows, a worker pool needs to be added. This requires parameterize some of the jobs.
Update: Main goal is to be able to deploy updated flows to a remote prefect instance, i.e from the laptop to scopem-openem.ethz.ch
Solution proposals Add a docker worker
Definition of Done