The job manager needs to keep the cluster busy as good as possible, to ensure that X jobs finish within estimated time.
Currently, we see that it can take a (relatively) long time between finishing a job, dealing with the output, and creating and starting a new job.
One trick could be to also have a queue of created jobs, that can be started as soon as another job is finished. The creation of jobs can then happen outside of the critical path between job finish and start.
Extra good would be if we could even already start the next job, and only then start handling results of finished job.
Note: if openEO backend has better support for job queue's, like yarn, it could be possible to already start the job as well, but this is not the case on current CDSE.
The job manager needs to keep the cluster busy as good as possible, to ensure that X jobs finish within estimated time. Currently, we see that it can take a (relatively) long time between finishing a job, dealing with the output, and creating and starting a new job.
One trick could be to also have a queue of created jobs, that can be started as soon as another job is finished. The creation of jobs can then happen outside of the critical path between job finish and start.
Extra good would be if we could even already start the next job, and only then start handling results of finished job.
Note: if openEO backend has better support for job queue's, like yarn, it could be possible to already start the job as well, but this is not the case on current CDSE.