Comparison of job schedulers used by workflow managers - Githubissues

kinow / kinoshita.eti.br

kinow website

https://kinoshita.eti.br

Other

4 stars 2 forks source link

Comparison of job schedulers used by workflow managers #198

Open kinow opened 1 year ago

kinow commented 1 year ago

e.g.

ECMWF's Troika (used by ecFlow)
DRMAA (and some of its implementations…)
Cylc job handlers
Autosubmit platforms (with wrappers)
Lemmings (& https://cerfacs.fr/coop/lemmings-general-intro)
toil batch schedulers (CWL runner)
StreamFlow connectors (they support both HPC and k8s, cloud)
Pegasus (is the smart allocation based on data-usage part of the job scheduler?)

Questions:

How do these different implementations compare to each other?
Any unique features? (e.g. Autosubmit wrappers to mitigate queueing time in competitive systems, or StreamFlow support for k8s/cloud...)
Could they be replaced by a common tool/library, like DRMAA/Lemmings/Troika?