Weaver: Workflow Execution Management Service (EMS); Application, Deployment and Execution Service (ADES); OGC API - Processes; WPS; CWL Application Package
In order to dispatch a Process execution onto a cluster with Slurm job queue management, a utility could be added to Weaver.
Requirements
[ ] Add a SlurmRequirement that would indicate to CWL that the Process job must be wrapped in a Slurm connector/handler.
Could include additional parameters for configuration of execution strategies, such as maximum number of concurrent Slurm-jobs.
Should be able to consider and validate other requirements, such as #104 (requested resources should be available on cluster).
[ ] Consider limitations by clusters for running containers (e.g.: DockerRequirement often blocked due to security reasons, Singularity permitted but requires that the utility is installed on the cluster).
[ ] Add settings such as weaver.cwl_slurm_hostname, weaver.cwl_slurm_username, weaver.cwl_slurm_password, etc. to define the SSH connection details to the cluster where Slurm is available.
Those definitions should not be defined in the CWL package and requirements defintiions to avoid leaking the sensible details when requested through package endpoints and stored in the database.
[ ] Document examples on how to define a Process and the corresponding CWL using this functionality.
Description
In order to dispatch a Process execution onto a cluster with Slurm job queue management, a utility could be added to Weaver.
Requirements
SlurmRequirementthat would indicate to CWL that the Process job must be wrapped in a Slurm connector/handler.DockerRequirementoften blocked due to security reasons,Singularitypermitted but requires that the utility is installed on the cluster).weaver.cwl_slurm_hostname,weaver.cwl_slurm_username,weaver.cwl_slurm_password, etc. to define the SSH connection details to the cluster where Slurm is available.References