Integrate EOEPCA with a Slurm instance

[agstephens]

There is established experience with LOTUS of packaging applications as containers using Singularity (https://singularity-docs.readthedocs.io), a technology developed specifically for use with batch computing which is compatible with Docker.

Our plan:

• [ ] Install Slurm in CaaS
• [ ] Investigate how we could submit, queue, allocate, execute ADES jobs in CaaS

Specifically investigate:

• [ ] How you might allocate resources to a job (e.g. n cores or memory)
• [ ] Would we run all jobs as user eoepca or actually have users?
  • We could use job IDs instead of the user ID - and map usage info via that
• [ ] How we might measure resource usage in terms of how that could feed back to monitoring usage and billing (potentially)
• [ ] When tracking resources: you could track the request rather than the actual measured usage - worth considering both approaches.

[agstephens]

Update: 14/04/2023

• Slurm has an API that can be used internally - but does exist

  • we could plug that in with a POST request
  • could integrate nicely with Singularity

• Steps would be:

  1. User request to run a process - through the ADES
  2. ADES would generate a job-script
  3. ADES would send job-script as POST request to Slurm API
  4. Slurm will use the job-script to invoke a Singularity container based on the Dockerfile (pulled from EOEPCA registry)
  5. ADES would monitor the job via Slurm API (probably)
  6. Some kind of finalise/clean-up actions will happen (e.g. registering outputs)

• We need to understand stage-in and stage-out processes - and where they are managed

ACTIONS:

• [ ] Find out the full lifecycle of a job: e.g. waiting/monitoring/staging/finalising