CliMA / slurm-buildkite

Run buildkite jobs on a slurm cluster
Other
8 stars 1 forks source link

Better handling of mutually exclusive slurm flags #47

Open simonbyrne opened 1 year ago

simonbyrne commented 1 year ago

In https://github.com/CliMA/ClimaAtmos.jl/pull/1998#issuecomment-1697818899, we hit an issue where slurm_mem was set globally, and slurm_mem_per_cpu was set as an override. Unfortunately according to the docs, these are mutually exclusive, and so the job would not schedule.

At the moment, we just pass the args straight through to sbatch, but we could have some mechanism where we give one priority?

Also, should we cancel the whole build if a Slurm submission fails?