CliMA / slurm-buildkite

Run buildkite jobs on a slurm cluster
Other
7 stars 0 forks source link

add job timeouts #16

Closed jakebolewski closed 4 years ago

jakebolewski commented 4 years ago

we should add job timeouts on both the buildkite side and on the slurm side of things to avoid getting stuck in a pending / waiting state.

jakebolewski commented 4 years ago

this is probably better solved through a bors timeout, then deadlocked jobs will be canceled automatically. slurm already has a timeout once it's running and there is no pending timeout for sbatch