Closed jakebolewski closed 4 years ago
we should add job timeouts on both the buildkite side and on the slurm side of things to avoid getting stuck in a pending / waiting state.
this is probably better solved through a bors timeout, then deadlocked jobs will be canceled automatically. slurm already has a timeout once it's running and there is no pending timeout for sbatch
we should add job timeouts on both the buildkite side and on the slurm side of things to avoid getting stuck in a pending / waiting state.