search

facebookincubator / submitit

Python 3.8+ toolbox for submitting jobs to Slurm
MIT License
1.16k stars 115 forks source link

InfoWatch might get previous jobid info after slurm restart #1716

Open Liangtaiwan opened 1 year ago

Liangtaiwan commented 1 year ago

After slurm restarting, sacct -j [jobid] can still provide previous run information.

I face this issue when using hydra-submitit-launcher and I submit a PR that temporary solve the issue. https://github.com/facebookresearch/hydra/pull/2423

Ideally, we should solve this issue in submitit Infowatch part.

gwenzek commented 1 year ago

how do you get an old jobid ?