falkamelung
commented
3 years ago I modified sbatch_conditinal to do some logging when this occurs. It appears an max job numbers per queue issue:
run_08_gen..._30.job | 8 | 200/500 | 200/1000 | 31/80 | 25/25 | Wait 5 min |
| run_08_gen..._30.job | 8 | 168/500 | 168/1000 | 31/80 | 18/25 | Submitted: 7763428 |
| run_08_gen..._31.job | 8 | 176/500 | 184/1000 | 32/80 | 20/25 | Submitted: 7763431 |
| run_08_gen..._32.job | 8 | 184/500 | 184/1000 | 33/80 | 22/25 | Submitted: 7763433 |
| run_08_gen..._33.job | 8 | 192/500 | 200/1000 | 34/80 | 23/25 | Submitted: 7763437 |
| run_08_gen..._34.job | 8 | 200/500 | 208/1000 | 35/80 | 24/25 | sbatch message:
-----------------------------------------------------------------
Welcome to the Stampede2 Supercomputer
-----------------------------------------------------------------
No reservation for this job
--> Verifying valid submit host (login3)...OK
--> Verifying valid jobname...OK
--> Enforcing max jobs per user...OK
--> Verifying availability of your home dir (/home1/05861/tg851601)...OK
--> Verifying availability of your work2 dir (/work2/05861/tg851601/stampede2)...OK
--> Verifying availability of your scratch dir (/scratch/05861/tg851601)...OK
--> Verifying valid ssh keys...OK
--> Verifying access to desired queue (skx-normal)...OK
--> Verifying job request is within current queue limits...FAILED
[*] Too many simultaneous jobs in queue.
--> Max job limits for skx-normal = 25 jobs
sbatch submit error: exit code 1. Sleep 60 seconds and try again
Jobs submitted: 
Ovec8hkin
While tracking down unexplained failures I noticed that a job did not get submitted. It skipped run_07*_17.job:
Do you have an explanation or is this a rogue failure? This might explain most of my unexplainable failures in the last weeks.
If a failure occurs it should try to resubmit a few times and if it is still not successful it should raise an exception and exit.
Two things to consider:
sbatchcommand and capture and analyzed the error When 25 jobs are already running and you execute sbatch it tells you:No reservation for this job --> Verifying valid submit host (login3)...OK --> Verifying valid jobname...OK --> Enforcing max jobs per user...OK --> Verifying availability of your home dir (/home1/05861/tg851601)...OK --> Verifying availability of your work dir (/work/05861/tg851601/stampede2)...OK --> Verifying availability of your scratch dir (/scratch/05861/tg851601)...OK --> Verifying valid ssh keys...OK --> Verifying access to desired queue (skx-normal)...OK --> Verifying job request is within current queue limits...FAILED
idev detected an error in your resource request (see details above): Here is the command you executed: /bin/idev -p skx-normal -N 1 -n 48
| run_09_mer..._33.job | 17 | 136/500 | 301/1000 | 34/37 | 20/25 | Submitted: 7737332 | | run_09_mer..._34.job | 17 | 153/500 | 348/1000 | 35/37 | 23/25 | sbatch message: sbatch submit error: exit code 1. Sleep 60 seconds and try again sbatch message: sbatch submit error: exit code 1. Exiting with status code 1. Submitted: | | run_09_mer..._35.job | 17 | 153/500 | 369/1000 | 36/37 | 25/25 | Wait 5 min | | run_09_mer..._35.job | 17 | 153/500 | 369/1000 | 36/37 | 23/25 | Submitted: 7737357 | | run_09_mer..._36.job | 2 | 153/500 | 369/1000 | 37/37 | 24/25 | Submitted: 7737362 |
Jobs submitted: 7737104 7737105 7737106 7737109 7737110 7737111 7737112 7737113 7737114 7737115 7737116 7737117 7737118 7737119 7737120 7737121 7737123 7737124 7737125 7737127 7737130 7737132 7737134 7737160 7737210 7737242 7737245 7737247 7737280 7737294 7737298 7737315 7737327 7737332 7737357 7737362 Timedout with walltime of 0:08:32. Resubmitting file (/scratch/05861/tg851601/KokoxiliBigChunk38SenAT114/run_files/run_09_merge_burst_igram_4.job) with new walltime of 00:10:14 Resubmitted as jobumber: 7737366. Timedout with walltime of 0:08:32. Resubmitting file (/scratch/05861/tg851601/KokoxiliBigChunk38SenAT114/run_files/run_09_merge_burst_igram_11.job) with new walltime of 00:10:14 Resubmitted as jobumber: 7737369. Timedout with walltime of 0:08:32. Resubmitting file (/scratch/05861/tg851601/KokoxiliBigChunk38SenAT114/run_files/run_09_merge_burst_igram_24.job) with new walltime of 00:10:14 Resubmitted as jobumber: 7737395. Timedout with walltime of 0:08:32. Resubmitting file (/scratch/05861/tg851601/KokoxiliBigChunk38SenAT114/run_files/run_09_merge_burst_igram_25.job) with new walltime of 00:10:14 Resubmitted as jobumber: 7737413. KokoxiliBigChunk38SenAT114, run_09_merge_burst_igram, 36 jobs: 24 COMPLETED, 3 RUNNING , 8 PENDING , 1 WAITING . KokoxiliBigChunk38SenAT114, run_09_merge_burst_igram, 36 jobs: 24 COMPLETED, 3 RUNNING , 9 PENDING , 0 WAITING ```
########################
. Another case:
########################
########################
. Another case:
########################