More than 3 simultaneous HPC jobs hits CPU limit?

[Aariq]

It seems like we are either getting close to the compute limit, or there's some other limit about the number of simultaneous jobs/nodes/cores.

(puma) [ericrscott@wentletrap ~]$ squeue -u ericrscott
             JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
           6753242  standard PEcAn-SA ericrsco PD       0:00      2 (AssocGrpCPUMinutesLimit)
           6753209  standard PEcAn-SA ericrsco  R   17:28:25      2 r1u11n2,r4u29n2
           6753173  standard PEcAn-SA ericrsco  R   17:40:42      2 r1u26n1,r2u32n1
           6753207  standard PEcAn-SA ericrsco  R   17:34:33      2 r4u08n2,r4u13n2

[Aariq]

Possible solutions or things to try:

• Use windfall partition (cons: long wait to get node, jobs can get interrupted(?))
• Reduce time requested (currently 7 days when probably only 4 days needed)
• Reduce number of cores per job (either by reducing number of runs or just having each job run twice as long)