Launching 200 jobs on ICAv2 at the moment (using FPGAs), means that about 20 run, 180 hang, and then about 12 hours later, 180 just start and then hang at the first step. This generates costs due to the FSx backed storage that the data sits on.
Instead we could consider jobs being submitted to 'queues' and that way the WES handler submits jobs AS there is availability (say no more than 20 FPGAs run simultaneously). We could also then add in 'priorities' for certain queues or certain jobs
Launching 200 jobs on ICAv2 at the moment (using FPGAs), means that about 20 run, 180 hang, and then about 12 hours later, 180 just start and then hang at the first step. This generates costs due to the FSx backed storage that the data sits on.
Instead we could consider jobs being submitted to 'queues' and that way the WES handler submits jobs AS there is availability (say no more than 20 FPGAs run simultaneously). We could also then add in 'priorities' for certain queues or certain jobs