One of the challenges of running kibo was the sheer number of jobs, ~50k. This required extra queue management to avoid submit limits, made rare startup problems common to encounter, and added overhead to reservations which sometimes had ~10% of the nodes idle while turning over jobs. To reduce the number of jobs,
bundle arc and psfnight jobs together for a 6 -> 1 reduction for those steps. Consider also bundling the ccdcalib job into this bunch.
bundle flat and nightlyflat jobs together for a 13 -> 1 reduction of those steps
Although there are still more science jobs than there are arc/psf/flat/nightlyflat jobs, these calib jobs are tightly coupled and blocking factors for getting to the more genuinely independent tilenight and ztile jobs, so bundling them would streamline the workflow.
One of the challenges of running kibo was the sheer number of jobs, ~50k. This required extra queue management to avoid submit limits, made rare startup problems common to encounter, and added overhead to reservations which sometimes had ~10% of the nodes idle while turning over jobs. To reduce the number of jobs,
Although there are still more science jobs than there are arc/psf/flat/nightlyflat jobs, these calib jobs are tightly coupled and blocking factors for getting to the more genuinely independent tilenight and ztile jobs, so bundling them would streamline the workflow.