Try adding a stagger/jitter to the start time of Query PanelApp jobs to mitigate the impact of multiple runs. This could be done at the production-pipelines level, or in the query script. Even just spacing the calls out a bit could let the PanelApp server breathe
Is it worth doing one bulky call, pulling all the salient details from PanelApp, then getting subsequent jobs to reference that cached data file?
gut feeling is.. no. This can help with usage at scale, but would make things worse for single runs.
alternatively this creates a CPG-only execution path to cope with running ~20 projects at once
PanelApp interaction is one of the main points of fragility (it's a hard cap on the number of jobs I can run in parallel)
https://batch.hail.populationgenomics.org.au/batches/445513/jobs/5
Query PanelAppjobs to mitigate the impact of multiple runs. This could be done at the production-pipelines level, or in the query script. Even just spacing the calls out a bit could let the PanelApp server breathe