[manualpipe] Elan 20220219

[BioWilko]

Brief description

Elan is timing out on save_manifest (exit code 65), we suspect that moving test elan to run at the exact same time (rather than 30 mins later) caused too much stress on majora and lead to the timeout

Pipeline exit code: 1

---

Executed by: @BioWilko BIRM

• [x] I am sure this pipeline is not already running (Only relevant for non-flocked pipelines)

---

• [x] Engage pipeline via mqtt from a clean dedicated mqtt environment
• [ ] Disable NXF cache destruction (only applicable for ENA BAM pipe)
• [ ] Check output if necessary
• [ ] Check mqtt message is sent confirming completion, or send one if required (check in #tael-steam)
• [ ] Re-enable NXF cache destruction (if disabled)

---

• [x] Set appropriate issue flags and assignee

• [x] Manual intervention complete

• "Normal" service disrupted? YES

• If yes, describe impact: past my bedtime

[BioWilko]

I have updated the test elan cronjob to run at 5AM instead to prevent recurrence tomorrow

[SamStudio8]

I think we've been teetering on the edge of the time limit for a little while but I think it's a good call to suspect running two task_get_sequencing_faster celery tasks at the same time might impact the performance. I'll bump the time limit and also use one of the task ids from today to prevent Elan needing to wait again.

[SamStudio8]

save_manifest OK

[BioWilko]

excellent, it's just using the task id correct? If so I may as well get test elan going too

[SamStudio8]

@BioWilko I've handled test elan

[SamStudio8]

Nearly 26k samples coming in -- a new record.

[SamStudio8]

25012 accepted by Elan ingest, firing requests at Ocarina

[SamStudio8]

25012 PAGs created in Majora, onto publish ~ ETA 25m

[SamStudio8]

and we're out of here