[manualpipe] Asklepian 20220220

SamStudio8 commented 2 years ago

Brief description

Ocarina failed as Majora has "gone away". Additionally, ceph has dismounted.

Pipeline exit code: 69

Executed by: [Name or initials] [Site code]

[x] I am sure this pipeline is not already running (Only relevant for non-flocked pipelines)

[x] Engage pipeline via mqtt from a clean dedicated mqtt environment
[ ] Disable NXF cache destruction (only applicable for ENA BAM pipe)
[ ] Check output if necessary
[ ] Check mqtt message is sent confirming completion, or send one if required (check in #tael-steam)
[ ] Re-enable NXF cache destruction (if disabled)

[x] Set appropriate issue flags and assignee
[x] Manual intervention complete
"Normal" service disrupted? YES/NO
If yes, describe impact:

SamStudio8 commented 2 years ago

waiting for ceph to be remounted by systems

SamStudio8 commented 2 years ago

we are go

SamStudio8 commented 2 years ago

an hour ago we timed out waiting for asklep table, which is unusual as the wait limit is around twice the expected time for the view. i restarted both majora and celery, and the task finished within parameters -- asklep is ticking along making tables now

SamStudio8 commented 2 years ago

back to automated control

COG-UK / dipi-group

[manualpipe] Asklepian 20220220 #190