[manualpipe] Asklepian 20220105 - Githubissues

COG-UK / dipi-group

Data integrity and pipeline integration working group

4 stars 1 forks source link

[manualpipe] Asklepian 20220105 #178

Closed BioWilko closed 2 years ago

BioWilko commented 2 years ago

Brief description

[Provide a sentence of background]

Pipeline exit code: 1

Executed by: @BioWilko BIRM

[x] I am sure this pipeline is not already running (Only relevant for non-flocked pipelines)

[x] Engage pipeline via mqtt from a clean dedicated mqtt environment
[ ] Disable NXF cache destruction (only applicable for ENA BAM pipe)
[ ] Check output if necessary
[x] Check mqtt message is sent confirming completion, or send one if required (check in #tael-steam)
[ ] Re-enable NXF cache destruction (if disabled)

[ ] Set appropriate issue flags and assignee
[ ] Manual intervention complete
"Normal" service disrupted? YES/NO
If yes, describe impact:

BioWilko commented 2 years ago

Asklepian failed after an attempted re-raise, the pipe appears to be failing at this part of the script.

SamStudio8 commented 2 years ago

Some absolute carnage sequences in here today. I've let @rmcolq know there may be issues downstream.

I think the plan of action is:

[x] Identify the bad eggs
[x] Suppress these "sequences" from the data set with Ocarina
[x] Expedite their removal by running the publish step for today again manually
[x] Raise Asklep
[x] Ensure Asklep makes it to MSA step
[x] Improve the fasta_quickcheck process in elan.nf because this was embarrassing

SamStudio8 commented 2 years ago

https://github.com/SamStudio8/majora/commit/916c5d4c3d4559c74456ed85e2833c4a4b575260 will allow select users to remove absolute garbage from the data set without approval from the uploading institute

SamStudio8 commented 2 years ago

directly assigned can_suppress_any_pags_via_api to @BioWilko who can now mark the 171 affected records in Majora

BioWilko commented 2 years ago

All affected records suppressed.

SamStudio8 commented 2 years ago

confirmed

(samstudio8) [nicholsz@bham control]$ ocarina --oauth --quiet --env get pag --mode pagfiles --test-name 'cog-uk-elan-minimal-qc' --pass --suppressed-after 2022-01-05 --task-wait --task-wait-attempts 15 --task-wait-minutes 1 | grep -c consensus
[WAIT] Giving Majora a minute to finish task fcc75b12-81ca-40d7-918d-cbf9c8870a67 (1)...
[WAIT] Finished waiting with status SUCCESS (1)...
171

SamStudio8 commented 2 years ago

Manually running cog-publish.sh now to force refresh today's data set

SamStudio8 commented 2 years ago

We've never re-run the publish script on the same day like this before so needed a few new tricks:

Updated publish script to run on empty add (as all new files had been processed earlier already there was nothing new to do) https://github.com/SamStudio8/elan-nextflow/commit/1db3637f334a90814ea67c12655c71d42ffed3d1
Fixed bug where BAMs were not suppressed correctly (not an issue as the FASTA were fine but happened to notice it today) https://github.com/SamStudio8/elan-nextflow/commit/7a3c42538e15da8cad7048bab7b2aeb98711e611
Added script to repoint latest to handle catastrophic rollbacks more easily https://github.com/SamStudio8/elan-nextflow/commit/642fae8d8cc16ceec88a82bb101bf59e7ab236e3

SamStudio8 commented 2 years ago

Looking promising:

[DONE] 1848739 hits, 0 misses, 0 skips, 171 kills

SamStudio8 commented 2 years ago

Removing Asklep 20220105 guards

SamStudio8 commented 2 years ago

@BioWilko can you raise

BioWilko commented 2 years ago

Raised

SamStudio8 commented 2 years ago

gofasta is streaming an MSA now so we're out of the woods and back to normal service :rocket:

SamStudio8 commented 2 years ago

Pulled @BioWilko's patch (https://github.com/SamStudio8/elan-nextflow/commit/4828e44e50d4859d8612d06e03e69ca30c1b2dc9) that will ensure the fasta_quickcheck step will boot out spurious garbage like this in future, conveniently also putting #38 to rest for good too.

SamStudio8 commented 2 years ago

MSA is finished and we're proceeding with the genome and variant tables as normal now. That's enough hot sauce for one day, nice work @BioWilko. Call me if anything else blows up!

BioWilko commented 2 years ago

This issue has had knock on effects, follow this issue for more information.

SamStudio8 commented 2 years ago

Close but no cigar on this one. I should have noticed this yesterday:

We've never re-run the publish script on the same day like this before so needed a few new tricks:

Updated publish script to run on empty add (as all new files had been processed earlier already there was nothing new to do) SamStudio8/elan-nextflow@1db3637

There should have been new work to do because we were publishing OVER the newly published data.