search

theiagen / public_health_bioinformatics

Bioinformatics workflows for genomic characterization, submission preparation, and genomic epidemiology of pathogens of public health concern.
GNU General Public License v3.0
37 stars 17 forks source link

[mercury] improve error handling in mercury task #615

Open kapsakcj opened 1 week ago

kapsakcj commented 1 week ago

:cool:

:pushpin: Explain the Request

This is relevant to the Mercury_prep_N_Batch workflow, specifically workflow v2.2.0 and higher which uses Mercury v1.0.8

I have seen instances where the mercury.py script that runs inside the mercury WDL task throws an error, for example a key error (due to missing required metadata) but the WDL task finishes and succeeds. I expect the task to fail in these instances.

It would allow the user and Theiagen team to troubleshoot more easily because the error is somewhat hidden until digging into the log file for the mercury task.

We could mitigate this by adding set -e to the WDL command block somewhere prior to this line: https://github.com/theiagen/public_health_bioinformatics/blob/6d44d5f6cd734548ee42b3109c5da5bdb4290916/tasks/utilities/submission/task_mercury.wdl#L34

An alternative and perhaps more robust method(?) is to improve the error handling in the mercury python code itself and make the script throw a non-zero exit code anytime an error is encountered. I'm not super knowledgeable on this, but that's my hunch.

kapsakcj commented 1 week ago

This has now occurred with 2 different labs. I propose we bump up the priority on this issue

kapsakcj commented 6 days ago

started a notion page here to document issues as they arise: https://www.notion.so/theiagen/Improve-Mercury-error-handling-103cb013bc93807599b6f71349d921b9

We can use this when updating Mercury python code and/or the Mercury_Prep_N_batch WDL code