search

mycobactopia-org / MTBseq-nf

MTBSeq made simple and easy using Nextflow and nf-core standard.
https://doi.org/10.5281/zenodo.5498063
MIT License
8 stars 1 forks source link

Evaluate the use of hard-coded exit status as a fallback for mtbseq statuses #63

Open abhi18av opened 2 years ago

abhi18av commented 2 years ago

Given the recent experience with files transfers on clusters, the errors raised by GATK tool were ignored as part of the current solution

Ideally we capture this via a shell construct (maybe from bin folder) and throw exit 1

##### ERROR A USER ERROR has occurred (version 3.8-0-ge9d806836):
abhi18av commented 2 years ago

@Mxrcon , before making further progress on https://github.com/mtb-bioinformatics/mtbseq-nf/issues/63 - let's iron out the details here and then invest further time there.

...
...
    1>>.command.out \
    2>>.command.err \
    || true

The necessity for the above snippet is to counter the irregularities we have seen in the MTBseq exit status and how Nextflow interpreted those in terms of the workflow.

One of the possibilities of these errors is due to the absence of logs (which the proposed bash script https://github.com/mtb-bioinformatics/mtbseq-nf/pull/65 addressed) but perhaps we can also address other such as parsing the log files generated by MTBseq within a process and check if there's any presence of ERROR etc.

What are you thoughts regarding other use-cases we can cover?

Mxrcon commented 2 years ago

@abhi18av I Don't like the idea of creating a new process to parse log, in my opinion every process should check the logs in the end of the job, so this way if a process causes an error we shouldn't continue using it outputs.

We already saw a lot of times when some samples didn't worked and the workflow continued using the outputs and failed into the next process due the missingness of input files, I think that our main work should be focused on solving this error cases. Maybe using a bash script to check the .command.out will be too much meta evaluation of logs, but seems to be a good alternative as it'd capture every output of the commands.

abhi18av commented 2 years ago

Thanks for sharing your thoughts - I think we might be out of sync a bit here, perhaps we discuss this again when we connect.

Mxrcon commented 2 years ago

sure! makes complete sense!

abhi18av commented 2 years ago

Adding a suggestion by Davi, to capture the exit code manually (in a non-NF dotfile)

https://www.cyberciti.biz/faq/bash-get-exit-code-of-command/