Open abhi18av opened 2 years ago
@Mxrcon , before making further progress on https://github.com/mtb-bioinformatics/mtbseq-nf/issues/63 - let's iron out the details here and then invest further time there.
...
...
1>>.command.out \
2>>.command.err \
|| true
The necessity for the above snippet is to counter the irregularities we have seen in the MTBseq
exit status and how Nextflow interpreted those in terms of the workflow.
One of the possibilities of these errors is due to the absence of logs (which the proposed bash script https://github.com/mtb-bioinformatics/mtbseq-nf/pull/65 addressed) but perhaps we can also address other such as parsing the log
files generated by MTBseq
within a process and check if there's any presence of ERROR
etc.
What are you thoughts regarding other use-cases we can cover?
@abhi18av I Don't like the idea of creating a new process to parse log, in my opinion every process should check the logs in the end of the job, so this way if a process causes an error we shouldn't continue using it outputs.
We already saw a lot of times when some samples didn't worked and the workflow continued using the outputs and failed into the next process due the missingness of input files, I think that our main work should be focused on solving this error cases. Maybe using a bash script to check the .command.out will be too much meta evaluation of logs, but seems to be a good alternative as it'd capture every output of the commands.
Thanks for sharing your thoughts - I think we might be out of sync a bit here, perhaps we discuss this again when we connect.
sure! makes complete sense!
Adding a suggestion by Davi, to capture the exit code manually (in a non-NF dotfile)
https://www.cyberciti.biz/faq/bash-get-exit-code-of-command/
Given the recent experience with files transfers on clusters, the errors raised by GATK tool were ignored as part of the current solution
Ideally we capture this via a shell construct (maybe from
bin
folder) and throwexit 1