mark-welsh
commented
5 years ago 54f8368 fixes the unset variable issue seen when testing this fix live in a pipeline run test. initially @genomics-geek had the set -Eeuxo pipefail; command but it was completely removed in c669e5a98b8aa498035ad1824c53a8c7fa12439e . By just removing -u, we can catch the failed scripts without being too strict about what nested commands do (e.g. the pyenv/bin/activate script with unset vars)
For UI issues
Description
During RNA-Seq testing, 1/6 pipelines failed because the inhouse Python scripts could not import modules from the pyenv binary. This was confusing since all 6 pipelines ran the same PreparePipelineResources subworkflow, so all 6 should have installed the same requirements. However, if the pyenv fails on the
pip install -r requirementsstep and one module is missing, the CreateVirtualenv task should be retried. Instead, it moves to the next bash command, which returns 0, and so the task writes anrcfile with 0 and the pipeline continues.Proposal
This task needs to include
set -Eeuxo pipefail;. Adding this will cause the entire task to exit 1 if any of the commands fails. In a running pipeline with DGD Cromwell configs, this will cause the task to be retried. If successful, it will continue. If not, it will fail here where the actual error is.Error Logs
Here is the stderr from CreateVirtualenv task in the failed pipeline. This definitely exited with 1 but the pipeline ran all the way to the end until the pyenv was needed.