search

epi2me-labs / wf-artic

ARTIC SARS-CoV-2 workflow and reporting
https://labs.epi2me.io/
Other
49 stars 36 forks source link

process runArtic silently failing #29

Closed phyden closed 2 years ago

phyden commented 2 years ago

Hi,

I am running your pipeline using the conda profile like this:

nextflow run epi2me-labs/wf-artic --scheme_name SARS-CoV-2 --scheme_version V1200 --min_len 200 --max_len 1100 --out_dir $output_dir --fastq $fastq_dir -profile conda --samples $samples -work-dir $workdir

And frequently got consensus sequences containing only one "N" for all barcodes although the coverage seems to be decent for most samples. However, for some reason this seemed to happen randomly depending on the $workdir I specified.

Looking at the closed issues on your repo and into the working directories of the failing pipeline I was able to track the error down. I'm working behind a proxy which I did not specify in my working environment, only in my testing environment as environment variable.. So basically medaka failed (similar to #21 , because it was not able to download the required model files (while conda takes the global proxy_servers.http setting for setting up the conda-env).

The error message was:

Traceback (most recent call last):
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/site-packages/medaka/medaka.py", line 35, in __call__
    model_fp = medaka.models.resolve_model(val)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/site-packages/medaka/models.py", line 66, in resolve_model
    raise DownloadError(
medaka.models.DownloadError: The model file for r941_prom_variant_g360 is not already installed and could not be downloaded. Check you are connected to the internet and try again.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/bin/medaka", line 11, in <module>
    sys.exit(main())
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/site-packages/medaka/medaka.py", line 686, in main
    args = parser.parse_args()
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1768, in parse_args
    args, argv = self.parse_known_args(args, namespace)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1800, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1988, in _parse_known_args
    positionals_end_index = consume_positionals(start_index)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1965, in consume_positionals
    take_action(action, args)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1874, in take_action
    action(self, namespace, argument_values, option_string)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1159, in __call__
    subnamespace, arg_strings = parser.parse_known_args(arg_strings, None)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1800, in parse_known_args
    namespace, args = self._parse_known_args(args, namespace)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 2006, in _parse_known_args
    start_index = consume_optional(start_index)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1946, in consume_optional
    take_action(action, args, option_string)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/argparse.py", line 1874, in take_action
    action(self, namespace, argument_values, option_string)
  File "/srv/sequencer/MiSeqAnalysis_OUT/work/conda/epi2melabs-nf-artic-08dd4ae442f90d9110476d0a93da01f2/lib/python3.8/site-packages/medaka/medaka.py", line 38, in __call__
    raise RuntimeError(msg.format(self.dest, str(e)))
RuntimeError: Error validating model from '--model' argument: The model file for r941_prom_variant_g360 is not already installed and could not be downloaded. Check you are connected to the internet and try again..
Running hacked up minion.py

However, since the process script is out-sourced to the bin/run_artic.sh file, and there is no error handling in this script, it will always return an exit status of 0 and the nextflow process never fails as long as the output files are present. This makes debugging quite difficult for users with no experience in nextflow. I assume that issue #9 was also related to this underlying problem.

Maybe you can think of a cleaner solution like the errorStrategy ignore directive?

bede commented 2 years ago

This appears still to be a problem in the latest pipeline release. It would be helpful if there were some error handling or logging in this step, which is fragile and internet connection dependent.