Pangolin container separated from Illumina and Nanopore workflows

JD2112 commented 1 year ago

The purpose of the code changes are as follows:

Pangolin typing docker container separated from Illumina and Nanopore (update regularly through GH action #34).
pycoQC container for Nanopore long-reads QC (only for nanopolish)
Added separate package version (#42) files for each workflow -
- versions : for Illumina and Nanopore
- pangoversion : for pangolin typing
Illumina analysis additional features - required for docker-sc2reporter
- flagstat
- depth
- VEP annotation
Nanopore analysis additional features (nanopolish & medaka)- as added #63
- fastqc
- multiqc
- pycoQC (only for nanopolish)
README file updated
Pipeline workflow figure added in README
Latest DAG files updated

Standard test procedure

[x] The self-test mentioned in the README finished successfully
[x ] Produced files contain expected values
[ ] The code has been reviewed by @octocat

This version is a:

[x] MAJOR - when you make incompatible API changes
[ ] MINOR - when you add functionality in a backwards compatible manner
[ ] PATCH - when you make backwards compatible bug fixes or documentation/instructions

henningonsbring commented 1 year ago

Great job @JD2112

I notice that sometimes -primer is used in file names, and sometimes it is removed. I do not see the pattern, think about this one more time and then make sure it is consistent.

Can you please let me know when you have read my comments and taken action/discarded them and then I will run this on our machine and see how it goes.

JD2112 commented 1 year ago

Thanks @henningonsbring Yes, primer scheme 'nCoV-2019-primer' name need to be changed for consistency. I changed all filenames of this directory from V1 to V4.1. I checked with V3 and V4, works fine for me, but not with others. Yes, please test the version and let me know if it works fine. I hope you will find command details on README very easily now :)

henningonsbring commented 1 year ago

Nanopore run worked well, but analysis for Illumina data got stuck with error:

Command error:
  Traceback (most recent call last):
    File "/home/proj/stage/mutant/MUTANT/mutant/externals/gms-artic/bin/type_vcf.py", line 12, in <module>
      import vcf
  ModuleNotFoundError: No module named 'vcf'

henningonsbring commented 1 year ago

When trying to import vcf it does not work:

python
Python 3.9.7 (default, Sep 16 2021, 08:50:36) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import vcf
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'vcf'

When I ran conda install -c bioconda pyvcf

Then I could import vcf:

python
Python 3.9.7 (default, Sep 16 2021, 08:50:36) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import vcf
>>>

JD2112 commented 1 year ago

@henningonsbring I really don't understand why you got this error. The Illumina container has the same pyvcf==0.6.8 in the environment yml file. I'll check this again inside the container. Thanks for the update.

henningonsbring commented 1 year ago

@JD2112 I really do not understand it either, I can run

singularity exec artic-ncov2019-illumina.sif python
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import vcf
>>>

JD2112 commented 1 year ago

but it works for me...

jyotirmoy@z6g4:~$ singularity exec docker://genomicmedicinesweden/gms-artic-illumina:latest python
INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...
Python 3.7.12 | packaged by conda-forge | (default, Oct 26 2021, 06:08:53) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import vcf
>>>

JD2112 commented 1 year ago

@henningonsbring Hi again! Would it be possible to try with docke:// jd21/gms-artic-illumina:220725? It is the same container.

henningonsbring commented 1 year ago

@henningonsbring Hi again! Would it be possible to try with docke:// jd21/gms-artic-illumina:220725? It is the same container.

I will try and get back to you

henningonsbring commented 1 year ago

@JD2112 I solved the problem with import vcf by adding withName:typeVariants to my config for Illumina, so that should be added to nextflow.config as well.

However, now there is another error:

  /// MultiQC 🔍 | v1.11

|           multiqc | MultiQC Version v1.12 now available!
|           multiqc | Only using modules: cutadapt, fastqc, ivar, picard
Traceback (most recent call last):
  File "/opt/conda/envs/artic-ncov2019-illumina/bin/multiqc", line 6, in <module>
    from multiqc.__main__ import multiqc
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/multiqc/__main__.py", line 53, in <module>
    multiqc.run_cli(prog_name="multiqc")
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/multiqc/multiqc.py", line 277, in run_cli
    kwargs=kwargs,
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/multiqc/multiqc.py", line 590, in run
    template_mod = config.avail_templates[config.template].load()
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2470, in load
    self.require(*args, **kwargs)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2493, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/pkg_resources/__init__.py", line 800, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (typing-extensions 3.10.0.2 (/home/hiseq.clinical/.local/lib/python3.7/site-packages), Requirement.parse('typing-extensions<5.0,>=4.0.0; python_version < "3.9"'), {'rich'})

henningonsbring commented 1 year ago

Still problem:

singularity exec /home/proj/stage/mutant/MUTANT/mutant/externals/gms-artic/artic-ncov2019-illumina.sif multiqc . --filename ablequagga_220729-183623_multiqc.html --data-format json --cl_config "picard_config: { general_stats_target_coverage: [10,30,50,100] }" --module cutadapt --module fastqc --module ivar --module picard

  /// MultiQC 🔍 | v1.11

|           multiqc | MultiQC Version v1.12 now available!
|           multiqc | Only using modules: cutadapt, fastqc, ivar, picard
Traceback (most recent call last):
  File "/opt/conda/envs/artic-ncov2019-illumina/bin/multiqc", line 6, in <module>
    from multiqc.__main__ import multiqc
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/multiqc/__main__.py", line 53, in <module>
    multiqc.run_cli(prog_name="multiqc")
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/multiqc/multiqc.py", line 277, in run_cli
    kwargs=kwargs,
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/multiqc/multiqc.py", line 590, in run
    template_mod = config.avail_templates[config.template].load()
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2470, in load
    self.require(*args, **kwargs)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/pkg_resources/__init__.py", line 2493, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/opt/conda/envs/artic-ncov2019-illumina/lib/python3.7/site-packages/pkg_resources/__init__.py", line 800, in resolve
    raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (typing-extensions 3.10.0.2 (/home/hiseq.clinical/.local/lib/python3.7/site-packages), Requirement.parse('typing-extensions<5.0,>=4.0.0; python_version < "3.9"'), {'rich'})

JD2112 commented 1 year ago

but it worked for me :(

singularity exec docker://genomicmedicinesweden/gms-artic-illumina:latest multiqc ncovIllumina_sequenceAnalysis_fastqc/ --filename test_multiqc.html --data-format json --cl_config "picard_config: { general_stats_target_coverage: [10,30,50,100] }" --module cutadapt --module fastqc --module ivar --module picard
INFO:    Using cached SIF image
INFO:    Converting SIF file to temporary sandbox...

  /// MultiQC 🔍 | v1.11

|           multiqc | MultiQC Version v1.12 now available!
|           multiqc | Only using modules: cutadapt, fastqc, ivar, picard
|           multiqc | Search path : /home/jyotirmoy/ncovIllumina_sequenceAnalysis_fastqc
|         searching | ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 8/8  
|            fastqc | Found 4 reports
|           multiqc | Compressing plot data
|           multiqc | Report      : test_multiqc.html
|           multiqc | Data        : test_multiqc_data
|           multiqc | MultiQC complete
INFO:    Cleaning up image...

henningonsbring commented 1 year ago

I really can't solve it, have tried the whole day, built many containers on my own, constantly getting stuck with this pkg_resources.ContextualVersionConflict: (typing-extensions 3.10.0.2 (/home/hiseq.clinical/.local/lib/python3.7/site-packages), Requirement.parse('typing-extensions<5.0,>=4.0.0; python_version < "3.9"'), {'rich'}) :( :( :(

henningonsbring commented 1 year ago

What version of singularity are you using @ljmesi ?

henningonsbring commented 1 year ago

It works for us now. To make it work I had to make sure the nanopore container (that got multiqc) was used during the Illumina analysis. Since multiqc will crash if the Illumina container is used, i.e. I replaced this:

withName:multiqc {
                    container = "file:///${baseDir}/artic-ncov2019-illumina.sif"
                }

and instead used this in our config:

withName:multiqc {
                    container = "file:///${baseDir}/artic-ncov2019-nanopore.sif"
                }

JD2112 commented 1 year ago

@henningonsbring Nice...it worked at last :) . I also got the confirmation from GU (Daniel S), it worked for them without any problem.

ljmesi commented 1 year ago

What version of singularity are you using @ljmesi ?

@henningonsbring I was running the workflow using conda environment with these programs included:

name: test-gms-artic-v2
channels:
  - conda-forge
  - bioconda
  - defaults
dependencies:
  - nextflow = 21
  - singularity = 3.8

JD2112 commented 1 year ago

@henningonsbring @ljmesi @sofstam Hi! Thank you very much for your review and approval. I'll merge this PR now.

genomic-medicine-sweden / gms-artic