Closed antoine4ucsd closed 2 years ago
Hello @antoine4ucsd,
Does this work?
nextflow run epi2me-labs/wf-mpx --fastq test_data/reads.fastq.gz -profile conda
Thanks
Matt
after activate my nextflow env, I ran your suggested cmd line and got this error
nextflow run epi2me-labs/wf-mpx --fastq test_data/reads.fastq.gz -profile conda
N E X T F L O W ~ version 21.10.6
Launching `epi2me-labs/wf-mpx` [insane_elion] - revision: 038b325037 [master]
Core Nextflow options
revision : master
runName : insane_elion
launchDir : /Users/me/Dropbox/_mxpv/wf-mpx-master
workDir : /Users/me/Dropbox/_mxpv/wf-mpx-master/work
projectDir : /Users/me/.nextflow/assets/epi2me-labs/wf-mpx
userName : me
profile : conda
configFiles : /Users/me/.nextflow/assets/epi2me-labs/wf-mpx/nextflow.config, /Users/me/Dropbox/_mxpv/wf-mpx-master/nextflow.config
Basic Input/Output Options
fastq : test_data/reads.fastq.gz
Meta Data
disable_ping : false
Other parameters
process_label: wfmpx
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use epi2me-labs/wf-mpx for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
[- ] process > start_ping:pingMessage -
[- ] process > handleSingleFile -
[- ] process > pipeline:summariseReads -
[- ] process > pipeline:alignReads -
[- ] process > pipeline:coverageCalc -
[- ] process > pipeline:medakaVariants -
[- ] process > pipeline:makeConsensus -
[- ] process > pipeline:getVersions -
[- ] process > pipeline:getParams -
[- ] process > pipeline:flyeAssembly -
[- ] process > pipeline:makeReport -
[- ] process > output -
[- ] process > end_ping:pingMessage -
Checking fastq input.
Single file input detected.
Creating env using mamba: /Users/me/.nextflow/assets/epi2me-labs/wf-mpx/environment.yaml [cache /Users/me/Dropbox/_mxpv/wf-mpx-master/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01]
WARN: Output `set` must define at least two component -- Check process `pipeline:coverageCalc`
Error executing process > 'handleSingleFile (1)'
Caused by:
Failed to create Conda environment
command: mamba env create --prefix /Users/me/Dropbox/_mxpv/wf-mpx-master/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01 --file /Users/me/.nextflow/assets/epi2me-labs/wf-mpx/environment.yaml
status : 1
message:
Traceback (most recent call last):
File "/Users/me/opt/miniconda3/bin/mamba", line 11, in <module>
sys.exit(main())
File "/Users/me/opt/miniconda3/lib/python3.9/site-packages/mamba/mamba.py", line 882, in main
from conda.common.compat import ensure_text_type, init_std_stream_encoding
ImportError: cannot import name 'init_std_stream_encoding' from 'conda.common.compat' (/Users/me/opt/miniconda3/lib/python3.9/site-packages/conda/common/compat.py)
I think there is an issue with conda/mamba - there is an issue here that could be of help: https://github.com/mamba-org/mamba/issues/1706
You could try:
conda remove mamba
conda install mamba conda=4.12.0 -c conda-forge
I tried this but then I get a new error
from conda._vendor.auxlib.ish import dals
ModuleNotFoundError: No module named 'conda._vendor.auxlib'
Is there any reason why you can't use docker?
Thanks
Matt
ok. I will try this. as for your question regarding docker: the only (bad) reason is that I am not familiar with it. If I cannot fix my conda env issue, I will try to learn how to use Docker.
thank you for being responsive.
Not a bad reason - all you have to do is install the docker desktop app for MacOS and that's all you need to do!
I just got my mamba working by doing this:
conda activate base
conda install conda=4.12
Try that too! Let me know how you get on and if you can get it working in conda or if you decide to try docker!
I will keep you posted asap
Hey!
I am having a similar issue when running the wf via the conda profile (unfourtunatly our institution does not let us use docker, therefore I depend on the conda wf)
When running the command: nextflow run epi2me-labs/wf-mpx -profile conda --fastq ./input/ --out_dir ./output/
I get the following message:
N E X T F L O W ~ version 22.04.0
Launching (https://github.com/epi2me-labs/wf-mpx) [evil_avogadro] DSL2 - revision: 038b325037 [master]
Core Nextflow options
revision : master
runName : evil_avogadro
launchDir : /MPOX/epi2me_labs_wf-mpx/run220713_Gridion_test/Congo_sample
workDir :/MPOX/epi2me_labs_wf-mpx/run220713_Gridion_test/Congo_sample/work
projectDir : /home/User/.nextflow/assets/epi2me-labs/wf-mpx
userName : User
profile : conda
configFiles : /home/User/.nextflow/assets/epi2me-labs/wf-mpx/nextflow.config`
Basic Input/Output Options
out_dir : ./output/
fastq : ./input/
Meta Data
disable_ping : false
Other parameters
process_label: wfmpx
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use epi2me-labs/wf-mpx for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
[- ] process > start_ping:pingMessage -
[- ] process > start_ping:pingMessage -
[- ] process > start_ping:pingMessage [ 0%] 0 of 1
[- ] process > pipeline:summariseReads [ 0%] 0 of 1
executor > local (4)
[bc/6fc768] process > start_ping:pingMessage (1) [ 0%] 0 of 1
executor > local (4)
[bc/6fc768] process > start_ping:pingMessage (1) [ 0%] 0 of 1
[9f/61a8ee] process > pipeline:summariseReads (1) [ 0%] 0 of 1
[- ] process > pipeline:alignReads -
[- ] process > pipeline:coverageCalc -
[- ] process > pipeline:medakaVariants -
[- ] process > pipeline:makeConsensus -
[61/7eeec9] process > pipeline:getVersions [ 0%] 0 of 1
[b9/df411a] process > pipeline:getParams [ 0%] 0 of 1
[- ] process > pipeline:flyeAssembly -
[- ] process > pipeline:makeReport -
[- ] process > output -
[- ] process > end_ping:pingMessage -
Checking fastq input.
Single directory input detected.
Creating env using mamba: /home/User/.nextflow/assets/epi2me-labs/wf-mpx/environment.yaml [cache /MPOX/epi2me_labs_wf-mpx/run220713_Gridion_test/Congo_sample/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01]
executor > local (4)
[bc/6fc768] process > start_ping:pingMessage (1) [200%] 2 of 1, failed: 2 ✘
[9f/61a8ee] process > pipeline:summariseReads (1) [100%] 1 of 1, failed: 1 ✘
[- ] process > pipeline:alignReads -
[- ] process > pipeline:coverageCalc -
[- ] process > pipeline:medakaVariants -
[- ] process > pipeline:makeConsensus -
[61/7eeec9] process > pipeline:getVersions [200%] 2 of 1, failed: 2 ✘
[b9/df411a] process > pipeline:getParams [100%] 1 of 1, failed: 1 ✘
[- ] process > pipeline:flyeAssembly -
[- ] process > pipeline:makeReport -
[- ] process > output -
[- ] process > end_ping:pingMessage -
Checking fastq input.
Single directory input detected.
Creating env using mamba: /home/User/.nextflow/assets/epi2me-labs/wf-mpx/environment.yaml [cache /MPOX/epi2me_labs_wf-mpx/run220713_Gridion_test/Congo_sample/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01]
WARN: Output `set` must define at least two component -- Check process `pipeline:coverageCalc`
Error executing process > 'pipeline:getParams'
Caused by:
Process `pipeline:getParams` terminated with an error exit status (1)
Command executed:
# Output nextflow params object to JSON
echo '{
"help": false,
"fastq": "./input/",
"out_dir": "./output/",
"sample": null,
"sample_sheet": null,
"sanitize_fastq": false,
"wfversion": "v0.0.4",
"threads": 4,
"assembly": true,
"medaka_options": null,
"min_coverage": 20,
"aws_image_prefix": null,
"aws_queue": null,
"report_name": "report",
"disable_ping": false,
"process_label": "wfmpx",
"monochrome_logs": false,
"validate_params": true,
"show_hidden_params": false,
"schema_ignore_params": "show_hidden_params,validate_params,monochrome_logs,aws_queue,aws_image_prefix,wfversion,wf,process_label",
"wf": {
"example_cmd": [
"--fastq test_data/reads.fastq.gz"
]
},
"_reference": "/home/User/.nextflow/assets/epi2me-labs/wf-mpx/./data/references/MT903344.1.fasta",
"_genbank": "/home/User/.nextflow/assets/epi2me-labs/wf-mpx/./data/references/MT903344.1.gb"
}' > params.json
Command exit status:
1
Command output:
(empty)
Command wrapper:
Not a conda environment: /User/MPOX/epi2me_labs_wf-mpx/run220713_Gridion_test/Congo_sample/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01
Work dir:
/MPOX/epi2me_labs_wf-mpx/run220713_Gridion_test/Congo_sample/work/b9/df411ab86b0b5379bb9641760117d8
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh``
I than tried to run the test data set directly from the nextflow environment:
`(nextflow) User:~/.nextflow/assets/epi2me-labs/wf-mpx> nextflow run epi2me-labs/wf-mpx --fastq test_data/fastq/barcode01/ -profile conda
N E X T F L O W ~ version 22.04.0
Launching `https://github.com/epi2me-labs/wf-mpx` [scruffy_perlman] DSL2 - revision: 038b325037 [master]
Core Nextflow options
revision : master
runName : scruffy_perlman
launchDir : /home/User/.nextflow/assets/epi2me-labs/wf-mpx
workDir : /home/User/.nextflow/assets/epi2me-labs/wf-mpx/work
projectDir : /home/User/.nextflow/assets/epi2me-labs/wf-mpx
userName : User
profile : conda
configFiles : /home/User/.nextflow/assets/epi2me-labs/wf-mpx/nextflow.config
Basic Input/Output Options
fastq : test_data/fastq/barcode01/
Meta Data
disable_ping : false
Other parameters
process_label: wfmpx
!! Only displaying parameters that differ from the pipeline defaults !!
------------------------------------------------------
If you use epi2me-labs/wf-mpx for your analysis please cite:
* The nf-core framework
https://doi.org/10.1038/s41587-020-0439-x
executor > local (4)
executor > local (4)
[96/5b1ee3] process > start_ping:pingMessage (1) [100%] 1 of 1, failed: 1 ✘
[- ] process > pipeline:summariseReads (1) -
[- ] process > pipeline:alignReads -
[- ] process > pipeline:coverageCalc -
[- ] process > pipeline:medakaVariants -
[- ] process > pipeline:makeConsensus -
[ce/74df3b] process > pipeline:getVersions [100%] 1 of 1, failed: 1 ✘
[76/eb65b6] process > pipeline:getParams [100%] 1 of 1, failed: 1 ✘
[- ] process > pipeline:flyeAssembly -
[- ] process > pipeline:makeReport -
[- ] process > output -
[- ] process > end_ping:pingMessage -
Checking fastq input.
Single directory input detected.
Creating env using mamba: /home/ad/zrz-beca/.nextflow/assets/epi2me-labs/wf-mpx/environment.yaml [cache /home/User/.nextflow/assets/epi2me-labs/wf-mpx/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01]
WARN: Output `set` must define at least two component -- Check process `pipeline:coverageCalc`
WARN: Unable to fetch attribute for file: /home/User/.nextflow/assets/epi2me-labs/wf-mpx/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01 - Hash is inferred from Git repository commit Id
WARN: Unable to fetch attribute for file: /home/User/.nextflow/assets/epi2me-labs/wf-mpx/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01 - Hash is inferred from Git repository commit Id
WARN: Unable to fetch attribute for file: /home/User/.nextflow/assets/epi2me-labs/wf-mpx/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01 - Hash is inferred from Git repository commit Id
WARN: Unable to fetch attribute for file: /home/User/.nextflow/assets/epi2me-labs/wf-mpx/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01 - Hash is inferred from Git repository commit Id
Error executing process > 'pipeline:getVersions'
Caused by:
Process `pipeline:getVersions` terminated with an error exit status (1)
Command executed:
python -c "import pysam; print(f'pysam,{pysam.__version__}')" >> versions.txt
fastcat --version | sed 's/^/fastcat,/' >> versions.txt
medaka --version | sed 's/ /,/' >> versions.txt
bedtools --version | sed 's/ /,/' >> versions.txt
flye --version | sed 's/^/flye,/' >> versions.txt
bcftools --version | head -1 | sed 's/ /,/' >> versions.txt
Command exit status:
1
Command output:
(empty)
Command wrapper:
Not a conda environment: /home/User/.nextflow/assets/epi2me-labs/wf-mpx/work/conda/epi2melabs-wf-mpx-802521f0eaf1dd7830efb72ec6467b01
Work dir:
/home/User/.nextflow/assets/epi2me-labs/wf-mpx/work/ce/74df3bbe3e6451ad19c0cd6497a302
Tip: you can replicate the issue by changing to the process work dir and entering the command `bash .command.run``
I was hoping, as you are looking into conda issues with the wf you would also be able to help me.
Thanks so much!
Hi @MantaRay87 - I'm not convinced it's the same issue, but please try my suggestions above and see if that works. @antoine4ucsd's problem is related to an issue with conda and not the workflow itself.
Thanks
Matt
(I confirm it is a conda issue for me - works with docker...) thank you Matt.
great to hear @antoine4ucsd - please let me know how you get on with the workflow, if you have any comments or suggestions for improvements please drop me an email, it's relatively new and therefore we are still fine tuning to users needs.
matt dot parker at nanoporetech dot com
Thanks
@MantaRay87 - if you are still having an issue can you please open a new one? I did think after I replied that you could possibly try singularity as the profile - most institutions support it as an alternative to docker!
What happened?
Hello trying to run the workflow on the test mxp fastq. I have a Macbook Pro M2. I would like to use a conda dedicated environnment. I tried to use the yaml file but get conflicts (see log below). I also tried to install epiflow separately and ran the following command:
but it requires docker... do you have an updated yml file to use to create a working env?
thank you!
Operating System
macOS
Workflow Execution
Other (please describe)
Workflow Execution - EPI2ME Labs Versions
No response
Workflow Execution - Execution Profile
No response
Workflow Version
most recent github
Relevant log output