proteinosome
commented
1 year ago Hi @AGI-chandler , thanks for sharing your observations. Looks like Conda is messing up some of those packages. I did not change the environmental YAML file in the new version so I'm not sure what happened there. I typically very much prefer docker/singularity to avoid issues like this but glad to know that you got it working and thanks for sharing those steps.
For the denoise issue, can you check if there's anything inside ".command.err" and ".command.out"? The "/tmp/qiime2-q2cli-err-bp4mht2v.log" file might be on the compute node that runs the job if you're using the cluster mode, but otherwise I've seen in the past QIIME 2 has this weird issue of deleting the tmp file after it fails and asking users to check...
Can you also try running on the test samples just to rule out any weird issue?
Thanks.
AGI-chandler
dcopetti
Hi there, have had a few new issues with latest version and now am stuck on the final issue.
nextflow.config:
``` process { executor = 'slurm' queue = 'defq' } params.dada2_cpu="256" params.vsearch_cpu="256" params.cutadapt_cpu="256" // CPU limit if using local executor process { name = "Local" cpus = 32 } // Set cpus and memory. Process in "main.nf" will // contain label corresponding to one of these unless // the CPUs can be specified via specific parameters // (e.g. --vsearch_cpu) process { withLabel: cpu_def { cpus = 32 memory = 128.GB } withLabel: cpu8 { cpus = 64 memory = 256.GB } withLabel: cpu32 { cpus = 256 memory = 1024.GB } } profiles { standard { params.enable_conda = false params.enable_container=false singularity.enabled = false singularity.automounts = false docker.enabled = false podman.enabled = false shifter.enabled = false charliecloud.enabled = false } conda { conda { useMamba = false conda.enabled = true // Allow longer conda creation timeout createTimeout = '24 h' cacheDir = "$HOME/.conda/envs" } params.enable_conda = true params.enable_container=false singularity.enabled = false singularity.automounts = false docker.enabled = false podman.enabled = false shifter.enabled = false charliecloud.enabled = false } singularity { singularity.enabled = true singularity.autoMounts = true singularity.cacheDir = "$HOME/.conda/envs/singularity" params.enable_container=true docker.enabled = false podman.enabled = false shifter.enabled = false charliecloud.enabled = false } docker { singularity.enabled = false singularity.autoMounts = false docker.enabled = true params.enable_container=true podman.enabled = false shifter.enabled = false charliecloud.enabled = false } } // Generate report report { enabled = true file = "$params.outdir/report.html" overwrite = true } // Timeline timeline { enabled = true file = "$params.outdir/timeline.html" overwrite = true } // DAG dag { enabled = true file = "$params.outdir/dag.html" overwrite = true } ```First Issue: Error executing process > 'pb16S:cutadapt (1)'
Error output:
``` Caused by: Failed to create Conda environment command: conda env create --prefix ~/.conda/envs/qiime2-2022.2-py38-linux-conda-0f9d5d3ab8a678e45d02e511ce6c0160 --file ${pb_16S_nf_base}/env/qiime2-2022.2-py38-linux-conda.yml status : 120 message: ```Second Issue: Error executing process > 'pb16S:merge_sample_manifest' [csvtk: command not found]
Error output:
``` Caused by: Process `pb16S:merge_sample_manifest` terminated with an error exit status (127) Command executed: csvtk concat -t samplefile.txt > merged_sample_file.txt Command exit status: 127 Command output: (empty) Command error: .command.sh: line 2: csvtk: command not found Work dir: ${name}_working/work/73/981c2869a4629afcd01ea635a6ff2b Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named `.command.sh` ```Third Issue: Error executing process > 'pb16S:dada2_denoise (1)'
Full command and parameters:
``` $ alias pb_16S_nf="nextflow run ${pb_16S_nf_base}/main.nf -profile conda" $ mkdir ${name}_working && cd ${name}_working $ pb_16S_nf --input ../${name}_inputs/${name}_samples.tsv --metadata ../${name}_inputs/${name}_metadata.tsv --outdir ../${name}_outputs N E X T F L O W ~ version 23.01.0-edge Launching `${pb_16S_nf_base}/main.nf` [berserk_linnaeus] DSL2 - revision: 4047538644 Only 1 sample. min_asv_sample and min_asv_totalfreq set to 0. Parameters set for pb-16S-nf pipeline for PacBio HiFi 16S ========================================================= Number of samples in samples TSV: 1 Filter input reads above Q: 20 Trim primers with cutadapt: Yes Forward primer: AGRGTTYGATYMTGGCTCAG Reverse primer: AAGTCGTAACAAGGTARCY Minimum amplicon length filtered in DADA2: 1000 Maximum amplicon length filtered in DADA2: 1600 maxEE parameter for DADA2 filterAndTrim: 2 minQ parameter for DADA2 filterAndTrim: 0 Pooling method for DADA2 denoise process: pseudo Minimum number of samples required to keep any ASV: 0 Minimum number of reads required to keep any ASV: 0 Taxonomy sequence database for VSEARCH: ${pb_16S_nf_base}/databases/GTDB_ssu_all_r207.qza Taxonomy annotation database for VSEARCH: ${pb_16S_nf_base}/databases/GTDB_ssu_all_r207.taxonomy.qza Skip Naive Bayes classification: false SILVA database for Naive Bayes classifier: ${pb_16S_nf_base}/databases/silva_nr99_v138.1_wSpecies_train_set.fa.gz GTDB database for Naive Bayes classifier: ${pb_16S_nf_base}/databases/GTDB_bac120_arc53_ssu_r207_fullTaxo.fa.gz RefSeq + RDP database for Naive Bayes classifier: ${pb_16S_nf_base}/databases/RefSeq_16S_6-11-20_RDPv16_fullTaxo.fa.gz VSEARCH maxreject: 100 VSEARCH maxaccept: 100 VSEARCH perc-identity: 0.97 QIIME 2 rarefaction curve sampling depth: null Number of threads specified for cutadapt: 256 Number of threads specified for DADA2: 256 Number of threads specified for VSEARCH: 256 Script location for HTML report generation: ${pb_16S_nf_base}/scripts/visualize_biom.Rmd Container enabled via docker/singularity: false Version of Nextflow pipeline: 0.5 [ab/b27986] process > pb16S:write_log [100%] 1 of 1 ✔ [90/07120c] process > pb16S:QC_fastq (1) [100%] 1 of 1 ✔ [e9/f2bd61] process > pb16S:cutadapt (1) [100%] 1 of 1 ✔ [20/cd7cfa] process > pb16S:QC_fastq_post_trim (1) [100%] 1 of 1 ✔ [0b/6dd519] process > pb16S:collect_QC [100%] 1 of 1 ✔ [ee/440c69] process > pb16S:prepare_qiime2_manifest (1) [100%] 1 of 1 ✔ [0d/e4a74a] process > pb16S:merge_sample_manifest [100%] 1 of 1 ✔ [f7/776596] process > pb16S:import_qiime2 (1) [100%] 1 of 1 ✔ [d9/c83ad6] process > pb16S:demux_summarize (1) [100%] 1 of 1 ✔ [d1/3ac7f3] process > pb16S:dada2_denoise (1) [100%] 1 of 1, failed: 1 ✘ ```Error output:
``` Error executing process > 'pb16S:dada2_denoise (1)' Caused by: Process `pb16S:dada2_denoise (1)` terminated with an error exit status (1) Command executed: # Use custom script that can skip primer trimming mkdir -p dada2_custom_script cp run_dada_ccs.R dada2_custom_script/run_dada_ccs_original.R sed 's/minQ\ =\ 0/minQ=0/g' dada2_custom_script/run_dada_ccs_original.R > dada2_custom_script/run_dada_ccs.R chmod +x dada2_custom_script/run_dada_ccs.R export PATH="./dada2_custom_script:$PATH" which run_dada_ccs.R qiime dada2 denoise-ccs --i-demultiplexed-seqs samples.qza --o-table dada2-ccs_table.qza --o-representative-sequences dada2-ccs_rep.qza --o-denoising-stats dada2-ccs_stats.qza --p-min-len 1000 --p-max-len 1600 --p-max-ee 2 --p-front 'none' --p-adapter 'none' --p-n-threads 256 --p-pooling-method 'pseudo' Command exit status: 1 Command output: dada2_custom_script/run_dada_ccs.R Command error: Plugin error from dada2: An error was encountered while running DADA2 in R (return code -6), please inspect stdout and stderr to learn more. Debug info has been saved to /tmp/qiime2-q2cli-err-bp4mht2v.log Work dir: ${name}_working/work/d1/3ac7f378a25410b1e8b34a1bf06dab Tip: when you have fixed the problem you can continue the execution adding the option `-resume` to the run command line ```