Closed MAXINELSX closed 3 months ago
Hi,
I think this may be the problem:
containerOptions = '-v /home:/home -v /autofs/tong1/LSX/lishuxian/lishuxian/test_data:/data -v /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output:/output'
With that instruction, it means that /autofs/tong1/LSX/lishuxian/lishuxian/test_data
path on your server should be accessed by the Docker container in /data
, and your paths for --fastq_files
and --resultsDir
should contain /data
in place of /autofs/tong1/LSX/lishuxian/lishuxian/test_data/
.
So, you should either adjust the paths specified in your variables accordingly, or mount the directories of interest in a directory accessible to Docker with exactly the same path (as you did for -v /home:/home
).
I hope this solves the issue!
SM
Hi,
Thank you for your quick response and guidance.
After following your advice, I modified the containerOptions in the testUMInator.conf file to include the necessary volume mappings. Here is the updated line: containerOptions = '-v /home:/home -v /autofs/tong1/LSX/lishuxian/lishuxian/test_data:/autofs/tong1/LSX/lishuxian/lishuxian/test_data' Your assistance was invaluable in resolving the issues I was encountering, and I greatly appreciate your help!
Best regards, Max
Hi,
Thank you for your quick response and guidance.
After following your advice, I modified the containerOptions in the testUMInator.conf file to include the necessary volume mappings. Here is the updated line: containerOptions = '-v /home:/home -v /autofs/tong1/LSX/lishuxian/lishuxian/test_data:/autofs/tong1/LSX/lishuxian/lishuxian/test_data' Your assistance was invaluable in resolving the issues I was encountering, and I greatly appreciate your help!
Best regards, Max
sorry for bothering you again, there is a new issue;
(env_nf) lishuxian@tong2:~$ nextflow -c ~/UMInator/2UMInator.conf run ~/UMInator/UMInator.nf -bg --FW_adapter="CAAGCAGAAGACGGCATACGAGAT" --RV_adapter="AATGATACGGCGACCACCGAGATC" --FW_primer="AGRGTTYGATYMTGGCTCAG" --RV_primer="CGACATCGAGGTGCCAAAC" --fastq_files="/autofs/tong1/LSX/lishuxian/lishuxian/test_data/test_reads.fq" --results_dir=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output --minQ=7 --minLen=3000 --maxLen=6000 --min_UMI_freq=30 --maxF=10 --scripts_dir="$HOME/UMInator/scripts" --medaka_model=r941_min_high_g330 -profile docker
(env_nf) lishuxian@tong2:~$ N E X T F L O W ~ version 23.10.1
Launching /home/lishuxian/UMInator/UMInator.nf
[sharp_bose] DSL2 - revision: b6a013ff48
[8a/f8d88a] Submitted process > readsFiltering (1)
[77/e84941] Submitted process > candidateUMIsExtraction (1)
[07/1d6e78] Submitted process > candidateUMIsFiltering (1)
[4b/fcc555] Submitted process > readsUMIsAssignment (1)
[37/a498a2] Submitted process > draftConsensusCalling (1)
[42/bac2bc] Submitted process > draftConsensusCalling (6)
[85/760a18] Submitted process > draftConsensusCalling (3)
[69/2dbcc1] Submitted process > draftConsensusCalling (7)
[b3/a6f9c3] Submitted process > draftConsensusCalling (5)
[b8/c1f0f1] Submitted process > draftConsensusCalling (2)
[00/a74fb9] Submitted process > draftConsensusCalling (9)
[2e/cdf49e] Submitted process > draftConsensusCalling (8)
[a6/09014c] Submitted process > draftConsensusCalling (4)
[ce/f1a1db] Submitted process > consensusPolishing (1)
[5a/d68e00] Submitted process > consensusPolishing (2)
[8a/ac99d4] Submitted process > consensusPolishing (3)
[18/7edc5a] Submitted process > consensusPolishing (4)
[05/1cc2a5] Submitted process > consensusPolishing (5)
[f4/cfb215] Submitted process > consensusPolishing (6)
[1a/31949e] Submitted process > consensusPolishing (7)
[47/48b36c] Submitted process > consensusPolishing (8)
[bc/0aaf9c] Submitted process > consensusPolishing (9)
[de/4f1e06] Submitted process > QC (1)
[7f/63b9e0] Submitted process > primersTrimming (1)
ERROR ~ Error executing process > 'primersTrimming (1)'
Caused by:
Process primersTrimming (1)
terminated with an error exit status (1)
Command executed:
mkdir -p /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/primersTrimming mkdir -p /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/primersTrimming/test_reads
polished_consensus=$(find /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/consensusPolishing/test_reads | grep "_polished_consensus.fasta") cat $polished_consensus > /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/primersTrimming/test_reads/test_reads_consensus_polished.fasta
FW_primer_R=$(echo -e ">tmp\n" AGRGTTYGATYMTGGCTCAG | seqtk seq -r - | grep -v "^>" | tr -d ' ') RV_primer_R=$(echo -e ">tmp\n" CGACATCGAGGTGCCAAAC | seqtk seq -r - | grep -v "^>" | tr -d ' ')
if [[ -f "/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/primersTrimming/test_reads/test_reads_consensus_polished.fasta" ]]; then cutadapt -j 4 -e 0 --discard-untrimmed --match-read-wildcards -g AGRGTTYGATYMTGGCTCAG -g $RV_primer_R -a CGACATCGAGGTGCCAAAC -a $FW_primer_R -m 3000 -M 6000 -o /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/primersTrimming/test_reads/test_reads_consensus_polished_primersTrimmed.fasta /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/primersTrimming/test_reads/test_reads_consensus_polished.fasta fi
Command exit status: 1
Command output: (empty)
Command error: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
Work dir: /home/lishuxian/work/7f/63b9e065568c5aebebe840fcfa145a The code is the same as nextflow -c ~/UMInator/UMInator.conf run ~/UMInator/UMInator.nf -bg --FW_adapter="CAAGCAGAAGACGGCATACGAGAT" --RV_adapter="AATGATACGGCGACCACCGAGATC" --FW_primer="AGRGTTYGATYMTGGCTCAG" --RV_primer="CGACATCGAGGTGCCAAAC" --fastq_files="/autofs/tong1/LSX/lishuxian/lishuxian/test_data/test_reads.fq" --results_dir=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output --minQ=7 --minLen=3000 --maxLen=6000 --min_UMI_freq=30 --maxF=10 --scripts_dir="$HOME/UMInator/scripts" --medaka_model=r941_min_high_g330 -profile docker and I only changed the 178 line to containerOptions = '-v /home:/home -v /autofs/tong1/LSX/lishuxian/lishuxian/test_data:/autofs/tong1/LSX/lishuxian/lishuxian/test_data'
So sorry for bothering you again.
Hi, please check whether the previous processes completed successfully. In particular, please check if you have draft and polished consensus sequences files and, if yes, what they contain (you can run assembly-stats for that). SM
stats for umi9/umi9_polished_consensus_tmp.fasta sum = 4559, n = 1, ave = 4559.00, largest = 4559 N50 = 4559, n = 1 N60 = 4559, n = 1 N70 = 4559, n = 1 N80 = 4559, n = 1 N90 = 4559, n = 1 N100 = 4559, n = 1 N_count = 0 Gaps = 0
Perfect, it seems only the last primersTrimming process had some issues.
Does the file ${params.results_dir}/primersTrimming/${sample}/${sample}_consensus_polished.fasta
or at least the folder exist?
SM
unfortunately, the file or folder doesn't exist(env_nf) lishuxian@tong2:/autofs/tong1/LSX/lishuxian/lishuxian/test_data/2output/primersTrimming/test_reads$ ll total 0 drwxr-xr-x 2 root root 10 Apr 3 20:39 ./ drwxr-xr-x 3 root root 32 Apr 3 20:39 ../
Difficult to say, I don't see any specific errors. Please try removing the output folder and rerunning it again. SM
I have tried this with changing the --results_dir=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output to --results_dir=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/2output --results_dir=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/3output I find all three output have the same problem
Since this is the last step; I can manually cut adapter. with cutadapt cutadapt -j 4 -e 0 --discard-untrimmed --match-read-wildcards \ -g CAAGCAGAAGACGGCATACGAGAT -a AATGATACGGCGACCACCGAGATC \ -g AGRGTTYGATYMTGGCTCAG -a CGACATCGAGGTGCCAAAC \ -m 3000 -M 6000 -o test_reads_consensus_polished.fasta all.fasta
I think you forgot setting --min_UMI_freq=10
, could this be the case? However, this should have resulted in no consensus sequences being produced at all.
In any case yes, you can manually concatenate all polished consensus sequences and run the command you reported.
At the moment I'm not able to solve the issue, as I don't see any specific errors (just check the requested resources for that process are available).
SM
Still have this:
Caused by:
Process primersTrimming (1)
terminated with an error exit status (1)
But I can do this manually!
Thank you so much for your professional and prompt response. I appreciate it a lot, it has been already very helpful, thank you so much!
Ok, best! SM
Sorry to bother you; I am using this (env_nf) lishuxian@tong2:~/UMInator$ nextflow -c ~/UMInator/testUMInator.conf run ~/UMInator/UMInator.nf -bg --FW_adapter="CAAGCAGAAGACGGCATACGAGAT" --RV_adapter="AATGATACGGCGACCACCGAGATC" --FW_primer="AGRGTTYGATYMTGGCTCAG" --RV_primer="CGACATCGAGGTGCCAAAC" --fastq_files="/autofs/tong1/LSX/lishuxian/lishuxian/test_data/test_reads.fq" --results_dir=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output --minQ=7 --minLen=3000 --maxLen=6000 --min_UMI_freq=30 --maxF=10 --scripts_dir="$HOME/UMInator/scripts" --medaka_model=r941_min_high_g330 -profile docker -w /tmp/nextflow_run and I have change the testUMInator.conf line 178 to containerOptions = '-v /home:/home -v /autofs/tong1/LSX/lishuxian/lishuxian/test_data:/data -v /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output:/output' according to the former issue in this and it doesn't work... the detail showed (env_nf) lishuxian@tong2:~/UMInator$ nextflow -c ~/UMInator/testUMInator.conf run ~/UMInator/UMInator.nf -bg --FW_adapter="CAAGCAGAAGACGGCATACGAGAT" --RV_adapter="AATGATACGGCGACCACCGAGATC" --FW_primer="AGRGTTYGATYMTGGCTCAG" --RV_primer="CGACATCGAGGTGCCAAAC" --fastq_files="/autofs/tong1/LSX/lishuxian/lishuxian/test_data/test_reads.fq" --results_dir=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output --minQ=7 --minLen=3000 --maxLen=6000 --min_UMI_freq=30 --maxF=10 --scripts_dir="$HOME/UMInator/scripts" --medaka_model=r941_min_high_g330 -profile docker -w /tmp/nextflow_run
(env_nf) lishuxian@tong2:~/UMInator$ (env_nf) lishuxian@tong2:~/UMInator$ N E X T F L O W ~ version 23.10.1 Launching
/home/lishuxian/UMInator/UMInator.nf
[golden_wescoff] DSL2 - revision: b6a013ff48 [69/f42a71] Submitted process > readsFiltering (1) [fc/2a7749] Submitted process > candidateUMIsExtraction (1) ERROR ~ Error executing process > 'candidateUMIsExtraction (1)'Caused by: Process
candidateUMIsExtraction (1)
terminated with an error exit status (1)Command executed:
mkdir -p /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output mkdir -p /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction mkdir -p /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads
obtain reverse complement of primers and adapters sequences
FW_primer_R=$(echo -e ">tmp\n" AGRGTTYGATYMTGGCTCAG | seqtk seq -r - | grep -v "^>" | tr -d ' ') RV_primer_R=$(echo -e ">tmp\n" CGACATCGAGGTGCCAAAC | seqtk seq -r - | grep -v "^>" | tr -d ' ') FW_adapter_R=$(echo -e ">tmp\n" CAAGCAGAAGACGGCATACGAGAT | seqtk seq -r - | grep -v "^>" | tr -d ' ') RV_adapter_R=$(echo -e ">tmp\n" AATGATACGGCGACCACCGAGATC | seqtk seq -r - | grep -v "^>" | tr -d ' ')
double UMI design
obtain first and last bases of reads
READS_START=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/first_200bp.fastq READS_END=/autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/last_200bp.fastq seqtk trimfq -L 200 /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/readsFiltering/test_reads/test_reads_filtered.fastq > $READS_START seqtk seq -r /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/readsFiltering/test_reads/test_reads_filtered.fastq | seqtk trimfq -L 200 - | seqtk seq -r - > $READS_END
search candidate UMIs with exact length between adapters and primers
cutadapt -j 1 -e 0.2 -O 11 -m 18 -M 18 --discard-untrimmed --match-read-wildcards -g CAAGCAGAAGACGGCATACGAGAT...AGRGTTYGATYMTGGCTCAG -g AATGATACGGCGACCACCGAGATC...CGACATCGAGGTGCCAAAC -G $RV_primer_R...$RV_adapter_R -G $FW_primer_R...$FW_adapter_R -o /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part1_db_tmp1.fastq -p /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part2_db_tmp1.fastq $READS_START $READS_END
collapse the 5' and 3' end UMIs of each read
paste -d "" <( sed -n '1~4s/^@/>/p;2~4p' /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part1_db_tmp1.fastq ) <( sed -n '1~4s/^@/>/p;2~4p' /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part2_db_tmp1.fastq ) | cut -d " " -f1 > /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_db_tmp1.fasta
search candidate UMIs with approximate length between adapters and primers
cutadapt -j 1 -e 0.2 -O 11 -m 18 -l 18 --discard-untrimmed --match-read-wildcards -g CAAGCAGAAGACGGCATACGAGAT -g AATGATACGGCGACCACCGAGATC -G $RV_primer_R -G $FW_primer_R -o /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part1_candidates.fastq -p /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part2_candidates.fastq $READS_START $READS_END
collapse the 5' and 3' end UMIs of each read
paste -d "" <( sed -n '1~4s/^@/>/p;2~4p' /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part1_candidates.fastq ) <( sed -n '1~4s/^@/>/p;2~4p' /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_part2_candidates.fastq ) | cut -d " " -f1 > /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_candidates.fastq
convert UMI candidates to fasta
seqtk seq -A /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_candidates.fastq > /autofs/tong1/LSX/lishuxian/lishuxian/test_data/output/candidateUMIsExtraction/test_reads/UMI_candidates.fasta
Command exit status: 1
Command output: (empty)
Command error: WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap. [E::stk_trimfq] failed to open the input file/stream.
Work dir: /tmp/nextflow_run/fc/2a774916efa78a25afa295547fc27c
Tip: you can try to figure out what's wrong by changing to the process work dir and showing the script file named
.command.sh
-- Check '.nextflow.log' file for details
and the nextflow.log
(env_nf) lishuxian@tong2:~/UMInator$ tail .nextflow.log Apr-03 17:40:43.327 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process
candidateUMIsExtraction (1)
terminated with an error exit status (1) Apr-03 17:40:43.362 [main] DEBUG nextflow.Session - Session await > all processes finished Apr-03 17:40:43.363 [Task monitor] DEBUG nextflow.Session - The following nodes are still active: [process] primersTrimmingApr-03 17:40:43.372 [main] DEBUG nextflow.Session - Session await > all barriers passed Apr-03 17:40:43.373 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop Apr-03 17:40:43.379 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=1.1s; failedDuration=624ms; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=1; peakCpus=1; peakMemory=10 GB; ] Apr-03 17:40:43.386 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Apr-03 17:40:43.397 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye
Looking forward to your reply; massive thanks