search

epi2me-labs / wf-pore-c

Other
34 stars 9 forks source link

--chunk_size parameter of the run command again, but when I ran it again, the bug appeared #72

Closed wangguiqian closed 3 days ago

wangguiqian commented 2 months ago 
          I followed your suggestion to modify the --chunk_size parameter of the run command again, but when I ran it again, the bug appeared again and the run was automatically interrupted on the 7th day.

I don't know why this happened. Is there any other way to run it? My fastq data is about 80GB. How can I run it in a distributed manner?

Below is the command I ran and the information I just reported： nextflow run /public/home/yqming/Software/wf-pore-c-1.1.0/ --fastq ../raw_data/Pore-C_part1.fq.gz --ref ../Gh_genome.fa --cutter DpnII -c increase_memory.config --threads 35 --paired_end_minimum_distance 100 --paired_end_maximum_distance 200 --chunk_size 40000 --filter_pairs --paired_end --hi_c --out_dir PoreC_part1 -profile singularity -with-singularity ~/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif -with-trace -resume Aug-16 22:08:20.349 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 23.10.1 Aug-16 22:08:20.382 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/public/home/yqming/.nextflow/plugins; core-plugins: nf-amazon@2.1.4,nf-azure@1.3.3,nf-cloudcache@0.3.0,nf-codecommit@0.1.5,nf-console@1.0.6,nf-ga4gh@1.1.0,nf-google@1.8.3,nf-tower@1.6.3,nf-wave@1.0.1 Aug-16 22:08:20.410 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: [] Aug-16 22:08:20.412 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: [] Aug-16 22:08:20.417 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode Aug-16 22:08:20.430 [main] INFO org.pf4j.AbstractPluginManager - No plugins Aug-16 22:08:21.794 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /public/home/yqming/Software/wf-pore-c-1.1.0/nextflow.config Aug-16 22:08:21.815 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /public/home/yqming/Work_plate/Cotton_data/Pore_data/increase_memory.config Aug-16 22:08:21.816 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /public/home/yqming/Software/wf-pore-c-1.1.0/nextflow.config Aug-16 22:08:21.816 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /public/home/yqming/Work_plate/Cotton_data/Pore_data/increase_memory.config Aug-16 22:08:21.835 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: singularity Aug-16 22:08:22.041 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: singularity Aug-16 22:08:22.070 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, conda, singularity, awsbatch, local] Aug-16 22:08:22.081 [main] WARN nextflow.config.ConfigBuilder - It appears you have never run this project before -- Option -resume is ignored Aug-16 22:08:22.108 [main] DEBUG nextflow.config.ConfigBuilder - Enabling execution in Singularity container as requested by command-line option -with-singularity null Aug-16 22:08:22.144 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion Aug-16 22:08:22.179 [main] INFO nextflow.cli.CmdRun - Launching /public/home/yqming/Software/wf-pore-c-1.1.0/main.nf [mad_elion] DSL2 - revision: cc183d633a Aug-16 22:08:22.180 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[] Aug-16 22:08:22.180 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[] Aug-16 22:08:22.194 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /public/home/yqming/.nextflow/secrets/store.json Aug-16 22:08:22.199 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@67332b1e] - activable => nextflow.secret.LocalSecretsProvider@67332b1e Aug-16 22:08:22.295 [main] DEBUG nextflow.Session - Session UUID: a11272e8-7185-4501-81d4-75efc2655b24 Aug-16 22:08:22.296 [main] DEBUG nextflow.Session - Run name: mad_elion Aug-16 22:08:22.297 [main] DEBUG nextflow.Session - Executor pool size: 36 Aug-16 22:08:22.309 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null Aug-16 22:08:22.315 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=108; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false Aug-16 22:08:22.398 [main] DEBUG nextflow.cli.CmdRun - Version: 23.10.1 build 5891 Created: 12-01-2024 22:01 UTC (13-01-2024 06:01 CDT) System: Linux 3.10.0-862.el7.x86_64 Runtime: Groovy 3.0.19 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21 Encoding: UTF-8 (UTF-8) Process: 275840@c05n10 [10.1.5.10] CPUs: 36 - Mem: 180 GB (179.5 GB) - Swap: 0 (0) Aug-16 22:08:22.440 [main] DEBUG nextflow.Session - Work-dir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work [gpfs] Aug-16 22:08:22.511 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[] Aug-16 22:08:22.525 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory Aug-16 22:08:22.580 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory Aug-16 22:08:22.597 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 37; maxThreads: 1000 Aug-16 22:08:22.770 [main] DEBUG nextflow.Session - Session start Aug-16 22:08:22.779 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /public/home/yqming/Work_plate/Cotton_data/Pore_data/PoreC_part1/execution/trace.txt Aug-16 22:08:22.800 [main] DEBUG nextflow.Session - Using default localLib path: /public/home/yqming/Software/wf-pore-c-1.1.0/lib Aug-16 22:08:22.863 [main] DEBUG nextflow.Session - Adding to the classpath library: /public/home/yqming/Software/wf-pore-c-1.1.0/lib Aug-16 22:08:22.864 [main] DEBUG nextflow.Session - Adding to the classpath library: /public/home/yqming/Software/wf-pore-c-1.1.0/lib/nfcore_external_javadeps.jar Aug-16 22:08:25.385 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution Aug-16 22:08:27.387 [main] INFO nextflow.Nextflow - |||||||||| ____ _ __ _ |||||||||| | _| _ | | \/ | __| | | | | ||||| | | | |) | | ) | |\/| | _| ___| |/ ` | ' \/ | ||||| | |_| /| | / /| | | | |_|| | (| | |) _ \ |||||||||| |____|_| |_|___|| ||| ||\,|._/|/ |||||||||| wf-pore-c v1.1.0 -------------------------------------------------------------------------------- Core Nextflow options runName : mad_elion containerEngine : singularity container : [withLabel:wfporec:/public/home/yqming/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif, withLabel:pore_c_py:/public/home/yqming/Software/wf-pore-c-1.1.0/ontresearch-pore-c-py-shad5608368dcce93646de5559f2f391005408da6c2.sif, withLabel:wf_common:/public/home/yqming/Software/wf-pore-c-1.1.0/wf-common_sha1c5febff9f75143710826498b093d9769a5edbb9.sif, default:~/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif] launchDir : /public/home/yqming/Work_plate/Cotton_data/Pore_data workDir : /public/home/yqming/Work_plate/Cotton_data/Pore_data/work projectDir : /public/home/yqming/Software/wf-pore-c-1.1.0 userName : yqming profile : singularity configFiles : /public/home/yqming/Software/wf-pore-c-1.1.0/nextflow.config, /public/home/yqming/Work_plate/Cotton_data/Pore_data/increase_memory.config

Input Options fastq : ../raw_data/Pore-C_part1.fq.gz ref : ../Gh_genome.fa cutter : DpnII

Output Options out_dir : PoreC_part1 hi_c : true

Advanced Options chunk_size : 40000 threads : 35

Paired-end BAM Options paired_end : true filter_pairs : true paired_end_minimum_distance: 100 paired_end_maximum_distance: 200

!! Only displaying parameters that differ from the pipeline defaults !! -------------------------------------------------------------------------------- If you use epi2me-labs/wf-pore-c for your analysis please cite:

_This is the error message:_ Aug-20 12:14:22.681 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:19:21.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:19:22.732 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:24:21.670 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:24:22.785 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:29:21.685 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:29:22.842 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:34:21.784 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:34:22.916 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:39:21.850 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:39:22.961 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:44:21.892 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:44:23.010 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:49:21.971 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:49:23.164 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:49:37.792 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: COMPLETED; exit: 127; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:49:37.837 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Aug-20 12:49:37.837 [Task submitter] INFO nextflow.Session - [a5/65a3c8] Submitted process > POREC:pair_stats_report (1) Aug-20 12:49:37.841 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=POREC:prepare_hic (1); work-dir=/public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6 error [nextflow.exception.ProcessFailedException]: Process POREC:prepare_hic (1) terminated with an error exit status (127) Aug-20 12:49:38.100 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'POREC:prepare_hic (1)'

Caused by: Process POREC:prepare_hic (1) terminated with an error exit status (127)

Command executed:

cut -f1,2 fasta.fai > sizes.genome pairtools flip input.pairs.gz -c sizes.genome > flipped.pairs.tmp pairtools sort flipped.pairs.tmp > sorted.pairs.tmp pairtools dedup --chunksize 100000 sorted.pairs.tmp > dedup.pairs.tmp java -jar /home/epi2melabs/juicer_tools_1.22.01.jar pre dedup.pairs.tmp "null.hic" sizes.genome rm -rf "*.pairs.tmp"

Command exit status: 127

Command output: (empty)

Command error: .command.sh: line 6: java: command not found

Work dir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out Aug-20 12:49:38.121 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process POREC:prepare_hic (1) terminated with an error exit status (127) Aug-20 12:49:38.269 [main] DEBUG nextflow.Session - Session await > all processes finished Aug-20 12:49:38.269 [main] DEBUG nextflow.Session - Session await > all barriers passed Aug-20 12:49:38.283 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop Aug-20 12:49:38.301 [main] WARN n.processor.TaskPollingMonitor - Killing running tasks (1) Aug-20 12:49:38.318 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1819; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=1; runningCount=-1; retriesCount=0; abortedCount=1; succeedDuration=108d 3h 59m 45s; failedDuration=2d 5h 24m 16s; cachedDuration=0ms;loadCpus=-30; loadMemory=0; peakRunning=2; peakCpus=35; peakMemory=122 GB; ] Aug-20 12:49:38.319 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file Aug-20 12:49:38.410 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report Aug-20 12:49:39.809 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline Aug-20 12:49:40.126 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Aug-20 12:49:40.150 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Originally posted by @wangguiqian in https://github.com/epi2me-labs/wf-pore-c/issues/70#issuecomment-2310320844

wangguiqian commented 2 months ago

wang.txt this my log file,please help for me to check .

sarahjeeeze commented 2 months ago

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion?

wangguiqian commented 2 months ago

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion? I have tested your data: wget https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-pore-c/wf-pore-c-demo.tar.gz tar -xzvf wf-pore-c-demo.tar.gz After getting the data, I ran the command offline. The command is as follows: export NXF_SINGULARITY_CACHEDIR=~/Software/wf-pore-c-1.1.0 export NXF_OFFLINE=true nextflow run ~/Software/wf-pore-c-1.1.0/ --bam 'wf-pore-c-demo/porec_test.concatemers.bam' --threads 12 --chunk_size 100 --cutter 'NlaIII' --hi_c --mcool --paired_end --paired_end_maximum_distance 200 --paired_end_minimum_distance 100 --phased_vcf 'wf-pore-c-demo/porec_test.phased_variants.vcf.gz' --ref 'wf-pore-c-demo/porec_test.fasta' --vcf 'wf-pore-c-demo/porec_test.phased_variants.vcf.gz' --out_dir PoreC_test -profile singularity -with-singularity '~/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif' -with-trace

But I still get an error after running it. What is the problem in this case? [15/348233] process > POREC:getVersions [100%] 1 of 1 ✔
[95/f91d93] process > POREC:getParams [100%] 1 of 1 ✔
[b7/db5c87] process > POREC:makeReport [100%] 1 of 1 ✔
[fa/33dacf] process > POREC:prepare_hic (1) [100%] 1 of 1, failed: 1 ✘
[25/b07bdb] process > POREC:get_filtered_out_bam (1) [100%] 1 of 1 ✔
[3c/dce576] process > output (1) [100%] 1 of 1 ✔
ERROR ~ Error executing process > 'POREC:prepare_hic (1)'

Caused by:
Process POREC:prepare_hic (1) terminated with an error exit status (127)

Command executed:

cut -f1,2 fasta.fai > sizes.genome
pairtools flip input.pairs.gz -c sizes.genome > flipped.pairs.tmp
pairtools sort flipped.pairs.tmp > sorted.pairs.tmp
pairtools dedup --chunksize 100000 sorted.pairs.tmp > dedup.pairs.tmp
java -jar /home/epi2melabs/juicer_tools_1.22.01.jar pre dedup.pairs.tmp "null.hic" sizes.genome
rm -rf "*.pairs.tmp"

Command exit status:
127

Command output:
(empty)

Command error:
.command.sh: line 6: java: command not found

Work dir:
/public/home/yqming/Work_plate/work/fa/33dacfe5877c71c4ddced4472c7264

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

This my log file ,please help for to check : pore_c.log

wangguiqian commented 2 months ago

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion?

pairtools dedup --chunksize 100000 sorted.pairs.tmp > dedup.pairs.tmp java -jar /home/epi2melabs/juicer_tools_1.22.01.jar pre dedup.pairs.tmp "null.hic" sizes.genome

But I saw the error message, it seems that there is an error in the pre command of juicer to build HiC. What is the problem? Could you please share a complete pore-C log file for me to see? I will try to run it step by step.

wangguiqian commented 2 months ago

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion?

I got the eorrs,can you help to solve?

sarahjeeeze commented 2 months ago

As you are running offline and there are multiple containers, you need to define each of the containers in the config with process.container = 'file:///path/to/singularity.img' see https://www.nextflow.io/docs/latest/container.html#id24

sarahjeeeze commented 1 month ago

Hi, please try the latest version 1.3.0 - I see you are pinned to 1.1.0 which did not allocate enough memory for some of the processes. If it still fails can you send me the nextflow.log file please.

wangguiqian commented 1 month ago

@sarahjeeeze Thank you for your help. I downloaded the 1.3.0 version of wf-pore-c according to your suggestion and ran it again, but I encountered an error at the beginning. The command I ran was as follows: nextflow run ~/Software/wf-pore-c-1.3.0/ --fastq ../raw_data/Pore-C_part1.fq.gz --ref ../Gh_genome.fa --cutter DpnII -c increase_memory.config --threads 30 --paired_end_minimum_distance 100 --paired_end_maximum_distance 200 --chunk_size 40000 --filter_pairs --paired_end --hi_c --out_dir PoreC_part1 -profile singularity -with-singularity '~/Software/wf-pore-c-1.3.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif' -with-trace -resume

And I used 30 cores to run this time, and my original fastq data was about 60G. Please look my data size: QQ_1729359835400

Below is the log file I ran. Please help me check it. Thank you. However, I read the log file later and it said that there was a problem with the fastcat command. It showed that the parameter setting of the fastcat command was wrong. I will share my log file with you. Please help me check it. Thank you. nextflow.log

wangguiqian commented 1 month ago

QQ_1729360058693

Command executed:

mkdir fastcat_stats mkdir fastq_chunks

Save file as compressed fastq

fastcat -s Pore-C_part1 -f fastcat_stats/per-file-stats.tsv -i fastcat_stats/per-file-runids.tsv -l fastcat_stats/p bgzip -@ 4 > fastq_chunks/seqs.fastq.gz else split -l null -d --additional-suffix=.fastq.gz --filter='bgzip -@ 4 > $FILE' - fastqchunks/seqs; fi

mv histograms/* fastcat_stats

get n_seqs from per-file stats - need to sum them up

awk 'NR==1{for (i=1; i<=NF; i++) {ix[$i] = i}} NR>1 {c+=$ix["n_seqs"]} END{print c}' fastcat_stats/per-file-stats.tsv > fastcat_stats/n_seq

get unique run IDs (we add -F '\t' as awk uses any stretch of whitespace

as field delimiter per default and thus ignores empty columns)

awk -F '\t' '
NR==1 {for (i=1; i<=NF; i++) {ix[$i] = i}}

only print run_id if present

  NR>1 && $ix["run_id"] != "" {print $ix["run_id"]}

' fastcat_stats/per-file-runids.tsv | sort | uniq > fastcat_stats/run_ids

get unique basecall models

get unique basecall models

awk -F '\t' '
NR==1 {for (i=1; i<=NF; i++) {ix[$i] = i}}

only print basecall model if present

  NR>1 && $ix["basecaller"] != "" {print $ix["basecaller"]}

' fastcat_stats/per-file-basecallers.tsv | sort | uniq > fastcat_stats/basecallers

Command exit status:
64

Command output:
(empty)

Command error:
fastcat: invalid option -- 'i'
Try fastcat --help' orfastcat --usage' for more information.

Work dir:
/public/home/yqming/Work_plate/Cotton_data/Pore_data/work/d4/b299681604472b84d8529ec7fb42a7

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out
Oct-20 00:53:24.848 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process POREC:fastcat (1) terminated with an error exit statu Oct-20 00:53:24.862 [Task monitor] DEBUG nextflow.Session - The following nodes are still active:
[process] POREC:digest_align_annotate
status=ACTIVE
port 0: (queue) OPEN ; channel: -
port 1: (cntrl) - ; channel: $

[process] POREC:haplotag_alignments
status=ACTIVE
port 0: (queue) OPEN ; channel: -
port 1: (cntrl) - ; channel: $

[process] POREC:merge_coordsorted_bams
status=ACTIVE
port 0: (queue) OPEN ; channel: -
port 1: (cntrl) - ; channel: $

[process] POREC:merge_namesorted_bams

wangguiqian commented 1 month ago

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion?

I personally think that there was an error in the command when you wrote this new version of the package?

sarahjeeeze commented 3 days ago

Responding on the other issue linked