--chunk_size parameter of the run command again, but when I ran it again, the bug appeared

[wangguiqian]

          I followed your suggestion to modify the --chunk_size parameter of the run command again, but when I ran it again, the bug appeared again and the run was automatically interrupted on the 7th day.

I don't know why this happened. Is there any other way to run it? My fastq data is about 80GB. How can I run it in a distributed manner?

Below is the command I ran and the information I just reported： nextflow run /public/home/yqming/Software/wf-pore-c-1.1.0/ --fastq ../raw_data/Pore-C_part1.fq.gz --ref ../Gh_genome.fa --cutter DpnII -c increase_memory.config --threads 35 --paired_end_minimum_distance 100 --paired_end_maximum_distance 200 --chunk_size 40000 --filter_pairs --paired_end --hi_c --out_dir PoreC_part1 -profile singularity -with-singularity ~/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif -with-trace -resume Aug-16 22:08:20.349 [main] INFO nextflow.cli.CmdRun - N E X T F L O W ~ version 23.10.1 Aug-16 22:08:20.382 [main] DEBUG nextflow.plugin.PluginsFacade - Setting up plugin manager > mode=prod; embedded=false; plugins-dir=/public/home/yqming/.nextflow/plugins; core-plugins: nf-amazon@2.1.4,nf-azure@1.3.3,nf-cloudcache@0.3.0,nf-codecommit@0.1.5,nf-console@1.0.6,nf-ga4gh@1.1.0,nf-google@1.8.3,nf-tower@1.6.3,nf-wave@1.0.1 Aug-16 22:08:20.410 [main] INFO o.pf4j.DefaultPluginStatusProvider - Enabled plugins: [] Aug-16 22:08:20.412 [main] INFO o.pf4j.DefaultPluginStatusProvider - Disabled plugins: [] Aug-16 22:08:20.417 [main] INFO org.pf4j.DefaultPluginManager - PF4J version 3.4.1 in 'deployment' mode Aug-16 22:08:20.430 [main] INFO org.pf4j.AbstractPluginManager - No plugins Aug-16 22:08:21.794 [main] DEBUG nextflow.config.ConfigBuilder - Found config base: /public/home/yqming/Software/wf-pore-c-1.1.0/nextflow.config Aug-16 22:08:21.815 [main] DEBUG nextflow.config.ConfigBuilder - User config file: /public/home/yqming/Work_plate/Cotton_data/Pore_data/increase_memory.config Aug-16 22:08:21.816 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /public/home/yqming/Software/wf-pore-c-1.1.0/nextflow.config Aug-16 22:08:21.816 [main] DEBUG nextflow.config.ConfigBuilder - Parsing config file: /public/home/yqming/Work_plate/Cotton_data/Pore_data/increase_memory.config Aug-16 22:08:21.835 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: singularity Aug-16 22:08:22.041 [main] DEBUG nextflow.config.ConfigBuilder - Applying config profile: singularity Aug-16 22:08:22.070 [main] DEBUG nextflow.config.ConfigBuilder - Available config profiles: [standard, conda, singularity, awsbatch, local] Aug-16 22:08:22.081 [main] WARN nextflow.config.ConfigBuilder - It appears you have never run this project before -- Option -resume is ignored Aug-16 22:08:22.108 [main] DEBUG nextflow.config.ConfigBuilder - Enabling execution in Singularity container as requested by command-line option -with-singularity null Aug-16 22:08:22.144 [main] DEBUG nextflow.cli.CmdRun - Applied DSL=2 from script declararion Aug-16 22:08:22.179 [main] INFO nextflow.cli.CmdRun - Launching /public/home/yqming/Software/wf-pore-c-1.1.0/main.nf [mad_elion] DSL2 - revision: cc183d633a Aug-16 22:08:22.180 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins default=[] Aug-16 22:08:22.180 [main] DEBUG nextflow.plugin.PluginsFacade - Plugins resolved requirement=[] Aug-16 22:08:22.194 [main] DEBUG n.secret.LocalSecretsProvider - Secrets store: /public/home/yqming/.nextflow/secrets/store.json Aug-16 22:08:22.199 [main] DEBUG nextflow.secret.SecretsLoader - Discovered secrets providers: [nextflow.secret.LocalSecretsProvider@67332b1e] - activable => nextflow.secret.LocalSecretsProvider@67332b1e Aug-16 22:08:22.295 [main] DEBUG nextflow.Session - Session UUID: a11272e8-7185-4501-81d4-75efc2655b24 Aug-16 22:08:22.296 [main] DEBUG nextflow.Session - Run name: mad_elion Aug-16 22:08:22.297 [main] DEBUG nextflow.Session - Executor pool size: 36 Aug-16 22:08:22.309 [main] DEBUG nextflow.file.FilePorter - File porter settings maxRetries=3; maxTransfers=50; pollTimeout=null Aug-16 22:08:22.315 [main] DEBUG nextflow.util.ThreadPoolBuilder - Creating thread pool 'FileTransfer' minSize=10; maxSize=108; workQueue=LinkedBlockingQueue[10000]; allowCoreThreadTimeout=false Aug-16 22:08:22.398 [main] DEBUG nextflow.cli.CmdRun - Version: 23.10.1 build 5891 Created: 12-01-2024 22:01 UTC (13-01-2024 06:01 CDT) System: Linux 3.10.0-862.el7.x86_64 Runtime: Groovy 3.0.19 on OpenJDK 64-Bit Server VM 11.0.13+7-b1751.21 Encoding: UTF-8 (UTF-8) Process: 275840@c05n10 [10.1.5.10] CPUs: 36 - Mem: 180 GB (179.5 GB) - Swap: 0 (0) Aug-16 22:08:22.440 [main] DEBUG nextflow.Session - Work-dir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work [gpfs] Aug-16 22:08:22.511 [main] DEBUG nextflow.executor.ExecutorFactory - Extension executors providers=[] Aug-16 22:08:22.525 [main] DEBUG nextflow.Session - Observer factory: DefaultObserverFactory Aug-16 22:08:22.580 [main] DEBUG nextflow.cache.CacheFactory - Using Nextflow cache factory: nextflow.cache.DefaultCacheFactory Aug-16 22:08:22.597 [main] DEBUG nextflow.util.CustomThreadPool - Creating default thread pool > poolSize: 37; maxThreads: 1000 Aug-16 22:08:22.770 [main] DEBUG nextflow.Session - Session start Aug-16 22:08:22.779 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow started -- trace file: /public/home/yqming/Work_plate/Cotton_data/Pore_data/PoreC_part1/execution/trace.txt Aug-16 22:08:22.800 [main] DEBUG nextflow.Session - Using default localLib path: /public/home/yqming/Software/wf-pore-c-1.1.0/lib Aug-16 22:08:22.863 [main] DEBUG nextflow.Session - Adding to the classpath library: /public/home/yqming/Software/wf-pore-c-1.1.0/lib Aug-16 22:08:22.864 [main] DEBUG nextflow.Session - Adding to the classpath library: /public/home/yqming/Software/wf-pore-c-1.1.0/lib/nfcore_external_javadeps.jar Aug-16 22:08:25.385 [main] DEBUG nextflow.script.ScriptRunner - > Launching execution Aug-16 22:08:27.387 [main] INFO nextflow.Nextflow - |||||||||| ____ _ __ _ |||||||||| | _| _ | | \/ | __| | | | | ||||| | | | |) | | ) | |\/| | _| ___| |/ ` | ' \/ | ||||| | |_| /| | / /| | | | |_|| | (| | |) _ \ |||||||||| |____|_| |_|___|| ||| ||\,|._/|/ |||||||||| wf-pore-c v1.1.0 -------------------------------------------------------------------------------- Core Nextflow options runName : mad_elion containerEngine : singularity container : [withLabel:wfporec:/public/home/yqming/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif, withLabel:pore_c_py:/public/home/yqming/Software/wf-pore-c-1.1.0/ontresearch-pore-c-py-shad5608368dcce93646de5559f2f391005408da6c2.sif, withLabel:wf_common:/public/home/yqming/Software/wf-pore-c-1.1.0/wf-common_sha1c5febff9f75143710826498b093d9769a5edbb9.sif, default:~/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif] launchDir : /public/home/yqming/Work_plate/Cotton_data/Pore_data workDir : /public/home/yqming/Work_plate/Cotton_data/Pore_data/work projectDir : /public/home/yqming/Software/wf-pore-c-1.1.0 userName : yqming profile : singularity configFiles : /public/home/yqming/Software/wf-pore-c-1.1.0/nextflow.config, /public/home/yqming/Work_plate/Cotton_data/Pore_data/increase_memory.config

Input Options fastq : ../raw_data/Pore-C_part1.fq.gz ref : ../Gh_genome.fa cutter : DpnII

Output Options out_dir : PoreC_part1 hi_c : true

Advanced Options chunk_size : 40000 threads : 35

Paired-end BAM Options paired_end : true filter_pairs : true paired_end_minimum_distance: 100 paired_end_maximum_distance: 200

!! Only displaying parameters that differ from the pipeline defaults !! -------------------------------------------------------------------------------- If you use epi2me-labs/wf-pore-c for your analysis please cite:

• The nf-core framework https://doi.org/10.1038/s41587-020-0439-x

_This is the error message:_ Aug-20 12:14:22.681 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:19:21.607 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:19:22.732 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:24:21.670 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:24:22.785 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:29:21.685 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:29:22.842 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:34:21.784 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:34:22.916 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:39:21.850 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:39:22.961 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:44:21.892 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:44:23.010 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:49:21.971 [Task monitor] DEBUG n.processor.TaskPollingMonitor - !! executor local > tasks to be completed: 1 -- submitted tasks are shown below ~> TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: RUNNING; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:49:23.164 [Task submitter] DEBUG n.processor.TaskPollingMonitor - %% executor local > tasks in the submission queue: 1 -- tasks to be submitted are shown below ~> TaskHandler[id: 1821; name: POREC:pair_stats_report (1); status: NEW; exit: -; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/a5/65a3c8c05975fdef858bed2bc02278] Aug-20 12:49:37.792 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 1820; name: POREC:prepare_hic (1); status: COMPLETED; exit: 127; error: -; workDir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6] Aug-20 12:49:37.837 [Task submitter] DEBUG n.executor.local.LocalTaskHandler - Launch cmd line: /bin/bash -ue .command.run Aug-20 12:49:37.837 [Task submitter] INFO nextflow.Session - [a5/65a3c8] Submitted process > POREC:pair_stats_report (1) Aug-20 12:49:37.841 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=POREC:prepare_hic (1); work-dir=/public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6 error [nextflow.exception.ProcessFailedException]: Process POREC:prepare_hic (1) terminated with an error exit status (127) Aug-20 12:49:38.100 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > 'POREC:prepare_hic (1)'

Caused by: Process POREC:prepare_hic (1) terminated with an error exit status (127)

Command executed:

cut -f1,2 fasta.fai > sizes.genome pairtools flip input.pairs.gz -c sizes.genome > flipped.pairs.tmp pairtools sort flipped.pairs.tmp > sorted.pairs.tmp pairtools dedup --chunksize 100000 sorted.pairs.tmp > dedup.pairs.tmp java -jar /home/epi2melabs/juicer_tools_1.22.01.jar pre dedup.pairs.tmp "null.hic" sizes.genome rm -rf "*.pairs.tmp"

Command exit status: 127

Command output: (empty)

Command error: .command.sh: line 6: java: command not found

Work dir: /public/home/yqming/Work_plate/Cotton_data/Pore_data/work/b3/cbc440b8048ee9a1097a1867d67ce6

Tip: view the complete command output by changing to the process work dir and entering the command cat .command.out Aug-20 12:49:38.121 [Task monitor] DEBUG nextflow.Session - Session aborted -- Cause: Process POREC:prepare_hic (1) terminated with an error exit status (127) Aug-20 12:49:38.269 [main] DEBUG nextflow.Session - Session await > all processes finished Aug-20 12:49:38.269 [main] DEBUG nextflow.Session - Session await > all barriers passed Aug-20 12:49:38.283 [Task monitor] DEBUG n.processor.TaskPollingMonitor - <<< barrier arrives (monitor: local) - terminating tasks monitor poll loop Aug-20 12:49:38.301 [main] WARN n.processor.TaskPollingMonitor - Killing running tasks (1) Aug-20 12:49:38.318 [main] DEBUG n.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=1819; failedCount=1; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=1; runningCount=-1; retriesCount=0; abortedCount=1; succeedDuration=108d 3h 59m 45s; failedDuration=2d 5h 24m 16s; cachedDuration=0ms;loadCpus=-30; loadMemory=0; peakRunning=2; peakCpus=35; peakMemory=122 GB; ] Aug-20 12:49:38.319 [main] DEBUG nextflow.trace.TraceFileObserver - Workflow completed -- saving trace file Aug-20 12:49:38.410 [main] DEBUG nextflow.trace.ReportObserver - Workflow completed -- rendering execution report Aug-20 12:49:39.809 [main] DEBUG nextflow.trace.TimelineObserver - Workflow completed -- rendering execution timeline Aug-20 12:49:40.126 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Aug-20 12:49:40.150 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

Originally posted by @wangguiqian in https://github.com/epi2me-labs/wf-pore-c/issues/70#issuecomment-2310320844

[wangguiqian]

wang.txt this my log file,please help for me to check .

[sarahjeeeze]

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion?

[wangguiqian]

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion? I have tested your data: wget https://ont-exd-int-s3-euwst1-epi2me-labs.s3.amazonaws.com/wf-pore-c/wf-pore-c-demo.tar.gz tar -xzvf wf-pore-c-demo.tar.gz After getting the data, I ran the command offline. The command is as follows: export NXF_SINGULARITY_CACHEDIR=~/Software/wf-pore-c-1.1.0 export NXF_OFFLINE=true nextflow run ~/Software/wf-pore-c-1.1.0/ --bam 'wf-pore-c-demo/porec_test.concatemers.bam' --threads 12 --chunk_size 100 --cutter 'NlaIII' --hi_c --mcool --paired_end --paired_end_maximum_distance 200 --paired_end_minimum_distance 100 --phased_vcf 'wf-pore-c-demo/porec_test.phased_variants.vcf.gz' --ref 'wf-pore-c-demo/porec_test.fasta' --vcf 'wf-pore-c-demo/porec_test.phased_variants.vcf.gz' --out_dir PoreC_test -profile singularity -with-singularity '~/Software/wf-pore-c-1.1.0/ontresearch-wf-pore-c-sha3787c234c0cacf66a67fb77da223cc2e1cb0baf0.sif' -with-trace

But I still get an error after running it. What is the problem in this case? [15/348233] process > POREC:getVersions [100%] 1 of 1 ✔
[95/f91d93] process > POREC:getParams [100%] 1 of 1 ✔
[b7/db5c87] process > POREC:makeReport [100%] 1 of 1 ✔
[fa/33dacf] process > POREC:prepare_hic (1) [100%] 1 of 1, failed: 1 ✘
[25/b07bdb] process > POREC:get_filtered_out_bam (1) [100%] 1 of 1 ✔
[3c/dce576] process > output (1) [100%] 1 of 1 ✔
ERROR ~ Error executing process > 'POREC:prepare_hic (1)'

Caused by:
Process POREC:prepare_hic (1) terminated with an error exit status (127)

Command executed:

cut -f1,2 fasta.fai > sizes.genome
pairtools flip input.pairs.gz -c sizes.genome > flipped.pairs.tmp
pairtools sort flipped.pairs.tmp > sorted.pairs.tmp
pairtools dedup --chunksize 100000 sorted.pairs.tmp > dedup.pairs.tmp
java -jar /home/epi2melabs/juicer_tools_1.22.01.jar pre dedup.pairs.tmp "null.hic" sizes.genome
rm -rf "*.pairs.tmp"

Command exit status:
127

Command output:
(empty)

Command error:
.command.sh: line 6: java: command not found

Work dir:
/public/home/yqming/Work_plate/work/fa/33dacfe5877c71c4ddced4472c7264

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

This my log file ,please help for to check : pore_c.log

[wangguiqian]

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion?

pairtools dedup --chunksize 100000 sorted.pairs.tmp > dedup.pairs.tmp java -jar /home/epi2melabs/juicer_tools_1.22.01.jar pre dedup.pairs.tmp "null.hic" sizes.genome

But I saw the error message, it seems that there is an error in the pre command of juicer to build HiC. What is the problem? Could you please share a complete pore-C log file for me to see? I will try to run it step by step.

[wangguiqian]

Hi, Looks like some kind of environment error. Can you successfully run the demo data in the read me to completion?

I got the eorrs,can you help to solve?