search

cgroza / GraffiTE

GraffiTE is a pipeline that finds polymorphic transposable elements in genome assemblies and/or long reads, and genotypes the discovered polymorphisms in read sets using genome-graphs.
Other
106 stars 4 forks source link

ERROR ~ Error executing process > 'graph_align_reads (41)' #33

Open yykaya opened 1 month ago

yykaya commented 1 month ago

Hi,

I'm attempting to run GraffiTE in GT-sv-GA mode, but I encounter an error during the execution of the graph_align_reads (41) process, as shown in the message below. (However, when I run this step manually using GraphAligner, it completes successfully but the job gets killed.) Do you have any idea what might be causing this issue?

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details

executor > local (507) [21/3da086] process > map_asm (1) [100%] 115 of 115 ✔ [cb/7c0a52] process > svim_asm (115) [100%] 115 of 115 ✔ [13/513669] process > survivor_merge [100%] 1 of 1 ✔ [b8/15025a] process > repeatmask_VCF (1) [100%] 1 of 1 ✔ [a8/d2e2df] process > tsd_prep (1) [100%] 1 of 1 ✔ [94/e3d127] process > tsd_search (148) [100%] 157 of 157 ✔ [60/be72ec] process > tsd_report (1) [100%] 1 of 1 ✔ [77/79f066] process > make_graph (1) [100%] 1 of 1 ✔ [1a/75fbbf] process > graph_align_reads (49) [100%] 115 of 115, failed: 89 ✘ [- ] process > vg_call - [- ] process > merge_VCFs - ERROR ~ Error executing process > 'graph_align_reads (41)'

Caused by: Process graph_align_reads (41) terminated with an error exit status (134)

Command executed:

GraphAligner -t 1 -x vg -g index/index.vg -f X.fastq.gz -a X.gam

vg pack -x index/index.vg -g X.gam -o X.pack -Q 0

Command exit status: 134

Command output: GraphAligner bioconda 1.0.13- Load graph from index/index.vg Build minimizer seeder from the graph Minimizer seeds, length 15, window size 20, density 10 Seed cluster size 1 Alignment bandwidth 10 Clip alignment ends with identity < 66% X-drop DP score cutoff 14705 write alignments to X.gam Align

Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: fuse2fs not found, will not be able to mount EXT3 filesystems GraphAligner bioconda 1.0.13- GraphAligner bioconda 1.0.13- Load graph from index/index.vg Build minimizer seeder from the graph Minimizer seeds, length 15, window size 20, density 10 Seed cluster size 1 Alignment bandwidth 10 Clip alignment ends with identity < 66% X-drop DP score cutoff 14705 write alignments to X.gam Align Signal 11. Read: 2a9030c3-0fd1-4219-8496-ad2d7f1fb33b runid=b8d4812b34a5a9fdf693319045b7ee7e26775c97 read=30068 ch=2021 start_time=2019-11-30T05:26:38Z flow_cell_id=PAD94007 protocol_group_id=191128-CLIMARES sample_id=MULTIPLEX6-2-M40-S 00-L14 barcode=barcode07. Seed: 34132+,3021,15,752 .command.sh: line 2: 16 Aborted GraphAligner -t 1 -x vg -g index/index.vg -f X.fastq.gz -a X.gam

This is from .nextflow log file:

Jul-12 03:21:25.913 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (35); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/afee1f76707eac941eba261f63cc7d error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (35) failed Jul-12 03:21:25.914 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 426; name: graph_align_reads (34); status: COMPLETED; exit: -; error: nextflow.exception.ProcessE xception: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/39/b2a0764669a366a56a5e598a98e656] Jul-12 03:21:25.914 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (34); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/39/b2a0764669a366a56a5e598a98e656 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (34) failed Jul-12 03:21:25.915 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 440; name: graph_align_reads (48); status: COMPLETED; exit: -; error: nextflow.exception.ProcessE xception: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/dd/a38cb1d0b16fac105668ec2bf9234a] Jul-12 03:21:25.915 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (48); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/dd/a38cb1d0b16fac105668ec2bf9234a error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (48) failed Jul-12 03:21:25.915 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 396; name: graph_align_reads (4); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/97/785f099c658f091c3b16c7b0abe50e] Jul-12 03:21:25.915 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (4); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/97/785f099c658f091c3b16c7b0abe50e error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (4) failed Jul-12 03:21:25.916 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 434; name: graph_align_reads (42); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/24/4cc79fecfa1f92029986601a22bd88] Jul-12 03:21:25.916 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (42); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/24/4cc79fecfa1f92029986601a22bd88 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (42) failed Jul-12 03:21:25.929 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 483; name: graph_align_reads (91); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/fd/a12ae6a90d2858d7514f4992c60e24] Jul-12 03:21:25.929 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (91); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/fd/a12ae6a90d2858d7514f4992c60e24 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (91) failed Jul-12 03:21:25.953 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 405; name: graph_align_reads (13); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/5a8dbb35508e8365cbef0793e7f1f1] Jul-12 03:21:25.954 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (13); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/5a8dbb35508e8365cbef0793e7f1f1 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (13) failed Jul-12 03:21:25.954 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 462; name: graph_align_reads (70); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/6143156afdcd9ead22067413ce58d3] Jul-12 03:21:25.954 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (70); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/6143156afdcd9ead22067413ce58d3 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (70) failed Jul-12 03:21:25.955 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 484; name: graph_align_reads (92); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/50/5f76f2b85046ea4410442f61413fb8] Jul-12 03:21:25.955 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (92); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/50/5f76f2b85046ea4410442f61413fb8 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (92) failed Jul-12 03:21:25.955 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 441; name: graph_align_reads (49); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/1a/75fbbffbd47b5475851ec1121ee26c] Jul-12 03:21:25.956 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (49); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/1a/75fbbffbd47b5475851ec1121ee26c error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (49) failed Jul-12 03:21:26.093 [main] DEBUG nextflow.Session - Session await > all barriers passed Jul-12 03:21:26.103 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false) Jul-12 03:21:26.210 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=418; failedCount=89; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=9d 21h 44m 9s; failedDuration=43d 16h 36m 38s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=115; peakCpus=115; peakMemory=480 GB; ] Jul-12 03:21:26.517 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Jul-12 03:21:26.571 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false) Jul-12 03:21:26.571 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

The command I tried:

/u/ykaya/nextflow run https://github.com/cgroza/GraffiTE --genotype true \
   --assemblies assemblies.csv \
   --TE_library TE.nonredun.fa \
   --reference /ptmp/X.fasta \
   --graph_method graphaligner \
   --reads longreads.csv
cgroza commented 1 month ago

Seems like you are running out of time and the job gets killed.

You can try increasing the allocated time for GraphAligner, there is a parameter for that.

-------- Original Message -------- On 7/12/24 5:06 AM, Yasin K. wrote:

Hi,

I'm attempting to run GraffiTE in GT-sv-GA mode, but I encounter an error during the execution of the graph_align_reads (41) process, as shown in the message below. (However, when I run this step manually using GraphAligner, it completes successfully but the job gets killed.) Do you have any idea what might be causing this issue?

Tip: you can replicate the issue by changing to the process work dir and entering the command bash .command.run

-- Check '.nextflow.log' file for details

executor > local (507) [21/3da086] process > map_asm (1) [100%] 115 of 115 ✔ [cb/7c0a52] process > svim_asm (115) [100%] 115 of 115 ✔ [13/513669] process > survivor_merge [100%] 1 of 1 ✔ [b8/15025a] process > repeatmask_VCF (1) [100%] 1 of 1 ✔ [a8/d2e2df] process > tsd_prep (1) [100%] 1 of 1 ✔ [94/e3d127] process > tsd_search (148) [100%] 157 of 157 ✔ [60/be72ec] process > tsd_report (1) [100%] 1 of 1 ✔ [77/79f066] process > make_graph (1) [100%] 1 of 1 ✔ [1a/75fbbf] process > graph_align_reads (49) [100%] 115 of 115, failed: 89 ✘ [- ] process > vg_call - [- ] process > merge_VCFs - ERROR ~ Error executing process > 'graph_align_reads (41)'

Caused by: Process graph_align_reads (41) terminated with an error exit status (134)

Command executed:

GraphAligner -t 1 -x vg -g index/index.vg -f X.fastq.gz -a X.gam

vg pack -x index/index.vg -g X.gam -o Cant-1.pack -Q 0

Command exit status: 134

Command output: GraphAligner bioconda 1.0.13- Load graph from index/index.vg Build minimizer seeder from the graph Minimizer seeds, length 15, window size 20, density 10 Seed cluster size 1 Alignment bandwidth 10 Clip alignment ends with identity < 66% X-drop DP score cutoff 14705 write alignments to Cant-1.gam Align

Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: fuse2fs not found, will not be able to mount EXT3 filesystems GraphAligner bioconda 1.0.13- GraphAligner bioconda 1.0.13- Load graph from index/index.vg Build minimizer seeder from the graph Minimizer seeds, length 15, window size 20, density 10 Seed cluster size 1 Alignment bandwidth 10 Clip alignment ends with identity < 66% X-drop DP score cutoff 14705 write alignments to Cant-1.gam Align Signal 11. Read: 2a9030c3-0fd1-4219-8496-ad2d7f1fb33b runid=b8d4812b34a5a9fdf693319045b7ee7e26775c97 read=30068 ch=2021 start_time=2019-11-30T05:26:38Z flow_cell_id=PAD94007 protocol_group_id=191128-CLIMARES sample_id=MULTIPLEX6-2-M40-S 00-L14 barcode=barcode07. Seed: 34132+,3021,15,752 .command.sh: line 2: 16 Aborted GraphAligner -t 1 -x vg -g index/index.vg -f X.fastq.gz -a X.gam

This is from .nextflow log file:

Jul-12 03:21:25.913 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (35); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/afee1f76707eac941eba261f63cc7d error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (35) failed Jul-12 03:21:25.914 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 426; name: graph_align_reads (34); status: COMPLETED; exit: -; error: nextflow.exception.ProcessE xception: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/39/b2a0764669a366a56a5e598a98e656] Jul-12 03:21:25.914 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (34); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/39/b2a0764669a366a56a5e598a98e656 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (34) failed Jul-12 03:21:25.915 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 440; name: graph_align_reads (48); status: COMPLETED; exit: -; error: nextflow.exception.ProcessE xception: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/dd/a38cb1d0b16fac105668ec2bf9234a] Jul-12 03:21:25.915 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (48); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/dd/a38cb1d0b16fac105668ec2bf9234a error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (48) failed Jul-12 03:21:25.915 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 396; name: graph_align_reads (4); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/97/785f099c658f091c3b16c7b0abe50e] Jul-12 03:21:25.915 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (4); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/97/785f099c658f091c3b16c7b0abe50e error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (4) failed Jul-12 03:21:25.916 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 434; name: graph_align_reads (42); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/24/4cc79fecfa1f92029986601a22bd88] Jul-12 03:21:25.916 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (42); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/24/4cc79fecfa1f92029986601a22bd88 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (42) failed Jul-12 03:21:25.929 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 483; name: graph_align_reads (91); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/fd/a12ae6a90d2858d7514f4992c60e24] Jul-12 03:21:25.929 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (91); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/fd/a12ae6a90d2858d7514f4992c60e24 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (91) failed Jul-12 03:21:25.953 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 405; name: graph_align_reads (13); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/5a8dbb35508e8365cbef0793e7f1f1] Jul-12 03:21:25.954 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (13); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/5a8dbb35508e8365cbef0793e7f1f1 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (13) failed Jul-12 03:21:25.954 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 462; name: graph_align_reads (70); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/6143156afdcd9ead22067413ce58d3] Jul-12 03:21:25.954 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (70); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/58/6143156afdcd9ead22067413ce58d3 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (70) failed Jul-12 03:21:25.955 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 484; name: graph_align_reads (92); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/50/5f76f2b85046ea4410442f61413fb8] Jul-12 03:21:25.955 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (92); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/50/5f76f2b85046ea4410442f61413fb8 error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (92) failed Jul-12 03:21:25.955 [Task monitor] DEBUG n.processor.TaskPollingMonitor - Task completed > TaskHandler[id: 441; name: graph_align_reads (49); status: COMPLETED; exit: -; error: nextflow.exception.ProcessException: Process exceeded running time limit (12h); workDir: /raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/1a/75fbbffbd47b5475851ec1121ee26c] Jul-12 03:21:25.956 [Task monitor] DEBUG nextflow.processor.TaskProcessor - Handling unexpected condition for task: name=graph_align_reads (49); work-dir=/raven/ptmp/ykaya/Pangenome_project/SV/TE_SV/work/1a/75fbbffbd47b5475851ec1121ee26c error [nextflow.exception.ProcessFailedException]: Process graph_align_reads (49) failed Jul-12 03:21:26.093 [main] DEBUG nextflow.Session - Session await > all barriers passed Jul-12 03:21:26.103 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'PublishDir' shutdown completed (hard=false) Jul-12 03:21:26.210 [main] DEBUG nextflow.trace.WorkflowStatsObserver - Workflow completed > WorkflowStats[succeededCount=418; failedCount=89; ignoredCount=0; cachedCount=0; pendingCount=0; submittedCount=0; runningCount=0; retriesCount=0; abortedCount=0; succeedDuration=9d 21h 44m 9s; failedDuration=43d 16h 36m 38s; cachedDuration=0ms;loadCpus=0; loadMemory=0; peakRunning=115; peakCpus=115; peakMemory=480 GB; ] Jul-12 03:21:26.517 [main] DEBUG nextflow.cache.CacheDB - Closing CacheDB done Jul-12 03:21:26.571 [main] DEBUG nextflow.util.ThreadPoolManager - Thread pool 'FileTransfer' shutdown completed (hard=false) Jul-12 03:21:26.571 [main] DEBUG nextflow.script.ScriptRunner - > Execution complete -- Goodbye

The command I tried:

/u/ykaya/nextflow run https://github.com/cgroza/GraffiTE --genotype true \ --assemblies assemblies.csv \ --TE_library TE.nonredun.fa \ --reference /ptmp/X.fasta \ --graph_method graphaligner \ --reads longreads.csv

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.Message ID: @.***>

yykaya commented 1 month ago

I disagree that the issue is solely due to time limits. Despite attempting the task three times with different configurations and time limits, I consistently encounter the error "ERROR ~ Error executing process > 'graph_align_reads (105)'" for various accessions. My jobs run on HPC nodes equipped with GPUs, allowing for a 24-hour execution window. The tool typically runs for 18-19 hours before encountering these errors.

Here are the updated commands I'm using (currently running for 115 samples):

/u/ykaya/nextflow run https://github.com/cgroza/GraffiTE --genotype true \
   --assemblies assemblies.csv \
   --TE_library TE.nonredun.fa \
   --reference /ptmp/X.fasta \
   --graph_method graphaligner \
   --cores 4 \
   --repeatmasker_memory 18G \
   --graph_align_memory 18G \
   --vg_call_memory 18G \
   --make_graph_memory 18G \
   --map_longreads_memory 18G \
   --map_longreads_time 5h \
   --sniffles_time 8h \
   --graph_align_time 18h \
   --repeatmasker_time 5h \
   --min_mapq 5 \
   --longreads longreads.csv
cgroza commented 1 month ago

I see this line in your logs coming from GraphAligner:

Signal 11. Read: 2a9030c3-0fd1-4219-8496-ad2d7f1fb33b runid=b8d4812b34a5a9fdf693319045b7ee7e26775c97 read=30068 ch=2021 start_time=2019-11-30T05:26:38Z flow_cell_id=PAD94007 protocol_group_id=191128-CLIMARES sample_id=MULTIPLEX6-2-M40-S
00-L14 barcode=barcode07. Seed: 34132+,3021,15,752

It indicates the name of the read that crashed GraphAligner. This is either a bug in GraphAligner or the read crashes GraphAligner. Is it possible that it's running out of memory?

clemgoub commented 1 month ago

I had similar issue in the past, and I would lean onto the possibility that this is a bug in GraphAligner. I would suggest to perhaps remove this read from the dataset (though another one may be problematic).

This error seem to be non-deterministic. What seemed to work in some case was changing the number of CPU. I had mentioned the problem here but did not get any answer. I will update my message there in case they can help.

Best,

Clément

yykaya commented 1 month ago

Despite multiple attempts to run with GraphAligner, the issue persists.

I have now attempted the genotyping step using vg giraffie, but encountered the following error message. This issue appears to have been addressed in commit For reference, you can review the related issue

executor > local (621) [d0/8f092f] process > map_longreads (76) [100%] 115 of 115 ✔ [fd/0cc43b] process > sniffles_sample_call (115) [100%] 115 of 115 ✔ [44/c4b4f6] process > sniffles_population_call (1) [100%] 1 of 1 ✔ [ba/97c7eb] process > map_asm (105) [100%] 115 of 115 ✔ [45/77378f] process > svim_asm (115) [100%] 115 of 115 ✔ [b4/e3ced0] process > survivor_merge [100%] 1 of 1 ✔ [2c/60a236] process > merge_svim_sniffles2 (1) [100%] 1 of 1 ✔ [91/1b394e] process > repeatmask_VCF (1) [100%] 1 of 1 ✔ [50/838199] process > tsd_prep (1) [100%] 1 of 1 ✔ [c5/ed6b2e] process > tsd_search (3) [100%] 154 of 154 ✔ [c7/9d28ed] process > tsd_report (1) [100%] 1 of 1 ✔ [d5/b8a4b2] process > make_graph (1) [100%] 1 of 1, failed: 1 ✘ [- ] process > graph_align_reads - [- ] process > vg_call - [- ] process > merge_VCFs - ERROR ~ Error executing process > 'make_graph (1)'

Caused by: Process make_graph (1) terminated with an error exit status (134)

Command executed:

bcftools sort -Oz -o sorted.vcf.gz pangenome.vcf tabix sorted.vcf.gz mkdir index

  vg autoindex --tmp-dir $PWD  -p index/index -w giraffe -v sorted.vcf.gz -r X.fasta

vg snarls index/index.giraffe.gbz > index/index.pb

Command exit status: 134

Command output: (empty)

Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: fuse2fs not found, will not be able to mount EXT3 filesystems Writing to /tmp/bcftools.OAKXDo [W::bcf_hrec_check] Invalid tag name: "svim-asm_SUPP" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVLEN" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVTYPE" [W::bcf_hrec_check] Invalid tag name: "svim-asm_ID" Merging 1 temporary files [W::bcf_hrec_check] Invalid tag name: "svim-asm_SUPP" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVLEN" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVTYPE" [W::bcf_hrec_check] Invalid tag name: "svim-asm_ID" Cleaning Done [vg autoindex] Executing command: vg autoindex --tmp-dir /raven/ptmp/ykaya/path/work/d5/b8a4b27a8b0aa20f39868bb276d9a5 -p index/index -w giraffe -v sorted.vcf.gz -r X.fasta [IndexRegistry]: Checking for phasing in VCF(s). [IndexRegistry]: Chunking inputs for parallelism. index file X.fasta.fai not found, generating... [IndexRegistry]: Chunking FASTA(s). [IndexRegistry]: Chunking VCF(s). [IndexRegistry]: Constructing VG graph from FASTA and VCF input. warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr2; coercing to N. warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr3; coercing to N. warning:[vg::Constructor] Skipping duplicate variant with hash 7a4a5efd14440d32a6d1bb6f91a992cd6178fcf7 at Chr3:3110404 warning:[vg::Constructor] Skipping duplicate variant with hash d1c7edc8494207c7c32084674a18a25159078735 at Chr4:4150882 warning:[vg::Constructor] Skipping duplicate variant with hash e37e77706134d082ccb2d9eb8ac109fa35b4f3a3 at Chr2:4593323 warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr4; coercing to N. vg: src/constructor.cpp:528: vg::ConstructedChunk vg::Constructor::construct_chunk(std::string, std::string, std::vector, size_t) const: Assertion `!variant->isSymbolicSV()' failed. ERROR: Signal 6 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug. Stack trace path: /tmp/vg_crash_9IYtPC/stacktrace.txt Please include the stack trace file in your bug report!

any comments and suggestions are welcome!

cgroza commented 1 month ago

Seems that your data contains unsupported IUPAC nucleotides. I had this break GraphAligner in the past and is consistent with the new warning messages in VG.

-------- Original Message -------- On 7/19/24 8:34 AM, Yasin K. wrote:

Despite multiple attempts to run with GraphAligner, the issue persists.

I have now attempted the genotyping step using vg giraffie, but encountered the following error message. This issue appears to have been addressed in commit For reference, you can review the related issue

executor > local (621) [d0/8f092f] process > map_longreads (76) [100%] 115 of 115 ✔ [fd/0cc43b] process > sniffles_sample_call (115) [100%] 115 of 115 ✔ [44/c4b4f6] process > sniffles_population_call (1) [100%] 1 of 1 ✔ [ba/97c7eb] process > map_asm (105) [100%] 115 of 115 ✔ [45/77378f] process > svim_asm (115) [100%] 115 of 115 ✔ [b4/e3ced0] process > survivor_merge [100%] 1 of 1 ✔ [2c/60a236] process > merge_svim_sniffles2 (1) [100%] 1 of 1 ✔ [91/1b394e] process > repeatmask_VCF (1) [100%] 1 of 1 ✔ [50/838199] process > tsd_prep (1) [100%] 1 of 1 ✔ [c5/ed6b2e] process > tsd_search (3) [100%] 154 of 154 ✔ [c7/9d28ed] process > tsd_report (1) [100%] 1 of 1 ✔ [d5/b8a4b2] process > make_graph (1) [100%] 1 of 1, failed: 1 ✘ [- ] process > graph_align_reads - [- ] process > vg_call - [- ] process > merge_VCFs - ERROR ~ Error executing process > 'make_graph (1)'

Caused by: Process make_graph (1) terminated with an error exit status (134)

Command executed:

bcftools sort -Oz -o sorted.vcf.gz pangenome.vcf tabix sorted.vcf.gz mkdir index

vg autoindex --tmp-dir $PWD -p index/index -w giraffe -v sorted.vcf.gz -r X.fasta

vg snarls index/index.giraffe.gbz > index/index.pb

Command exit status: 134

Command output: (empty)

Command error: INFO: Environment variable SINGULARITYENV_TMPDIR is set, but APPTAINERENV_TMPDIR is preferred INFO: fuse2fs not found, will not be able to mount EXT3 filesystems Writing to /tmp/bcftools.OAKXDo [W::bcf_hrec_check] Invalid tag name: "svim-asm_SUPP" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVLEN" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVTYPE" [W::bcf_hrec_check] Invalid tag name: "svim-asm_ID" Merging 1 temporary files [W::bcf_hrec_check] Invalid tag name: "svim-asm_SUPP" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVLEN" [W::bcf_hrec_check] Invalid tag name: "svim-asm_SVTYPE" [W::bcf_hrec_check] Invalid tag name: "svim-asm_ID" Cleaning Done [vg autoindex] Executing command: vg autoindex --tmp-dir /raven/ptmp/ykaya/path/work/d5/b8a4b27a8b0aa20f39868bb276d9a5 -p index/index -w giraffe -v sorted.vcf.gz -r X.fasta [IndexRegistry]: Checking for phasing in VCF(s). [IndexRegistry]: Chunking inputs for parallelism. index file X.fasta.fai not found, generating... [IndexRegistry]: Chunking FASTA(s). [IndexRegistry]: Chunking VCF(s). [IndexRegistry]: Constructing VG graph from FASTA and VCF input. warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr2; coercing to N. warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr3; coercing to N. warning:[vg::Constructor] Skipping duplicate variant with hash 7a4a5efd14440d32a6d1bb6f91a992cd6178fcf7 at Chr3:3110404 warning:[vg::Constructor] Skipping duplicate variant with hash d1c7edc8494207c7c32084674a18a25159078735 at Chr4:4150882 warning:[vg::Constructor] Skipping duplicate variant with hash e37e77706134d082ccb2d9eb8ac109fa35b4f3a3 at Chr2:4593323 warning:[vg::Constructor] Unsupported IUPAC ambiguity codes found in Chr4; coercing to N. vg: src/constructor.cpp:528: vg::ConstructedChunk vg::Constructor::construct_chunk(std::string, std::string, std::vectorvcflib::Variant, size_t) const: Assertion `!variant->isSymbolicSV()' failed. ERROR: Signal 6 occurred. VG has crashed. Visit https://github.com/vgteam/vg/issues/new/choose to report a bug. Stack trace path: /tmp/vg_crash_9IYtPC/stacktrace.txt Please include the stack trace file in your bug report!

any comments and suggestions are welcome!

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>