Hung when running NanoporeBC_UMI_finder-2.1.jar assignumis

[kwglam]

Hi,

I am running 'Step 3: UMI assignment' with a 124G pass.bam file. The program ran well for an hour but hung since then. It generated a passedParsed.bam file, a passedParsed.bai file, and an empty log file . It works well when I run a chromosome-split bam file individually. But the same issue appears when running it using all chromosome-split bam files in a loop. Is there anything that I can change in the config.xml to fix it? Thanks

[RainerWaldmann]

Hi,

Sorry for the late reply. Could you post the last lines of output before it got stuck.

Rainer

[kwglam]

Hi Rainer, Thanks very much for spending time on this. The program just got stuck at another place when I ran it again. This time a truncated bam file was generated, but empty bai and log files.

Last few lines of the bam file: efda6482-6c6a-4210-b19a-77f3c874a7e6_FWD_PS=794_PE=819_AE=847_bc=TTATTGCCATGGACAG_ed=0_ed_sec=2147483647_bcStart=846_bcEnd=831_rk=1078_X=AAAAAAAAAAAAACCGTCGATTGCCTGTCCATGGCAATAAAGA_Q=30.3_1mam40 272 chr24613300 0 122S19M1I44M2I211M5D18M1I7M1D12M2D3M1D13M2I48M2D4M1I64M1D3M1D213M130S 0 0 B1:i:0 U1:Z:0 s1:i:560 B2:Z:2147483647 U2:Z:3 U7:Z:GCAATCGACGGT U8:Z:GCAATCGACGGT BB:Z:846 BC:Z:TTATTGCCATGGACAG UC:Z: AE:i:847 BE:Z:831 GE:Z:Gm28437 PE:i:819 BF:Z:831 XF:Z:UTR BH:Z:1078 NM:i:32 AS:i:579 GS:Z:- PS:i:794 BU:Z:TTATTGCCATGGACAG BV:Z:846 BW:i:0 BX:Z:N.A. SX:Z:97912944 BZ:Z:TTATTGCCATGGACAG de:f:0.0358 rl:i:33 cm:i:175 nn:i:0 tp:A:S ms:i:579 ts:A:+ eff50a12-a213-4829-8e8d-579a418c601a_FWD_PS=791_PE=829_AE=850_bc=GTCACTCAGCATGCAG_ed=0_ed_sec=1_bcStart=849_bcEnd=834_rk=88_X=AAAAAAAAAAAACAGTAAAACTATCTGCATGCTGAGTGACAGA_Q=27.3_g5xq1 272 chr1 24613300 130S158M1D122M3D31M1I126M1D5M1D6M1D43M3D171M131S 0 0 B1:i:0 U1:Z:0 s1:i:626 B2:Z:1 U2:Z:1 U7:Z:ATAGTTTTACTG U8:Z:ATAGTTTTACTG BB:Z:849 BC:Z:GTCACTCAGCATGCAG UC:Z: AE:i:850 BE:Z:834 GE:Z:Gm28437 PE:i:829 BF:Z:834 XF:Z:UTR BH:Z:88 NM:i:20 AS:i:610 GS:Z:- PS:i:791 BU:Z:GTCACTCAGCATGCAG BV:Z:849 BW:i:0 BX:Z:1 SX:Z:27150841 BZ:Z:GTCACTCAGCATGCAG de:f:0.0239 rl:i:39 cm:i:178 nn:i:0 tp:A:S ms:i:610 ts:A:+ f0120ede-acea-4de4-8121-04ec3a9d731e_FWD_PS=398_PE=426_AE=454_bc=TCCGGGAAGGGTGAGG_ed=1_ed_sec=2147483647_bcStart=453_bcEnd=437_rk=675_X=AAAAAAAAAAATAGCCACATCATACTCCACCCTTCCCGGAAGA_Q=16.8_217vt9 16 chr24613300 0 127S26M1D109M1I22M1D38M2D66M139S 0 0 TGTGACAAGCAAAGTTGTCGGTGTCTTTGTGACTTGCCTGTCGCTCTATCTTCCTACACGACGCTCTTCCGATCTTCCGGGAAGGGTGGAGTATGATGTGGCTATTTTTTTTTTTTTTTTTTTTTTTTTTTCGTTGTCGTAGTAGGCAAACAACAGGAATGTTGATCCAATAATTATATGGAGTCCATGGAATCCAGTAGCCATGAAGAATGTAGAACCACAGATACCATCTGAAATGGAGAATGATGTTTCAAAGTATTCTGGAAGCTTGGAGGATGGTGAAGTAAGTCCTAGTATAATGGCAATTAGTAGGGCTTGATTTATGGCTTCGTTTACCTTCTATAAGGCTATGATGAGCTCATGTAATTGAAATTCCAGATGCTAGAAGTCCCATGTACTCTGCGTTGATACCACTGCTTGCAATATCAGCACCAACAGAAACACAAAGACACCGACAACTTCTCCTATCACGGTAGGCTTAGATGAACTGGAGCAACACCAAACACTGTAGTAAATTGAACGAAGTAC $%%('&&()39:?>>;9742210,,4566310//,,,2/..''')0-.+--258:CB??A==>>?@DDEBCB=>74.,.-)),'''''7+(((((+./1248;;<===>>????????>>999:;;?>>>=;/)(()..44))(())0778;A>;:99999C:7(('()08;3++.01=?:62.,+(%%%%&)*;??@<<<::521'&&&(.ACDDDD?<<;322133<;;=@?CCCCC:BA@?72-,''(20111<;:;<<<8/.---)))&():;;;:::900=210/.---/06?75-+**063**('(**)+,,,,>;/--;<?@@<<AABBEDEDC88766668AB?*))(('%%&&'):22222:?7<824435740+*.022CEGFE@?@=++++,226:BDBFDFFDDACCD@=-(HEGEDFCDCD75455@;80++,-)))),,,--2/,+'%$$%%&&&&%&%$$''()74+,)&$$$$%%&&232))(((++18:984-( B1:i:1 s1:i:214 B2:Z:2147483647 s2:i:214 U7:Z:ATGATGTGGCTA U8:Z:ATGATGTGGCTA BB:Z:453 BC:Z:TCCGGGAAGGGTGAGG AE:i:454 BE:Z:437 GE:Z:Gm28437 PE:i:426 BF:Z:437 XF:Z:UTR BH:Z:675 NM:i:14 AS:i:221 GS:Z:- PS:i:398 BU:Z:TCCGGGAAGGGTGAGG BV:Z:453 BW:i:1 BX:Z:N.A. SX:Z:122979789 BZ:Z:TCCGGGAAGGGTGAGG UZ:Z: de:f:0.0491 rl:i:31 cm:i:45 nn:i:0 tp:A:P ms:i:221 ts:A:+

Last few lines of the nohup: O.K. Found and using /home/packages/sicelore-2.1/Jar/umiClusteringEditDistances.xml ............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................... 3,162,955 SAM records processed in PT-10M-44.825888483S Umis found: 1,149,085 (36.33 % ) Umis found by clustering: 1,149,085 (36.32 % ) ................................................................10:41:42.941 [ForkJoinPool-6-worker-171] FATAL com.rw.umifinder.analyzers.clustering.UmiClustering - One Clustering Job failed:

java.util.concurrent.CancellationException: null at java.util.concurrent.ForkJoinTask.uncheckedThrow(ForkJoinTask.java:602) ~[?:?] at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:567) ~[?:?] at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:670) ~[?:?] at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) ~[?:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:?] at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:679) ~[?:?] at com.rw.umifinder.analyzers.clustering.ClusterOne_MyClustering.clusterLocal(ClusterOne_MyClustering.java:196) ~[NanoporeBC_UMI_finder-2.1.jar:?] at com.rw.umifinder.analyzers.clustering.ClusterOne_MyClustering.call(ClusterOne_MyClustering.java:72) ~[NanoporeBC_UMI_finder-2.1.jar:?] at com.rw.umifinder.analyzers.clustering.ClusterOne_MyClustering.call(ClusterOne_MyClustering.java:37) ~[NanoporeBC_UMI_finder-2.1.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-31.1-jre.jar:?] at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) ~[guava-31.1-jre.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-31.1-jre.jar:?] at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423) ~[?:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) ~[?:?] at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311) ~[?:?] at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841) ~[?:?] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806) ~[?:?] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?] 10:41:43.766 [ForkJoinPool-6-worker-182] FATAL com.rw.umifinder.analyzers.clustering.UmiClustering - One Clustering Job failed:

java.util.concurrent.CancellationException: null at java.util.concurrent.ForkJoinTask.uncheckedThrow(ForkJoinTask.java:602) ~[?:?] at java.util.concurrent.ForkJoinTask.reportException(ForkJoinTask.java:567) ~[?:?] at java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:670) ~[?:?] at java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:159) ~[?:?] at java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:173) ~[?:?] at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:233) ~[?:?] at java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:596) ~[?:?] at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:679) ~[?:?] at com.rw.umifinder.analyzers.clustering.ClusterOne_MyClustering.clusterLocal(ClusterOne_MyClustering.java:196) ~[NanoporeBC_UMI_finder-2.1.jar:?] at com.rw.umifinder.analyzers.clustering.ClusterOne_MyClustering.call(ClusterOne_MyClustering.java:72) ~[NanoporeBC_UMI_finder-2.1.jar:?] at com.rw.umifinder.analyzers.clustering.ClusterOne_MyClustering.call(ClusterOne_MyClustering.java:37) ~[NanoporeBC_UMI_finder-2.1.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131) ~[guava-31.1-jre.jar:?] at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74) ~[guava-31.1-jre.jar:?] at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82) ~[guava-31.1-jre.jar:?] at java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1423) ~[?:?] at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:387) ~[?:?] at java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1311) ~[?:?] at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1841) ~[?:?] at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1806) ~[?:?] at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:177) ~[?:?]

[RainerWaldmann]

Hi,

This is actually within a parallel stream that should not fail. I am working on it. Did you get anywhere in the nohup.out an "java.lang.OutOfMemoryError" ?

Rainer

[kwglam]

Hi,

This is actually within a parallel stream that should not fail. I am working on it. Did you get anywhere in the nohup.out an "java.lang.OutOfMemoryError" ?

Rainer

For this particular nohup, I think I killed the run when I found it got stuck. Yes, in most cases, I saw "java.lang.OutOfMemoryError" after these messages. But I used >300G memory in most cases.

[RainerWaldmann]

Hi, This is actually within a parallel stream that should not fail. I am working on it. Did you get anywhere in the nohup.out an "java.lang.OutOfMemoryError" ? Rainer

For this particular nohup, I think I killed the run when I found it got stuck. Yes, in most cases, I saw "java.lang.OutOfMemoryError" after these messages. But I used >300G memory in most cases.

Try to reduce the in the config.xml to 250,000 if it is higher than 250k or to 100k .

[theheking]

Just wanted to say this also happened to me, and the reduction of the chunk size worked!

[RainerWaldmann]

Some datasets (in particular targeted sequencing) have a high number of reads for a given genomic region. You end up with huge distance matrices for clustering and a rather high RAM requirement if the chunk size is too high. To cope with this, I changed the clustering code. I am now doing some pre-clustering and I am splitting abnormally huge clustering matrices to protect against high RAM usage. I am currently testing this and will release it soon.

[theheking]

Hi @RainerWaldmann - have you had success with adapting the code for the high RAM requirement? Cheers