Killed after receive Segmentation fault:SIGSEGV(11) signal

[dosshra]

running: kmtricks pipeline --file fof2 --run-dir ./kmer_pa --kmer-size 31 --mode kmer:pa:text -t 10 I get:

Killed after receive Segmentation fault:SIGSEGV(11) signal

running 120 samples, 420 read files (~14GB each) on Ubuntu 22.04, kernel 6.5.0-21-generic, kmtricks version v1.4.0. installed in conda environment. 125GB of RAM, 50BG sawp, 25 CPU. I monitored CPU and RAM using htop and did not see overuse of either CPU or RAM see log file:

Backtrace: 1 0x00007fe97f642520 (null) + 140640841377056 2 0x000055a6bf40c531 (null) + 94174661625137 3 0x000055a6bf40c33d (null) + 94174661624637 4 0x000055a6bf3ee1a3 gatb::core::kmer::impl::RepartitorAlgorithm<32ul>::computeRepartition(gatb::core::kmer::impl::Repartitor&) + 563 5 0x000055a6bf3eee6a gatb::core::kmer::impl::RepartitorAlgorithm<32ul>::execute() + 138 6 0x000055a6bf290914 (null) + 94174660069652 7 0x000055a6bf291098 (null) + 94174660071576 8 0x000055a6bf2917d0 (null) + 94174660073424 9 0x000055a6bf044260 main + 3312 10 0x00007fe97f629d90 (null) + 140640841276816 11 0x00007fe97f629e40 __libc_start_main + 128 12 0x000055a6bf0465e5 (null) + 94174657668581

Please advise? Thank you

[tlemane]

Hello,

Sorry for this issue. The problem seems to occur at an early stage of the process in an external library which basically reads a few sequences from each input file to compute a partitioning scheme. I have never encountered this kind of problem, can you post an example of your data ? Just a few lines from one of your fastq ?

Teo

[dosshra]

Thank you @tlemane for the response. See attached file. I should mention that a toy example of 3 fastq files from a different source worked well on the same machine.
Thank you ERR_example.fastq.txt

[dosshra]

Hello, I Downloaded a different version of the fastq files with a limit of 1M reads. kmtricks pipeline was running successfully on 20 files. I will try up-scaling.

[dosshra]

Hello I tried to run with all samples and data. 120 samples, 420 read files (~14GB each). kmtricks pipeline --file ./fof1 --run-dir ~/18tb/kmer_pa_bin --kmer-size 31 --mode kmer:pa:bin -t 10 --hard-min 2 2>kmtricks.err I got these running messages:

Compute SuperK [=======> ] [09h:33m:15s]
Compute SuperK [================================> ] [01d:11h:10m:14s]
Compute SuperK [==================================================] [02d:04h:29m:01s]
Count partitions [======================Compute SuperK [==================================================] [02d:04h:29m:01s] Count partitions [================================== ================] [02d:04h:29m:01s]
Merge partitions [> ] [00:00s]

This back trace:

Backtrace: 1 0x00007f6987042520 (null) + 140091213489440 2 0x00007f69870969fc pthread_kill + 300 3 0x00007f6987042476 raise + 22 4 0x00007f69870287f3 abort + 211 5 0x00007f6987476f00 __gnu_cxx::verbose_terminate_handler() + 192 6 0x00007f698747543c (null) + 140091217892412 7 0x00007f698747548e (null) + 140091217892494 8 0x000055bde4189705 (null) + 94274063996677 9 0x00007f6987045495 (null) + 140091213501589 10 0x00007f6987045610 on_exit + 0 11 0x000055bde424b98a (null) + 94274064791946 12 0x00007f6987042520 (null) + 140091213489440 13 0x00007f69870e57f8 clock_nanosleep + 200 14 0x00007f69870ea677 nanosleep + 23 15 0x000055bde425d66b (null) + 94274064864875 16 0x000055bde43bd7f0 (null) + 94274066307056 17 0x000055bde4170260 main + 3312 18 0x00007f6987029d90 (null) + 140091213389200 19 0x00007f6987029e40 __libc_start_main + 128 20 0x000055bde41725e5 (null) + 94274063902181

And these error log:

[2024-04-02 08:42:46.390] [info] Run with Kmer<32> - uint64_t implementation [2024-04-02 08:42:46.615] [info] Compute configuration... [2024-04-02 08:42:46.616] [info] 204 samples found (408 read files). [2024-04-02 08:43:47.104] [info] Use 169 partitions. [2024-04-02 08:43:47.108] [info] Compute minimizer repartition... terminate called after throwing an instance of 'std::runtime_error' terminate called recursively what(): Unable to open /home/user/18tb/kmer_pa_bin/counts/partition_6/TA2703.kmer terminate called recursively terminate called recursively [2024-04-06 07:04:10.765] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log

The file TA2703.kmer is there and about the same size of the other kmer files. It seems that some of the first steps wee completed. How can I run kmtricks to try and complete only the missing steps?
Thank you

[dosshra]

Hello I ran the following code: kmtricks pipeline --file ./fof1 --run-dir ~/18TB/kmer_pa_bin --kmer-size 31 --mode kmer:pa:bin -t 10 --hard-min 2 --until count 2>kmtiks.err This part runs successfully. And then I ran the following code: kmtricks merge --run-dir kmer_pa_bin --mode kmer:pa:text -t 15 -v debug 2>merge_kmtrics.err The command terminates immediately while generating the files listed below. Any help would be much appreciated kmtricks_backtrace.log.txt merge_kmtrics.err.txt .