Open bsmith89 opened 5 months ago
Just to further confirm that the number of open files is the problem:
$ ulimit -a
core file size (blocks, -c) 0
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) 4075957
max locked memory (kbytes, -l) 64
max memory size (kbytes, -m) unlimited
open files (-n) 1024
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) 819200
real-time priority (-r) 0
stack size (kbytes, -s) 8192
cpu time (seconds, -t) unlimited
max user processes (-u) 4075957
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited
With 12 threads, I still haven't exceeded 1000 open files (currently running, but it's gotten past the point where the 24-thread run consistently fails).
Run with 12 threads did eventually fail (4 hours later):
$ kmtricks pipeline --kmer-size 111 --hard-min 0 --share-min 1 --soft-min 2 --recurrence-min 3 --file data/group/xjin/r.proc.kmtricks_input.txt --run-dir $tmpdir --mode kmer:count:text --threads 12
[2024-03-22 09:32:04.718] [info] Run with Kmer<128> - uint64_t[4] implementation
[2024-03-22 09:32:04.781] [info] Compute configuration...
[2024-03-22 09:32:04.781] [info] 276 samples found (552 read files).
[2024-03-22 09:32:44.950] [info] Use 156 partitions.
[2024-03-22 09:32:45.632] [info] Compute minimizer repartition...
Compute SuperK [==================================================] [02h:22m:42s]
Count partitions [==================================================] [02h:22m:44s]
Merge partitions [> ] [00:00s]
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called after throwing an instance of 'std::runtime_error'
what(): <DIR>/counts/partition_2/xjin_AS_P2R2c.kmer
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
terminate called recursively
[2024-03-22 12:00:21.549] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log
[2024-03-22 12:00:21.549] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log
[2024-03-22 12:00:21.550] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log
backtrace:
Backtrace:
1 0x00007f8d88e41090 (null) + 140245863764112
2 0x00007f8d88e4100b gsignal + 203
3 0x00007f8d88e20859 abort + 299
4 0x00007f8d89213f00 __gnu_cxx::__verbose_terminate_handler() + 192
5 0x00007f8d8921243c (null) + 140245867766844
6 0x00007f8d8921248e (null) + 140245867766926
7 0x000055a817e6c705 (null) + 94180443866885
8 0x00007f8d88e448a7 (null) + 140245863778471
9 0x00007f8d88e44a60 on_exit + 0
10 0x000055a817f2e98a (null) + 94180444662154
11 0x00007f8d88e41090 (null) + 140245863764112
12 0x00007f8d88edb23f clock_nanosleep + 223
13 0x00007f8d88ee0ec7 nanosleep + 23
14 0x000055a817f3f95b (null) + 94180444731739
15 0x000055a8180b14d0 (null) + 94180446246096
16 0x000055a8180b2165 (null) + 94180446249317
17 0x000055a8180b9443 (null) + 94180446278723
18 0x000055a817e52afb main + 1419
19 0x00007f8d88e22083 __libc_start_main + 243
20 0x000055a817e555e5 (null) + 94180443772389
I'm getting the following error when I run kmtricks with 276 samples:
./kmtricks_backtrace.log:
infos:
I don't get the same error when I use a subset of 8 samples (instead of 276), nor when I use just 12 threads (instead of 24). This seems pretty clearly related to Issue #15 , where kmtricks is opening too many files. Indeed,
lsof
confirms this, with the crash occuring just as the number of open files ramps up. When I use 12 threads, <1000 files are opened and it doesn't crash.While using fewer threads works, I'd love a solution that maintains the high parallelization during other steps. Can you suggest a way to run the kmtricks pipeline so that the superkmers computation step doesn't open too many files, but I get as much parallelization as possible?
Thanks for your help, and for building a valuable tool!