tlemane
commented
2 years ago Hello,
Sorry for this issue. It seems that the maximum number of simultaneously opened files has been reached on your system. Since you have a large number of samples, kmtricks needs to open a lot of files during the merge.
You can try to increase this limit (see ulimit) and/or reduce the number of threads.
To avoid recomputing the first steps during your tests, I suggest to split the pipeline as follows:
1/ Count
kmtricks pipeline --file list_fastq_kmtricks --run-dir kmtricksDir --kmer-size 31 --hard-min 5 --mode kmer:count:bin --until count --cpr -t 122/ Merge
kmtricks merge --run-dir kmtricksDir --recurrence-min 10 --mode kmer:count:bin --cpr -t <nb_threads>This way you can do several merge tries without recounting the partitions.
I hope this help.
Téo
MorillonLab
Hello,
i installed kmtricks with conda, and i tried to run it on about 10000 fastq files, stored on an external drive.
here is the command line used : kmtricks pipeline --file list_fastq_kmtricks --run-dir kmtricksDir --kmer-size 31 --hard-min 5 --mode kmer:count:bin --recurrence-min 10 -t 12
and here are the message obtained from kmtricks :
[2022-04-16 20:17:08.096] [info] Run with Kmer<32> - uint64_t implementation [2022-04-16 20:17:08.320] [info] Compute configuration... [2022-04-16 20:17:08.320] [info] 3504 samples found (10512 read files). [2022-04-16 20:51:29.192] [info] Use 113 partitions. [2022-04-16 20:51:29.287] [info] Compute minimizer repartition... Compute SuperK [==================================================] [02d:11h:28m:38s]
Count partitions [==================================================] [02d:11h:28m:38s]
Merge partitions [> ] [00:00s]
terminate called after throwing an instance of 'std::runtime_error' terminate called after throwing an instance of 'std::runtime_error' terminate called recursively [2022-04-19 08:34:53.972] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log what(): Unable to open /media/ugo/Transcend3/scRNAseq_kmer/EMTAB_9067/kmtricks/counts/partition_1/ERR4147809.kmer what(): Unable to open /media/ugo/Transcend3/scRNAseq_kmer/EMTAB_9067/kmtricks/counts/partition_10/ERR4147809.kmer
[2022-04-19 08:34:53.990] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log [2022-04-19 08:34:53.990] [error] Killed after receive Aborted:SIGABRT(6) signal. Demangled backtrace dumped at ./kmtricks_backtrace.log. If the problem persists, please open an issue with the return of 'kmtricks infos' and the content of ./kmtricks_backtrace.log
i was not able to find the ./kmtricks_backtrace.log file i checked the file /media/ugo/Transcend3/scRNAseq_kmer/EMTAB_9067/kmtricks/counts/partition_1/ERR4147809.kmer, and it exists.
here is kmtricks infos: kmtricks v1.2.1
HOST - build host: Linux-5.4.0-54-generic run host: Linux 5.4.0-109-generic
BUILD - c compiler: GNU 11.2.0 cxx compiler: GNU 11.2.0 conda: ON static: OFF native: OFF modules: ON socks: ON howde: ON dev: OFF kmer: 32,64,96,128,160,192,224,256 max_c: 4294967295
GIT SHA1 / VERSION - kmtricks: 5c5c0b5 sdsl: c32874c bcli: 3e4f493 fmt: 0544a227 kff: 97d135e lz4: 4de56b3 spdlog: v1.2.1-1811-g5b4c4f3f xxhash: 6853ddc gtest: release-1.8.0-2774-g96f4ce02 croaring: v0.3.3-17-g2d5c927 robin-hood-hasing: 24b3f50 turbop: 4ab9f5b cfrcat: 2f9da97 indicators: v1.9-36-gcdcff01
Thanks for your help
Ugo