search

sib-swiss / pftools3

A suite of tools to build and search generalized profiles
GNU General Public License v2.0
10 stars 7 forks source link

pfsearchV3 crashes on big databases #21

Open hofmank0 opened 3 years ago

hofmank0 commented 3 years ago

I have been using an older version of pfsearchV3 for a long time without any problems. However, the old version did not correctly identify the number of cores on my new processor, so I just downloaded and compiled the current version. Initially, everything looked fine (after getting used to the renamed command line parameters), but I notice that the program crashes whenever I run a long profile against a long database.

There is no specific error message. I watch the process (by top) to use more and more memory, and after a few seconds hovering at 95% RAM usage (64MB total), the program stops and the terminal just says "killed".

I am using Version 3.2.6 built on Aug 10 2021 (compiled with standard option) and notice the problem in heuristic mode, a command line like pfsearchV3 -H 38388 -L -1 hect.prf /database/nr.seq > output

Edit: the profile is 348aa long, the database is more or less identical to uniprot (114GB fasta file). The problem occurs on two different computers. The old version has no problems, but does not use all cores on the Ryzen 5950x processor (but all cores on an older Core i7 processor)

Any idea what I might have done wrong?

smoretti commented 3 years ago

Hi @hofmank0 we will take a look.

It can be related to #20 or to long profile. An idea @euphemizm @beatrice79 ?

hofmank0 commented 3 years ago

I resorted to using the old binary of pfsearchV3, which can be convinced to use all cores by issuing "--no-affinity". This version works even for larger profiles against the entire uniprot database. The current version just generates a memory overflow after some time. This is not a good solution, though. I would prefer to run the most recent version. Also, the old version is not immune to memory overflow: if I run multiple instances at the same time, I also get random process killing due to memory problems.

duboism commented 3 years ago

It can be related to #20 or to long profile.

I could be wrong but it looks like an excessive memory usage; #20 causes the program to stop immediately so I don't think it's related.

smoretti commented 3 years ago

@hofmank0 could you provide your profile?