search

soedinglab / hh-suite

Remote protein homology detection suite.
https://bmcbioinformatics.biomedcentral.com/articles/10.1186/s12859-019-3019-7
GNU General Public License v3.0
529 stars 133 forks source link

HHblits process killed #253

Open sgorman90 opened 3 years ago

sgorman90 commented 3 years ago

Hi All,

I'm using HHblits in the context of Porter5 to help predict secondary structure. Porter executes HHblits with the flags -cpu 2 -n 3 -maxfilt 150000 -maxmem 8 -v 2. My system runs Pop!_OS linux with 16 gb memory and a Ryzen 5 5600x 6 core processor.

After I run a particular amino acid sequence (that I can make available if needed) HHblits ends early and the log file says the process was "Killed" (see below). Any advice would be greatly appreciated!

Killed

milot-mirdita commented 3 years ago

16GB RAM are probably too little for a database the size of the Uniprot20. The best solution would be to use a different machine or install more RAM.

You could try to restrict maximum alignment length by lowering -maxres, that might result in some potentially good hits not being found anymore though.

sgorman90 commented 3 years ago

I'm curious what the rule of thumb might be in deciding how much ram to use. Would something like 32 gb work better?

Thanks for your response!

Scott D. Gorman, Ph.D.

Postdoctoral Research Associate | Co-Department Rep | Structural Biology Department

St. Jude Children’s Research Hospital

o: 901-595-6136 | c: 617-842-0947 | Kriwacki Labhttp://www.stjuderesearch.org/kriwacki

From: Milot Mirdita @.> Sent: Sunday, March 28, 2021 6:44 PM To: soedinglab/hh-suite @.> Cc: Gorman, Scott Scott.Gorman@STJUDE.ORG; Author @.***> Subject: Re: [soedinglab/hh-suite] HHblits process killed (#253)

Caution: External Sender. Do not open unless you know the content is safe.

16GB RAM are probably too little for a database the size of the Uniprot20. The best solution would be to use a different machine or install more RAM.

You could try to restrict maximum alignment length by lowering -maxres, that might result in some potentially good hits not being found anymore though.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fsoedinglab%2Fhh-suite%2Fissues%2F253%23issuecomment-808980475&data=04%7C01%7Cscott.gorman%40stjude.org%7C353fe3b75e1e4babd23f08d8f243647b%7C22340fa892264871b677d3b3e377af72%7C0%7C0%7C637525718563165035%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=RSNjhEpyR%2BCZS88moWem%2FMI0NmCAnCFO9cr4zetO4vg%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAKQALETIILDX535BUL4UDHLTF7ENZANCNFSM4Z6KJQDQ&data=04%7C01%7Cscott.gorman%40stjude.org%7C353fe3b75e1e4babd23f08d8f243647b%7C22340fa892264871b677d3b3e377af72%7C0%7C0%7C637525718563175030%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=ZDNmdHVShkw%2FFbagkcl9%2BvJHBOY51ZFnyHH%2BQknWVP4%3D&reserved=0.

Email Disclaimer: www.stjude.org/emaildisclaimer Consultation Disclaimer: www.stjude.org/consultationdisclaimer

milot-mirdita commented 3 years ago

You could try using the SSE2 binaries, they need a bit less memory at the cost of a bit speed. I guess that you are quite close to the brink right now. I would go straight for 64GB though as RAM is relatively cheap these days.

nick-youngblut commented 2 years ago

I'm using a machine with 504 Gb of RAM and getting the same killed signal. The vmem used, based on htop is ~214 Gb. I'm querying against UniRef30. If I use a query of 336 AA, then hhblits completes successfully, but if I use a query that is 796 AA, then I get the killed signal.

Just before I get the killed signal, resident memory does quickly climb up to ~20 Gb (according to htop), and I get a lot of warnings:

- 07:36:42.879 INFO: HMMs passed 1st prefilter (gapless profile-profile alignment)  : 9214776

- 07:37:31.854 WARNING: Number of hits passing 2nd prefilter (reduced from 710225 to allowed maximum of 20000).
You can increase the allowed maximum using the -maxfilt <max> option.

- 07:37:31.921 WARNING: database contains sequences that exceed maximum allowed size (maxres = 20001). Max sequence length can be increased with parameter -maxres.

- 07:37:31.921 INFO: HMMs passed 2nd prefilter (gapped profile-profile alignment)   : 20000

- 07:37:31.921 INFO: HMMs passed 2nd prefilter and not found in previous iterations : 19795

- 07:37:31.921 INFO: Scoring 19795 HMMs using HMM-HMM Viterbi alignment

- 07:37:32.607 INFO: Alternative alignment: 0

- 07:37:32.658 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:32.692 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:34.370 INFO: 2000 alignments done

- 07:37:34.429 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:36.024 INFO: 4000 alignments done

- 07:37:36.030 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A6J3RMW2_consensus

- 07:37:36.030 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A6J3RMW2 LOW QUALITY PROTEIN: titin n=1 Tax=Tursiops truncatus TaxID=9739 RepID=A0A6J3RMW2_TURTR

- 07:37:36.030 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI000F443DA9 titin n=1 Tax=Lagenorhynchus obliquidens TaxID=90247 RepID=UPI000F443DA9

- 07:37:36.030 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI00122FA9B9 LOW QUALITY PROTEIN: titin n=1 Tax=Globicephala melas TaxID=9731 RepID=UPI00122FA9B9

- 07:37:36.030 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI0014411C62 titin n=1 Tax=Orcinus orca TaxID=9733 RepID=UPI0014411C62

- 07:37:36.034 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:36.063 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:36.090 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:38.162 INFO: 6000 alignments done

- 07:37:38.198 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:38.256 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:40.592 INFO: 8000 alignments done

- 07:37:40.654 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:42.708 INFO: 10000 alignments done

- 07:37:42.749 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A5E4B6R9_consensus

- 07:37:42.749 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A5E4B6R9 Uncharacterized protein n=2 Tax=Marmota monax TaxID=9995 RepID=A0A5E4B6R9_MARMO

- 07:37:42.749 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A5E4B6S7 Uncharacterized protein n=1 Tax=Marmota monax TaxID=9995 RepID=A0A5E4B6S7_MARMO

- 07:37:42.749 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A5E4B9M5 Uncharacterized protein n=1 Tax=Marmota monax TaxID=9995 RepID=A0A5E4B9M5_MARMO

- 07:37:42.752 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:42.778 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI000B511B8A_consensus

- 07:37:42.778 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI000B511B8A titin isoform X1 n=1 Tax=Aotus nancymaae TaxID=37293 RepID=UPI000B511B8A

- 07:37:42.779 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI00193DD89E LOW QUALITY PROTEIN: titin n=1 Tax=Saimiri boliviensis boliviensis TaxID=39432 RepID=UPI00193DD89E

- 07:37:42.781 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:44.765 INFO: 12000 alignments done

- 07:37:44.776 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:44.804 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:46.787 INFO: 14000 alignments done

- 07:37:46.796 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A480SN35_consensus

- 07:37:46.797 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_A0A480SN35 Titin isoform X6 n=1 Tax=Sus scrofa TaxID=9823 RepID=A0A480SN35_PIG

- 07:37:46.797 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI000A2B0F7C LOW QUALITY PROTEIN: titin n=1 Tax=Sus scrofa TaxID=9823 RepID=UPI000A2B0F7C

- 07:37:46.797 WARNING: maximum number of residues 32763 exceeded in sequence UniRef100_UPI0014550278 LOW QUALITY PROTEIN: titin n=1 Tax=Fukomys damarensis TaxID=885580 RepID=UPI0014550278

- 07:37:46.800 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:49.060 INFO: 16000 alignments done

- 07:37:49.119 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:49.165 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:51.143 INFO: 18000 alignments done

- 07:37:51.268 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:53.422 INFO: 19795 alignments done

- 07:37:53.430 INFO: Alternative alignment: 1

- 07:37:53.435 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:53.474 WARNING: Number of match columns too large. Only first 19999 match columns will be kept!

- 07:37:57.086 INFO: 5763 alignments done

- 07:37:57.114 INFO: Alternative alignment: 2

Killed
zj223332 commented 7 months ago

@nick-youngblut 遇到了和您类似的问题 13:29:14.096 WARNING: Warning in /home/vsts/work/1/s/src/hhalignment.cpp:3525: MergeMasterSlave: