Rfsearch big models - Githubissues

nawrockie commented 2 years ago

Changes to rfsearch for big models, motivated by my tests of rfsearch on the big LSU and SSU models (RF02543 is the biggest)

memory requested for cmcalibrate is now estimated using the cmcalibrate --memreq option to determine required memory
memory requested for cmsearch jobs is now increased for models with W>=2000 (currently only 6 models in Rfam 14.8 have W>= 2000). Fortunately, looks like the cluster has at least some hosts that can accommodate these large memory requirements
- For W>=4000 20Gb RAM per thread is requested (only RF02543)
- For W>=3000 16Gb RAM per thread is requested (RF02541 and RF02540)
- For W>=2000 12Gb RAM per thread is requested (RF03079, RF01960, RF02462)
-cgtailn option added to rfsearch.pl for RF02543 because calibration fails with default options: -cgtailn 200 is required (currently) to get RF02543 calibration to work

blakesweeney commented 2 years ago

Thanks Eric! This looks like a very useful improvement to the pipeline, I know we have had lots of trouble with the larger families. @emmaco can you test this out and see how it goes?

emmaco commented 2 years ago

Yep I’ll try it out then merge.

Rfam / rfam-family-pipeline

Rfsearch big models #85