bolosky
commented
3 years ago It shouldn't do that. The sort memory usage should be on top of the alignment amount, but it shouldn't be much more the amount you give it with -sm.
What version are you running? IIRC, some very old versions had problems with extra memory usage.
You can free up some memory by having SNAP release the memory for the index before starting sort by using the -di switch, which might help if you're close.
One other thing that might be happening is that you've got such a large number of duplicates of a particular read that they're taking up all the memory; they have to all be resident at one time to process them properly. If that's what's happening there's not that much you can do aside from using -di, unless you're willing to turn off duplicate marking altogether by using -S d. That said, taking the output and putting it through Picard to do duplicate marking may not help, because aside from being painfully slow, it can also use huge amounts of memory if you have millions of duplicates of a single read.
--Bill
From: teerjk @.> Sent: Wednesday, May 5, 2021 2:00 PM To: amplab/snap @.> Cc: Subscribed @.***> Subject: [amplab/snap] controlling memory usage for sort/index/markdup (#133)
I'm trying to control memory usage during the sort/index/markdup step (-so). I have set -sm, but memory usage is generally the same no matter what setting I give. -sm ranges from 0.5-4 as well as not setting -sm give similar usage. Only setting -sm to 20 seemed to provoke a change (ran out of memory on 165GB node). Memory usage also seemed similar with 4, 8, and 16 threads. The memory usage generally approaches all available on the node, as reported by "RES" in top as well as by the cluster software (PBS/Torque). On a 64GB node, usage was ~63G; on a 165GB node usage was ~156GB. Usage is pretty stable during alignment (40-60GB), but then increases to all available once "sorting" is output on stdout. I looked through the code, and didn't see anything immediately obvious, but am not expert enough in C++ to really dig in to the lower level stuff. Any thoughts or hints? Thanks!
- You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Famplab%2Fsnap%2Fissues%2F133&data=04%7C01%7Cbolosky%40microsoft.com%7Cf8df9bcc0f874c0cab4208d910006b19%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637558416250558899%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=aLcacCLmtwOuJba%2BCSU8wlch0geHXITMpSPCzx0VAF8%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAAHPTWMRH2CDPBQVJXUGXTDTMGPVNANCNFSM44FVEPQQ&data=04%7C01%7Cbolosky%40microsoft.com%7Cf8df9bcc0f874c0cab4208d910006b19%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637558416250568892%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=QFOCgbNhloOH3n3cUDuqn%2FVvs6LBfA8oSzLFFkQhagw%3D&reserved=0.
teerjk
arun-sub
I'm trying to control memory usage during the sort/index/markdup step (-so). I have set -sm, but memory usage is generally the same no matter what setting I give. -sm ranges from 0.5-4 as well as not setting -sm give similar usage. Only setting -sm to 20 seemed to provoke a change (ran out of memory on 165GB node). Memory usage also seemed similar with 4, 8, and 16 threads. The memory usage generally approaches all available on the node, as reported by "RES" in top as well as by the cluster software (PBS/Torque). On a 64GB node, usage was ~63G; on a 165GB node usage was ~156GB. Usage is pretty stable during alignment (40-60GB), but then increases to all available once "sorting" is output on stdout. I looked through the code, and didn't see anything immediately obvious, but am not expert enough in C++ to really dig in to the lower level stuff. Any thoughts or hints? Thanks!