search

junyangq / snpnet

snpnet: Fast and scalable lasso/elastic-net solver for large SNP data
32 stars 15 forks source link

Memory issues #37

Open privefl opened 3 years ago

privefl commented 3 years ago

I'm trying to run snpnet on UK Biobank data with 340K indivs and 500K variants (pgen is only 22GB). I use nodes with 16 cores and 128 GB RAM. I have tried to use parameter mem (trying 128000, 100000 and 50000) but always have an "out of memory" issue. What should I do?

privefl commented 3 years ago

I've added some print(gc(reset = TRUE)) in the code and asked for 500 GB instead. I still get this error and the gc reports more than 350 GB used. Is this normal?

biona001 commented 2 years ago

I am having a similar issue, did you ever figure out what to do?

privefl commented 2 years ago

nope

biona001 commented 2 years ago

@junyangq could you give me some guidelines on how much RAM I need to run ~350K samples and 600K SNPs?

junyangq commented 2 years ago

@biona001 I'm sorry for the delay. It depends on how many variables you'd expect the model to select (to be precise, the selected ones + the candidate ones for the next step). Since we store the candidate variables in memory and glmnet copies the model matrix, if we expect to have M variables, we would need memory that is about 2 times the memory to store 350K * M floating point numbers. For example, if you expect 10K variables, then that will end up using memory of 52.2G + some small overheads.

junyangq commented 2 years ago

@privefl I only realized this post just now. So sorry for the big delay in response and any inconvenience. If the model ends up being big, it can run into memory issue. Since we store the selected variables and additional candidate ones for next step as normal floating numbers in R, it could take up some memory. A rough guidance would be memory that can hold up 2 M N floating point numbers, where M is the number of variables and N is the number of sample. I hope this helps.

The mem argument is passed to PLINK when we use it to compute some summary statistics and check KKT conditions, and does not currently safe guard the number of selected variables for now. Thanks for bringing this up, and we expect to fix this soon.

RossDeVito commented 5 months ago

Any update on that fix? I'm trying to use BASIL on the UKB cloud and having this problem.