Open privefl opened 3 years ago
I've added some print(gc(reset = TRUE))
in the code and asked for 500 GB instead.
I still get this error and the gc reports more than 350 GB used.
Is this normal?
I am having a similar issue, did you ever figure out what to do?
nope
@junyangq could you give me some guidelines on how much RAM I need to run ~350K samples and 600K SNPs?
@biona001 I'm sorry for the delay. It depends on how many variables you'd expect the model to select (to be precise, the selected ones + the candidate ones for the next step). Since we store the candidate variables in memory and glmnet copies the model matrix, if we expect to have M variables, we would need memory that is about 2 times the memory to store 350K * M floating point numbers. For example, if you expect 10K variables, then that will end up using memory of 52.2G + some small overheads.
@privefl I only realized this post just now. So sorry for the big delay in response and any inconvenience. If the model ends up being big, it can run into memory issue. Since we store the selected variables and additional candidate ones for next step as normal floating numbers in R, it could take up some memory. A rough guidance would be memory that can hold up 2 M N floating point numbers, where M is the number of variables and N is the number of sample. I hope this helps.
The mem
argument is passed to PLINK when we use it to compute some summary statistics and check KKT conditions, and does not currently safe guard the number of selected variables for now. Thanks for bringing this up, and we expect to fix this soon.
Any update on that fix? I'm trying to use BASIL on the UKB cloud and having this problem.
I'm trying to run snpnet on UK Biobank data with 340K indivs and 500K variants (pgen is only 22GB). I use nodes with 16 cores and 128 GB RAM. I have tried to use parameter
mem
(trying 128000, 100000 and 50000) but always have an "out of memory" issue. What should I do?