I am running into an error during the memory allocation for sparse matrix factorization in step1 (w/ sparse GRM), which seems to be related to the spsolve() function from RccpArmadillo. Here is the error message from the SAIGE output file:
use sparse kinship to fit the model
Not enough memory to perform factorization.
warning: spsolve(): memory allocation failure
Error: spsolve(): solution not found
Timing stopped at: 252.4 4.728 258.1
Execution halted
The error only occurs with larger sample sizes and isn't resolved by requesting more memory for the job. The SLURM output indicates that the max memory used is only 10GB regardless of how much memory is requested (I've requested up to 100GB with no avail). Interestingly, the problem arises after a very modest increase in sample size - I am able to successfully run a job with 80,204 samples (job uses a max of 16GB memory), but I get the error when sample sizes exceed 80,224 (with all other input files the same). Can you advise how I might proceed to try and surpass this memory allocation failure?
If helpful, I am using SAIGE v1.3.1 and using the step1 with the sparse GRM.
Hi Wei,
I am running into an error during the memory allocation for sparse matrix factorization in step1 (w/ sparse GRM), which seems to be related to the spsolve() function from RccpArmadillo. Here is the error message from the SAIGE output file:
The error only occurs with larger sample sizes and isn't resolved by requesting more memory for the job. The SLURM output indicates that the max memory used is only 10GB regardless of how much memory is requested (I've requested up to 100GB with no avail). Interestingly, the problem arises after a very modest increase in sample size - I am able to successfully run a job with 80,204 samples (job uses a max of 16GB memory), but I get the error when sample sizes exceed 80,224 (with all other input files the same). Can you advise how I might proceed to try and surpass this memory allocation failure?
If helpful, I am using SAIGE v1.3.1 and using the step1 with the sparse GRM.
Any advice would be much appreciated!
Thank you, Emily