Closed Erythroxylum closed 2 years ago
Yes, I think the model may be too large. It looks like you have 6 populations linked by migration, and if each population has n samples, this will require creating an array of size n^7 at some point.
mem_chunk_size
affects the number of SNPs simultaneously evaluated, but at a certain point even evaluating a single SNP at a time is too memory intensive.
One thing you can try is to make n smaller, say n=2 per population. You can use the SnpAlleleCounts.down_sample()
method to do this:
https://momi2.readthedocs.io/en/latest/api.html#momi.SnpAlleleCounts.down_sample
Well that is a helpful comment since n=20 for each pop. I will try it out. Thanks!
Hello, I am trying to estimate tdiv and tmigration parameters and receiving errors as above. The crux is that even when I have reduced the number of leaves and time parameters, I am not observing a decrease in the necessary memory allocation, it sometimes increases. I have altered set_data(mem_chunk_size=###), but this seems to have no effect on my problem.
Why am I not seeing the allocation and the array shape also reduce under a smaller model? Are all of these models just way too large? Is there another way to limit the memory allocation?
The largest model has seven leaves with seven size parameters and eleven time parameters and requests 1.7 TB of memory.![image](https://user-images.githubusercontent.com/5709486/106978477-f829c200-6721-11eb-82c5-23b15ef47f57.png)
A smaller model has six leaves and seven time parameters, requesting 4.18 TB of memory:![image](https://user-images.githubusercontent.com/5709486/107421513-ac8b6580-6adf-11eb-8ac2-d874a9cbc23d.png)
I am getting the following error:
Unix 5.4.0-52-generic Python 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0] numpy 1.19.5