Closed jonbrenas closed 7 months ago
Thanks Jon, that shouldn't be happening, especially when you're only providing a relatively small genome region for the diversity computation.
The higher memory use is triggered by the site_class
parameter:
Hi @jonbrenas, just to mention I have a solution to this over in #501.
For some reason, running af.diversity_stats uses an enormous amount of RAM (at least on Colab and Datalab). It crashes even my Pro sessions with 12GB of RAM. I tried: af1.diversity_stats( cohorts="cohort_admin2_year", cohort_size=10, region="3RL:1,000,000-2,000,000", sample_query="country == 'Gabon'", site_mask='funestus', site_class="CDS_DEG_4" ) and the same for Ghana (resp. 40 samples in one cohort and 31+29 in 2 cohorts) that fail in the same way.
Trying the same thing with Ag works fine (though, it also crashes due to RAM use if 'site_mask' is dropped). I tested on Gabon with 5 cohorts including 3 that have more than 60 samples.