search

sgkit-dev / sgkit

Scalable genetics toolkit
https://sgkit-dev.github.io/sgkit
Apache License 2.0
233 stars 32 forks source link

Reduce default bgen rechunking worker memory default #424

Open eric-czech opened 3 years ago

eric-czech commented 3 years ago

https://github.com/pystatgen/sgkit/blob/f2567cb236f135d34a6b1ce34f16da3d690a009a/sgkit/io/bgen/bgen_reader.py#L427

This should be something significantly lower. 500MB-1GB is what I had to use to make rechunking work over UKB without memory errors (substantially more memory than this limit is used at times by workers).

tomwhite commented 3 years ago

substantially more memory than this limit is used at times by workers

Do you think this could be a problem with rechunker?

eric-czech commented 3 years ago

substantially more memory than this limit is used at times by workers

Do you think this could be a problem with rechunker?

I'm not sure -- All I know for certain is that the memory utilization across nodes running rechunking definitely exceeded n_workers * max_mem. Anecdotally, 4x whatever limit I gave it was closer to what was actually necessary for UKB BGEN. Here is a readout from a 32 vCPU node with 128G RAM (32 workers):

Screen Shot 2020-10-09 at 4 24 05 PM

The limit in this case was 1GB per worker (possibly 500MB) which would imply a max utilization of 32 GB but you can see usage creep up to 64GB.