search

sgkit-dev / sgkit

Scalable genetics toolkit
https://sgkit-dev.github.io/sgkit
Apache License 2.0
236 stars 32 forks source link

Investigate further chunking improvements for better GWAS performance #461

Open tomwhite opened 3 years ago

tomwhite commented 3 years ago

454 helped with GWAS performance, but as mentioned in https://github.com/pystatgen/sgkit/issues/390#issuecomment-768411149, there is scope for further improvement since the transfer time is still a significant proportion of the compute time.

tomwhite commented 3 years ago

See https://github.com/pystatgen/sgkit/issues/448#issuecomment-780655217 for the best chunk size found so far