search

mxmlnkn / rapidgzip

Gzip Decompression and Random Access for Modern Multi-Core Machines
Apache License 2.0
364 stars 7 forks source link

Only use available cores #6

Closed mxmlnkn closed 1 year ago

mxmlnkn commented 1 year ago

When running pragzip on an high-performance system managed with e.g. SLURM, only a subset of all cores might actually be usable. However, pragzip simply uses std::thread::hardware_concurrency, which does not account for that. Oversubscribing might degrade performance and increase the memory usage with no computational benefit. Furthermore, hardware_concurrency is supposedly only a hint and may also return 0, in which case only 1 thread will be used by pragzip. Both of these issues should be worked around by using the Linux-specific sched_getaffinity among others.

mxmlnkn commented 1 year ago

Fixed with https://github.com/mxmlnkn/indexed_bzip2/commit/eb13dbad1a60ccd400727071a4c445602e10be04. Will be included in pragzip 0.5.0.