Closed OgnjenMilicevic closed 3 years ago
Thanks for the question -- I clarified the doc as follows:
The working directory, used intensively as temporary space for external data sorting and scanning, should be on a local SSD to minimize I/O starvation of the available CPUs.
We're recommending local SSDs since that's what we deploy, but the real point is just that the node's storage subsystem should be fast enough to avoid leaving a lot of CPU cycles on the table due to I/O saturation. The I/O pattern of the individual threads isn't super random, it's just that there can be a lot of them at once.
Ecah GVCF input file is read sequentially, so it should be fine to read them from network storage (albeit many threads read different files at once). The output BCF is also streamed sequentially. It's the intermediate phase (managed by RocksDB) that hits the working directory storage pretty hard.
Thank you for a prompt and thorough answer.
Closing this!
Your performance consideration article states that due to a large number of intensive sorting operations one should use local SSD.
Can you clarify this further, preferably in the official documentation:
Thanks!