[feature request] Option to use temporary files for `bcftools merge`

[hjeremyli]

When merging larger numbers of VCFs, memory usage is often considerable; ad hoc testing indicates that merging ~60k variants across 1.5k samples uses >120Gb of memory. It would be useful to have an option like the -T option in bcftools sort which allows using temporary files instead of performing all operations in RAM in order to make this computation tenable for lower memory machines.

[pd3]

That's odd, the program should not be requiring that much memory, for each file it keeps in memory only the header, the index and a few lines with the same position. In a test that I ran just now it merged 1,500 files requiring only 5GB of memory.

Possible things to do:

• make sure you are using the latest (or at least a recent) version of bcftools, just because we will not be looking into problems that are not present anymore
• if the indices or the headers are huge, and that's what makes the memory requirements unreasonably big, you could do the merging hierarchically in batches

[hjeremyli]

@pd3 Thanks for the response. Updating to the version on master seems to have resolved this; it is now using a sane amount of memory (~3-5Gb) for the same task. I was previously using v1.16.