tsackton
commented
3 months ago Possible solution:
- refactor concat step to create a temp vcf file, using the -a and -d options and a file of files, but not doing the sort
- sort in a second step with high memory
Testing on a dataset with ~12k intervals: the sort/gather step at the end (to collect each genotyped interval into the final vcf) fails. It is not clear if this is a temp file issue, a memory issue, or just a problem with the way we handle very large inputs to bcf concat.
Probably the solution is multiple rounds of merging, which we can investigate. For now, posting this issue in case other people have problems with datasets with >10k intervals to merge.