search

samtools / bcftools

This is the official development repository for BCFtools. See installation instructions and other documentation here http://samtools.github.io/bcftools/howtos/install.html
http://samtools.github.io/bcftools/
Other
672 stars 240 forks source link

bcftools concat retry mechanisms? #2245

Open graphenn opened 2 months ago

graphenn commented 2 months ago

For the bcftools concat function, could we add some retry mechanisms when directly accessing s3 files? I am using bcftools concat to directly concatenate a large number of vcf files. Due to the huge volume, if there are network fluctuations during the concatenation process, the connection is prone to aborting halfway, resulting in a "software caused connection abort" error. This makes it difficult to successfully complete the concatenation. Would it be possible to add some retry mechanisms, such as retrying three times, which could significantly improve this situation of concatenation failures?

Currently, I am unable to reproduce this issue, presumably because the network fluctuations have subsided. Unfortunately, I didn't save the log. I recall that before the "software caused connection abort" error, there was an "[E::hts_hopen]" error

pd3 commented 1 month ago

I can imagine this, probably best done at htslib level and maybe controlled by an environment variable.

pd3 commented 1 month ago

Just to link it to a related issue in htslib: https://github.com/samtools/htslib/issues/1424