beftools merging 3540 individual files

[Biophylo2001]

Hi all, I have 3540 sequences individual vof files. I first created a list for all my vof files using this command: Is input_file*.vcf.gz list_of_file.txt

Then i created index for them and merged with this: beftools merge - merge.txt -Oz -0 merge.vcf.gz

Indexing command created tbi fles for each vof files however it shows this error : [E.:hts_idx_load3] Could not load local index file 'EP|_ISL_13330943.vcf.gz.tbi' Failed to open EP|_ISL_13330943.vcf.gz: could not load index

If i index this file individually, then it shows the same error, however it says failed to load csi file this time. I check my ulimit and it's 7823 while the number of items in my folder is just 7079. How do i fix this?

[pd3]

If i index this file individually, then it shows the same error, however it says failed to load csi file this time.

Do I understand this correctly that the file can be indexed, the index works individually on this file (for example bcftools merge --force-samples file.vcf.gz file.vcf.gz works), only the merging of all 3540 files is failing?

I check my ulimit and it's 7823 while the number of items in my folder is just 7079.

Those numbers are not that far apart, the limit is on ALL open files by the user, not just by one process. Try increasing the limit.

If nothing works, you might consider merging the files in batches.

[Biophylo2001]

Hi @pd3 . I increased my ulimit and it solved the problem.

But Yes , the indexing step worked but the merging step did not

[pd3]

Great, glad it worked