Getting [mpileup] and [E::hts_open_format] errors when running motus snv_call

[CLGLynnsane]

Hello, When running motus snv_call on maybe ~1000 .bam files, I consistently get the following sorts of errors:

[E::hts_open_format] Failed to open file </path/to/bamfile>

and

[mpileup] failed to open <path/to/bamfile>: Resource temporarily unavailable

The files that end up failing to open are not consistent between different run.

I have never received these errors when running in smaller batches, so I assume this is some sort of resource problem. It doesn't seem like I am using too much memory and the files it is trying to open are definitely there. ulimit -n is 1024, which is less than the number of bam files the process might be opening. What might be my issue?

[hjruscheweyh]

Hi!

I agree. It sounds like a problem with the maximum number of open files set by the ulimit option. Do you have the option to increase the ulimit temporarily?

Hans

[CLGLynnsane]

Thanks for the quick reply!

I did as you suggested and increased the ulimit option to 4096, however I still have the same errors in my log. Do you think this is not enough for ~1000 .bam files? Or is there some other issue you might suspect?

[hjruscheweyh]

I think it would be best to involved @LucasPaoli. He ran the map_snv step with a very large number of bam files. For me this reads more like a file system limitation.

[LucasPaoli]

Jumping in a bit late, but I recall having to change the ulimit at some point. I would recommend setting ulimit >> 2*Number of bamfiles. Let us know if you're still experiencing some issues.

Best,

Lucas

[hjruscheweyh]

No further issues were reported. Closing this issue.