search

splatlab / squeakr

Squeakr: An Exact and Approximate k -mer Counting System
BSD 3-Clause "New" or "Revised" License
85 stars 23 forks source link

/src/count.cc can silently drop input files #43

Open t-kranz opened 4 years ago

t-kranz commented 4 years ago

At l.299 a file reader gets silently deleted if reader::getFileReader() fails (at least when it does at l.192 in /include/reader.h).

I noticed this happening, because i increased ip_files limit/capacity too get around issue 42 and actually added an output at l.299 in count.cc.

In my particular case the number of (bz2 compressed) files i gave to squeakr count exceeded the configured maximum allowed open file count (ulimit -n). So after reaching that limit fopen() fails (after adding onother output - with "too many open files").

In that case (and from what i understand in every other case that would prevent a file from beeing opened) false is returned and the file reader just gets deleted. With nothing really indicating that this is happening!

I cant see any difference in reader.h l.184, l.187 (and l.198) so i think with plain fastq and gz compressed input the same could happen (but i havent tested it).

I think a warning output or even an error should be the response of count.cc after the delete at l.299 (maybe the error from fopen() would be nice too).

If i get the chance i will test if i can trigger this by just removing the reading rights of an input file.

Sincerely tkranz

rob-p commented 4 years ago

Hi @t-kranz ,

Thank you for the detailed report and the ensuing investigation into the source code! Paging @prashantpandey to take a look at this please.

—Rob

t-kranz commented 4 years ago

I checked with input files for which i removed the reading rights and as expected the silent drop happens with those as well.

I append a zip containing a git patch (fix#43.patch) for this. With that, a warning gets issued when squeakr count drops a file and reader.h prints the actual error with perror().

fix#43.zip