search

EarthScope / slinktool

Other
18 stars 15 forks source link

fix for "too many open files" bug with slinktool-4.3 #7

Closed seb-staneo closed 1 year ago

seb-staneo commented 4 years ago

we have found a problem with slinktool when dealing with a large number of channels with an output in minute-files (-A %...%M). The problem is that the number of open files reaches the limit (1024 is the default for many unices). It is related to the cache management of the output files. I have attached a patch against slinktool-4.3 that fixes the problem (and the modified dsarchive.c).

dsarchive.c is the only affected file, the fix consist of two main changes : 1/ use the MSR time and not the current system time to determine if a entry in the list of datastreams should be closed by the original management of datastreams (checking the oldness of the last access to the datastream) 2/ continuously count the number of open files, if a threshold is reached, then close somes of the oldest datastreams (actually, they are not closed but prepared to be closed by the original management of datastreams by setting the modtime field to 0)

A sample command to reproduce the problem is the following : slinktool -s "???" rtserve.iris.washington.edu -A "1minute_files/%s%n%l%n-%c-%y%j-%H-%M.mseed" -tw 2020,01,01,00,00,00: After some (short) time, original slinktool-4.3 does not output data any more but issues the following messages : opening new data stream file, Too many open files error: cannot write data to archive

Finally, the interruption of the process (SIGTERM/ctrl-C) leads to a segmentation fault (I did not try to find out where/why). kind regards sébastien

patch-and-source.zip

chad-earthscope commented 4 years ago

Thanks @seb-staneo, I appreciate the contribution. I'll have a look at this and a fix in the next release.

jschaeff commented 1 year ago

We had to deal with the same issue. We had to organize a bunch of slinktool processes with completely defined selector files based on the channels we need to collect.

I did not try your patch, I just discovered it. I'll give it a try.

chad-earthscope commented 1 year ago

@seb-staneo Thanks for the patch. Unfortunately this looks mostly like a duplication of capability in https://github.com/EarthScope/slarchive/blob/main/src/dsarchive.c

I've added a section to the README to recommend the use of slarchive instead of this program for long term collection of data streams: https://github.com/EarthScope/slinktool/tree/4.x

These two code bases have diverged and I do not want to maintain both with enhancements fixing the same issue differently. To avoid this confusion (and extra work) I plan on removing the "archiving" feature in slinktool in the next major release.

I strongly recommend using slarchive for archiving miniSEED via SeedLink, which has long addressed the "too many open files" issue.