Service maxes out number of open files

[wlymanambry]

This service was working great for me for a couple of days but then it maxes out the OS allowable files open at the same time and fails.

Gives a Failed to open FASTA index ... "Too many open files"

A lsof | wc -l shows -> 65,863 files, a huge number of db.sqlite3 as well as individual fasta files.

Is there a workaround for this?

[theferrit32]

Hi @wlymanambry can you give a sense of what kind of workload you are running? Do you have a seqrepo-rest-service process running continuously over a couple days? Are you sending a lot of requests in rapid succession?

[wlymanambry]

I have a dataset coming back from a lambda function that is being kept warm so I am hitting seq repo 100 times every 5 minutes as part of this. The seqrepo-rest-service is running continuously. (should this not be the case?)

[theferrit32]

It should be fine to leave running continuously. I'm just trying to guess whether the issue you're seeing might be resolved by the eventual fixes here:

https://github.com/biocommons/biocommons.seqrepo/issues/112#issuecomment-1611674050

[wlymanambry]

Reading over that thread you linked. It looks promising.

[davmlaw]

"Too many open files" is usually from the operating system. Check ulimit -n

while waiting for the fix above, you could temporarily adjust this via /etc/security/limits.conf

[wlymanambry]

Thanks. Are there any performance considerations for this? It will end up having hundreds of thousands of open files after a week or so.

[davmlaw]

I don't know... I think the overhead per file isn't much, and there are only like 0.5M Refseq transcripts.

It's probably fine on modern machines, maybe check RAM after a week.

[reece]

This appears to be a duplicate of #12. I'm going to close and we can continue discussion there.

[reece]

@wlymanambry

Thanks. Are there any performance considerations for this? It will end up having hundreds of thousands of open files after a week or so.

Increasing the fd limit will help, but only delay the failure. You won't have enough fds to stave off the issue forever, unfortunately.

[theferrit32]

@wlymanambry Can you try using the recent tag 0.2.2 in this repo and see if resolves the issue you were seeing? This is also available in pypi seqrepo-rest-service==0.2.2 and in dockerhub biocommons/seqrepo-rest-service:0.2.2

[wlymanambry]

@wlymanambry Can you try using the recent tag 0.2.2 in this repo and see if resolves the issue you were seeing? This is also available in pypi seqrepo-rest-service==0.2.2 and in dockerhub biocommons/seqrepo-rest-service:0.2.2

Will do as soon as I get a chance. Thanks!

[wlymanambry]

Sorry for the lengthy delay. After pulling this version it appears I'm still ending up with open files but at a much slower rate:

[wlymanambry]

Update on this. I've been hitting the service now for several weeks and it looks like the number of open files is staying constant.