search

higlass / higlass-server

Server component for HiGlass that manages and serves tiled data
MIT License
19 stars 21 forks source link

Error with too many open files #127

Closed sonnyarora closed 4 years ago

sonnyarora commented 4 years ago

I am running higlass through docker on CentOS 7.7.1908. I ran into an issue when I had several files loaded in HiGlass. In particular, when I have 8 bigWigs and an mcool file in HiGlass, after zooming in and out, I see the following error in the higlass container in /data/log/hgs.log :

[Errno 24] Too many open files: '/data/media/mm10.tsv'

Running some tests: With no loaded files, I check how many open files all the uwsgi processes have open:

lsof -p 2640,2649,2651,2652,2653 | wc -l

I get 553. After loading a single big wig, the number rises to: 1020. As I zoom in, the number keeps increasing: 1020,1031,1259. Passing this input to grep to search only for the names of the bigWig I have loaded in higlass, I see more lines appear as I zoom. I see the same pattern of the number of open files increasing as I zoom in different locations with all 8 bigWigs and the mcool file loaded until I get the above error.

The limit on number of open files for a single process set by default for my CentOS is 1024. If I increase this limit, I don't run into the same problem. Setting this limit high enough is a good enough solution for me for this problem.

nvictus commented 4 years ago

Thanks for reporting this! This seems to be caused by resource leakage when pybbi raises exceptions internally. Will work on a fix here: https://github.com/nvictus/pybbi/issues/10

nvictus commented 4 years ago

This is now fixed in pybbi 0.2.2

sonnyarora commented 4 years ago

Thanks for taking care of that! What are the next steps to get higlass-server to use the new version of pybbi?

nvictus commented 4 years ago

If you're using the docker image, it will be distributed in the next release, probably in the next day or so. If you're running higlass-server in a local environment, you just need to upgrade pybbi via pip.

pkerpedjiev commented 4 years ago

Should be updated in v0.6.39 of the Docker container 👍 Please let us know if you still encounter this issue.

And thanks for reporting it :-)

sonnyarora commented 4 years ago

Hey, thanks for taking care of that! I'm not seeing any open bigwigs in lsof anymore. Also, extra file handles don't appear to be opened when zooming. Marking this as closed.