search

galaxyproject / ephemeris

Library for managing Galaxy plugins - tools, index data, and workflows.
https://ephemeris.readthedocs.org/
Other
27 stars 38 forks source link

Lots of file descriptors open while testing installed tools #166

Open HugoJH opened 3 years ago

HugoJH commented 3 years ago

While doing shed-tools test over my galaxy instance, had a problems of reaching the limit of maximum number of open files. For the sake of keep running the test, I modified the limit editing /etc/security/limits. Relaunched the shed-tools test command and waited enough time to check the number of open files at that point.

I did lsof | grep shed-tool | grep -v deleted| wc and found out that there where more than 3000 opened files by the shed-tools process. Seems like the files opened for each test are not gettting closed once the test finishes. There should be a mechanism to close all unnecesary file descriptors once the test finishes.

rhpvorderman commented 3 years ago

Hi @HugoJH, I checked out the code and I cannot find the open statement that causes this. Can you give us some more information?

afgane commented 3 years ago

@almahmoud, you've experienced this behavior as well, right?

almahmoud commented 3 years ago

Yes! The problem seems to be in tool-util I think, cause it came up when running the tool tests directly not just with ephemeris. A temporary workaround was setting ulimit -n 20000 (or some other really high number depending on how many tests are run total)

mvdbeek commented 3 years ago

Are these files or sockets that are open ? Can you also check what version of galaxy-tool-util you've got in your environment ? (pip freeze| grep galaxy-tool-util)

almahmoud commented 3 years ago

The last time I tried it was installed from source from https://github.com/galaxyproject/galaxy/pull/10767/files. And as far as I can tell, it's at least files, not sure if sockets as well:

Small snippet of lsof 2>/dev/null | grep ubuntu | sort (from an older run with ephemeris)

shed-tool 11126 11143           ubuntu 1367u      REG                8,1       47      34440 /tmp/tmpzvhs83bdmultiCovBed_result1.bed (deleted)
shed-tool 11126 11143           ubuntu 1368u      REG                8,1      259      34448 /tmp/tmp750h1vdqnucBed_result2.bed (deleted)
shed-tool 11126 11143           ubuntu 1369u      REG                8,1       47      34442 /tmp/tmpsglw1g2xmultiCovBed_result1.bed
shed-tool 11126 11143           ubuntu 1370u      REG                8,1      161      34446 /tmp/tmpfotfnew7nucBed_result1.bed
shed-tool 11126 11143           ubuntu 1371u      REG                8,1      259      34450 /tmp/tmpip5lan0gnucBed_result2.bed
shed-tool 11126 11143           ubuntu 1372u      REG                8,1      280      34452 /tmp/tmp7b58k0nhnucBed_result3.bed (deleted)
shed-tool 11126 11143           ubuntu 1373u      REG                8,1      280      34454 /tmp/tmpn5kbmbx6nucBed_result3.bed
shed-tool 11126 11143           ubuntu 1374u      REG                8,1       26      34460 /tmp/tmp7quq5qiisubtractBed_result2.bed (deleted)
shed-tool 11126 11143           ubuntu 1375u      REG                8,1       13      34464 /tmp/tmpy_xh9zrksubtractBed_result3.bed (deleted)
shed-tool 11126 11143           ubuntu 1376u      REG                8,1       26      34462 /tmp/tmph4t2pdorsubtractBed_result2.bed
shed-tool 11126 11143           ubuntu 1377u      REG                8,1      412      34468 /tmp/tmp9c8u2cipbamToBed_result1.bed (deleted)
shed-tool 11126 11143           ubuntu 1378u      REG                8,1       26      34458 /tmp/tmpxj_5dtzusubtractBed_result1.bed
shed-tool 11126 11143           ubuntu 1379u      REG                8,1      412      34470 /tmp/tmpu4s4z_grbamToBed_result1.bed
mvdbeek commented 3 years ago

Alright, I've traced this too, and https://github.com/galaxyproject/galaxy/pull/10852 prevents accumulating those in a small test.