search

jasonmc / forked-daapd

A re-write of the firefly media server (mt-daapd). It's released under GPLv2+. Please note that this git repository is a mirror of the official one at git://git.debian.org/~jblache/forked-daapd.git
http://blog.technologeek.org/2009/06/12/217
GNU General Public License v2.0
328 stars 45 forks source link

Too many open files #107

Open StephenAtty opened 11 years ago

StephenAtty commented 11 years ago

I've got a lot of directories in my library and it looks like forked daapd keeps opening directories to scan then but doesn't close them and I end up with lots of entries like this in my log:

[2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Albums/Oasis - 1994 - Definitely Maybe (2 CD Australia Edition)/CD1: Too many open files [2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Albums/Oasis - 2000 - Standing on the Shoulder of Giants/CD2 - Bonus CD: Too many open files [2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Albums/Oasis - 2000 - Standing on the Shoulder of Giants/CD1: Too many open files [2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Compilations/Oasis - 2010 - Time Flies...1994-2009/CD4 - iTunes Live London Festival: Too many open files [2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Compilations/Oasis - 2010 - Time Flies...1994-2009/CD2: Too many open files [2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Compilations/Oasis - 2010 - Time Flies...1994-2009/CD1: Too many open files [2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Compilations/Oasis - 2006 - Stop The Clocks/CD2: Too many open files [2013-06-01 13:19:26] scan: Could not open directory /webstuff/shares/music/Rock - Pop/Oasis/Compilations/Oasis - 2006 - Stop The Clocks/CD1: Too many open files\

elwertk commented 11 years ago

It does close the directory handler after having scanned it. However the libdispatch version is coming with massive parallelism in scanning directories so there is a remote chance you ran out of file descriptors for a very huge library I guess.

What system you're on? Anything else running that may allready allocate a whole bunch of file descriptors?

On linux use lsof or try #sysctl fs.file-nr in a shell while a scan is running. sysctl command will give three numbers with the first one being number of allocated; second number of allocated but unused and third number the system max limit.

That may help revealing what is going on..

StephenAtty commented 11 years ago

I was doing a complete library rebuild. So what I did was shut it down, Trash the DB, rename the root folder for my library and then move the directories back in in smaller blocks. I've got samba and some other stuff running on the server (ubuntu 12.0.4) but I should still have about 27,000 file descriptors available.

I'm sure its the libdispatch doing it - not that its going to get fixed as the forked-daapd project seems to be pretty much dead....

elwertk commented 10 years ago

See #40 for a potential solution