search

Sonerezh / sonerezh

A self-hosted, web-based application to stream your music, everywhere.
https://www.sonerezh.bzh
GNU Affero General Public License v3.0
768 stars 122 forks source link

[bug] Library update process fails (exceeds allowed memory size) #32

Closed nodiscc closed 9 years ago

nodiscc commented 9 years ago

I can't index more than a few thousand songs on my server. From my server error log:

PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 1024 bytes) in /path/to/sonerezh/app/Vendor/Getid3/module.audio-video.asf.php on line 1584, referer: https://my.server.com/sonerezh/import

Debian, apache 2.2.22-13+deb7u4, libapache2-mod-php5 5.4.36-0+deb7u3

Do you think of a possible solution?

lGuillaume124 commented 9 years ago

What is your biggest file in your library?

nodiscc commented 9 years ago

@lGuillaume124 15 largest files with sizes in bytes

$ find . -printf '%s %p\n' | sort -nr | head -n 15
354749053 ./Misc/music/Cowboy Bebop FULL OST (Original Soundtrack) (101 Total Songs)-N82fP5hKZPY.m4a
341358342 ./Misc/music/6-Hour _ EvE Online Ambient Music-lEEmORyxpho.m4a
264531563 ./Mazzy Star - She Hangs Brightly/01 Halah.flac
192610766 ./Misc/SingleTracks0A/methamorphose(ben).mp3
177341614 ./Misc/SingleTracks0A/Audite - Trust in Bass Posdcast.mp3
162833191 ./Misc/music/Swing By My House Vol.6-brian-benjamin-ramirez-swing-by-my-house-vol6.mp3
153252395 ./Misc/SingleTracks0A/DJ Shaktipat       the harvest       DJ SHAKTIPAT  www.sonicmotionrecords.com              .mp3
142683129 ./Misc/music/Solid Steel Radio Show 6_6_2014 Part 3 + 4 - Alex Banks-152951263.mp3
142137096 ./Misc/Divers 2/elektrotek/Live Mix.mp3
133550080 ./Roni Size - Through the Eyes/CD2 01 - Through The Eyes, The Mix.mp3
124919395 ./Camille - Le Fil/15 Quand Je Marche.flac
115831555 ./Cat Stevens - Tea For The Tillerman/09 On The Road To Find Out.flac
113050931 ./Misc/SingleTracks0A/NUSKOOLFUNK.mp3
112668800 ./Misc/SingleTracks0A/mMix techno..mp3
108482249 ./Misc/SingleTracks0A/Mario Aureo - NDPodcast009.mp3
Knah-Tsaeb commented 9 years ago

Hi nodiscc, you are here too ;) I have same issue #23 can you test if you have directory with additional space like "My Music/dir1 /song.mp3".

nodiscc commented 9 years ago

I have no dirs ending with a whitespace or non-ascii character. How can I know on what file the updater chokes? Is there a debug flag?

Knah-Tsaeb commented 9 years ago

No idea, for found this issue I try directory by directory.

nodiscc commented 9 years ago

Error message changed. It's now invariably:

PHP Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 8388608 bytes) in /var/www/music/app/Vendor/Getid3/module.audio-video.asf.php on line 1335, referer: https://my.server.com/music/import

2176 songs imported

Edit: trying to increase values from the main .htaccess:

php_value max_execution_time 3600
php_value memory_limit 320M
lGuillaume124 commented 9 years ago

We suspect memory overflow due to the file size.

nodiscc commented 9 years ago

Increasing max values didn't help:

Allowed memory size of 335544320 bytes exhausted
karibuTW commented 9 years ago

Same issue here

Allowed memory size of 134217728 bytes exhausted

Any idea how to solve it?

Edit: I've even assiged 2G for the php.ini values on sonerezh but no change Allowed memory size of 2097152000 bytes exhausted

lGuillaume124 commented 9 years ago

@karibuTW what are the biggest files in your library please ?

nodiscc commented 9 years ago

Ok it seems your indexer or metadata parser dies on .wma files. I've run the database update process severl times, while keeping a lsof command running in a SSH shell, grepping for the path to my media directory. It keeps running and showing files being indexed by sonerezh, until I hit this:

$ while true; do lsof | grep /media; sleep 1; done
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma
apache2   27634            www-data   18r      REG              254,0   1879999   11142504 /media/TERABOITE/Musique/Sanseverino - Le Tango des Gens/Les films de guerre 04.wma

Then PHP throws an error in the log and stops indexing.
I've removed this infamous directory, but the next directory with WMA files I hit:

 while true; do lsof | grep /media; sleep 1; done
apache2   31698            www-data   17r      DIR              254,0      4096   10887225 /media/TERABOITE/Musique/Machinarium OST/bonus_ep
apache2   31698            www-data   18u      REG              254,0  28393051   11142179 /media/TERABOITE/Musique/Quantic Presenta Flowering Inferno - Death Of The Revolution/09 - Alegria En Bella Vista (Version).flac
apache2   31698            www-data   18u      REG              254,0   7258210   11272493 /media/TERABOITE/Musique/Zenzile - Modus Vivendi/09 Hippo.mp3
apache2   31698            www-data   18u      REG              254,0   4082292   10879480 /media/TERABOITE/Musique/Boards of Canada - Old Tunes Vol 2/12 Sir Prancelot Brainfire.mp3
apache2   27340            www-data   18u      REG              254,0   6805987   11141196 /media/TERABOITE/Musique/Kila - Tog E Go Bog E/02 on Taobh Tuathail Amach.mp3
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma
apache2   27340            www-data   18u      REG              254,0   7503850   10880012 /media/TERABOITE/Musique/Compilation Electronica 2008/06 Steinbolt.wma

It might be GetID3.

nodiscc commented 9 years ago

I "fixed" this bug by removing wma from line 135 in app/Controller/SongsController.php.

I suggest you remove WMA support until you sort this out (horrible format anyway).

Are you using the latest libs? Do you need me to send a sample wma file? Is there an IRC channel where we can get in touch?

pikzen commented 9 years ago

lolwut time: Using the same .wmas as nodiscc on my install (pulled yesterday) does not make the update process crash. The songs get added to my library just fine (even though they cannot be read by the browser).

lGuillaume124 commented 9 years ago

Hi,

Could you send us some wma files please? Unfortunately we don't have any IRC channel yet.

nodiscc commented 9 years ago

@lGuillaume124 Can you send me a mail so that I can link the files to you privately? I'm also available as nodiscc on irc.freenode.net, or via XMPP.

lGuillaume124 commented 9 years ago

hey@sonerezh.bzh :)

nodiscc commented 9 years ago

Thanks @lGuillaume124 mail sent with a link to the files.

lGuillaume124 commented 9 years ago

Hi,

It appears that the GetID3 library does not support wma files, so Sonerezh neither.

nodiscc commented 9 years ago

@lGuillaume124 Good news! Let wma die. Though GetID3 shouldn't probably crash the whole app like that. What's your next move? This workaround https://github.com/Sonerezh/sonerezh/issues/32#issuecomment-77703934 works but there may be related code somewhere else.

nodiscc commented 9 years ago

I see you've removed wma support, that should fix most issues. I'm fine with it. However I see that GetID3 is supposed to support wma. There might be a problem related to software used on the server (Debian stable 64-bit, apache 2.2.22-13+deb7u4, libapache2-mod-php5 5.4.36-0+deb7u3). I know that @pikzen's config (Ubuntu, apache) did not crash while indexing the same files....

@lGuillaume124 did my wma cause a crash on your server? Running Debian too?

lGuillaume124 commented 9 years ago

Even if GetID3 is supposed to support wma, the author says it could be improved. We have friends who encountered problems with wma files too.

Anyway we took the decision to remove that weird format, and Sonerezh will only look for MP3, OGG, FLAC and AAC files now.

The best would be remove the GetID3 dependency by developing our own metadata parser, may be for the 2.0.0 release ^^

And I was unable to parse your files :/ thank you for your investment by the way ;)

nodiscc commented 9 years ago

You're welcome!

@Knah-Tsaeb You may want to retest with the latest 1.0.0-beta and close https://github.com/Sonerezh/sonerezh/issues/23 if it's fixed for you too. Check for weird formats in your music directory.

I'll close this now. Thanks for your help