search

mopidy / mopidy-local

Mopidy extension for playing music from your local music archive
https://mopidy.com/ext/local/
Apache License 2.0
61 stars 26 forks source link

Runs out of memory while doing local scan of big library on Raspberry PI Zero W #22

Open Caenhorn opened 6 years ago

Caenhorn commented 6 years ago

Last output:

INFO     Scanned 18700 of 26522 files in 12201s, ~5103s left.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/DIRECTSO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FILEIO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FLASHASS.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTXTRA.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/INETURL.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MACROMIX.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MIXSERVI.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MPEG3IMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MUIDIALO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETFILE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETLINGO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/QT3ASSET.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDCON.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDIMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACNVRT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWADCMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAIMPOR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAOPT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWASTRM.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTXTRA.X32: No audio found in file.
INFO     Scanned 18800 of 26522 files in 12234s, ~5025s left.

(python:2337): GLib-ERROR **: /build/glib2.0-F5w919/glib2.0-2.50.3/./glib/gmem.c:100: failed to allocate 11017050 bytes
Trace/breakpoint trap

By the way, log file didn't catch that for some reason. Last line in log file is last progress report on scanning.

I'm running:

pi@mopidy-zero:/var/log/mopidy $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description:    Raspbian GNU/Linux 9.1 (stretch)
Release:        9.1
Codename:       stretch
pi@mopidy-zero:/var/log/mopidy $ uname -a
Linux mopidy-zero 4.9.59+ mopidy/mopidy#1047 Sun Oct 29 11:47:10 GMT 2017 armv6l GNU/Linux

pi@mopidy-zero:/var/log/mopidy $ dpkg -s mopidy
Package: mopidy
Status: install ok installed
Priority: optional
Section: sound
Installed-Size: 853
Maintainer: Stein Magnus Jodal <jodal@debian.org>
Architecture: all
Version: 2.1.0-1
Caenhorn commented 6 years ago

Effective configuration:

pi@mopidy-zero:/var/log/mopidy $ sudo mopidyctl config
Running "/usr/bin/mopidy --config /usr/share/mopidy/conf.d:/etc/mopidy/mopidy.conf config" as user mopidy
[core]
cache_dir = /var/cache/mopidy
config_dir = /etc/mopidy
data_dir = /var/lib/mopidy
max_tracklist_length = 10000
restore_state = false

[logging]
color = true
console_format = %(levelname)-8s %(message)s
debug_format = %(levelname)-8s %(asctime)s [%(process)d:%(threadName)s] %(name)s\n  %(message)s
debug_file = /var/log/mopidy/mopidy-debug.log
config_file = /etc/mopidy/logging.conf

[audio]
mixer = software
mixer_volume =
output = autoaudiosink
buffer_time =

[proxy]
scheme =
hostname =
port =
username =
password =

[moped]
enabled = true

[local-images]
enabled = true
library = json
base_uri = /images/
image_dir =
album_art_files =
  *.jpg
  *.jpeg
  *.png

[iris]
enabled = true
country = NZ
locale = en_NZ
spotify_authorization_url = https://jamesbarnsley.co.nz/auth_spotify.php
lastfm_authorization_url = https://jamesbarnsley.co.nz/auth_lastfm.php

[mpd]
enabled = true
hostname = ::
port = 6600
password =
max_connections = 20
connection_timeout = 60
zeroconf = Mopidy MPD server on $hostname
command_blacklist =
  listall
  listallinfo
default_playlist_scheme = m3u

[http]
enabled = true
hostname = ::
port = 6680
static_dir =
zeroconf = Mopidy HTTP server on $hostname

[stream]
enabled = true
protocols =
  http
  https
  mms
  rtmp
  rtmps
  rtsp
metadata_blacklist =
timeout = 5000

[m3u]
enabled = true
base_dir =
default_encoding = latin-1
default_extension = .m3u8
playlists_dir = /var/lib/mopidy/playlists

[softwaremixer]
enabled = true

[file]
enabled = true
media_dirs =
  $XDG_MUSIC_DIR|Music
  ~/|Home
excluded_file_extensions =
  .jpg
  .jpeg
show_dotfiles = false
follow_symlinks = false
metadata_timeout = 1000

[local]
enabled = true
library = json
media_dir = /mnt/music
scan_timeout = 15000
scan_flush_threshold = 100
scan_follow_symlinks = false
excluded_file_extensions =
  .directory
  .html
  .jpeg
  .jpg
  .log
  .nfo
  .png
  .txt
  .cue
kingosticks commented 6 years ago

Have you tried with mopidy-local-sqlite? That is the preferred local library to use.

Caenhorn commented 6 years ago

Thanks for the tip! I installed mopidy-local-sqlite, added it as a library of choice in config. Alas, it crashes with sqlite too.

INFO     Scanned 18500 of 26212 files in 11936s, ~4976s left.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/CDPLUS/Sub_Info.ja: gst-stream-error-quark: Could not determine type of stream. (4)
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/faceplate.dxr: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/media.cst: gst-stream-error-quark: Could not determine type of stream. (4)
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/MOVIES/BLANK.MOV: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/MOVIES/ozzy.mov: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/MEDIA/readme.htm: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00J: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00N: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00S: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/PICTURES/JACKET01.00T: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/WIN_QT4/QuickTimeInstallCache.qdat: gst-stream-error-quark: Could not determine type of stream. (4)
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/BUDAPI.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/DIRECTSO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FILEIO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FLASHASS.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/FONTXTRA.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/INETURL.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MACROMIX.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MIXSERVI.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MPEG3IMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/MUIDIALO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETFILE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/NETLINGO.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/QT3ASSET.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDCON.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SOUNDIMP.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWACNVRT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWADCMPR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAIMPOR.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWAOPT.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/SWASTRM.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTASSE.X32: No audio found in file.
WARNING  Failed local:track:Ozzy%20Osbourne/2001%20-%20Down%20To%20Earth/DATA/XTRAS/TEXTXTRA.X32: No audio found in file.
INFO     Scanned 18600 of 26212 files in 11967s, ~4897s left.
WARNING  Failed local:track:Pain%20of%20Salvation/2004%20-%20Be/01%20-%20Prologue-Animae%20Partus%20%28%27I%20Am%27%29.flac: Timeout after 15000ms

(python:1142): GLib-ERROR **: /build/glib2.0-F5w919/glib2.0-2.50.3/./glib/gmem.c:100: failed to allocate 11017050 bytes
Trace/breakpoint trap
kingosticks commented 6 years ago

26k files should be fine. This has been used with larger collections. Can you provide some data on the process size? How much ram have you allocated to system memory? Is it crashing when it hits a limit?

Caenhorn commented 6 years ago

I've collected some data on process size and memory dynamics. It looks like it does run out of free+cached+swap memory. Judging by log-file, process itself does not crash. process memory memory dynamics

Log-file: mopidy.log

Here is netdata snapshot. You can use any netdata demo site to import it and check metrics on your own. netdata-mopidy-zero-20171201-221224-9204.snapshot.gz

Most of the files in my library are lossless, if it matters. I used stretch lite image for mopidy installation. It is a dedicated system for mopidy, so only services coming by default in stretch lite are running. My config.txt is almost vanilla, except for disabled onboard audio and enabled Allo Mini Boss gpio extension board. I could free up some memory by allocating less GPU mem next time.

mabeltron commented 6 years ago

I can confirm I have seen this problem with mopidy and with other MPD servers. The scan function appears to have an in-memory cache and will fill memory to the point where the hardware will become unresponsive - I had this on a server with 8GB of RAM as well as RPis. I suspect there needs to be a method of flushing to disk more frequently.

monte-monte commented 5 years ago

I have the same behavior on a server using mopidy-local-sqlite with option "scan_flush_threshold = 50". It doesn't seem to free memory until it finishes scan. Had to hard reset machine several times before I finished initial scan. This for sure must be fixed. mopidy version 2.2.2

kingosticks commented 4 years ago

@jrowberg Can you confirm that your problem is not related to scanning like the others here? Yours is a memory issue when simply playing music, right? And the only difference between this Pi and the other 3 Pis you have is the exact local files being played? What model Rapsberry Pis are you using?

kingosticks commented 4 years ago

Yes please, a separate issue would be helpful.

01tot10 commented 4 years ago

I think I'm experiencing similar behavior with my instance of Mopidy + Mopidy-local.

I've had to setup Mopidy twice - once with Mopidy on top of Python2, and now with Python3. Both times, when adding Mopidy-local into the picture and doing an initial scan of my music library of ~3000 files, the system becomes unresponsive and supposedly hangs near the end of the scan. Luckily, after doing a hard power cycle and attempting to do the scan again, it continues from where it left off and the scan runs till the end. Still, not how it should be I suppose and something should be done.

I'm running Mopidy on Raspberry 3+ with 1Gb of RAM and Raspbian Stretch.

Uatschitchun commented 4 years ago

Experiencing the same with mopidy 3.0.1 on a Rpi3 raspbian buster. It's using up more and more memory during the initial scan. Needs app. 30min to fill the RAM until swapping starts and the PI gets stuck

Uatschitchun commented 4 years ago

It's not possible to scan in more than 4000 songs without sucking up the RPi3's full RAM and having to kill mopidy. Haven't tested with 'limit' yet...

Uatschitchun commented 4 years ago

When using 'limit' (3000), the scan works, but that means to restart the scan over and over until all files are scanned.

Uatschitchun commented 4 years ago

Using limit 5000 works, but with 150k files it's not the glorious solution

holtermp commented 4 years ago

Hi, I face the same problem here Trying to do an initial scan of 68450 files on a Raspberry Pi 3 Model B Rev 1.2 with Mopidy 3.0.1 always results in swapping after about 20 min. I than power cycle and restart and hope to get through eventually.

Any ideas on how to tackle this would be helpful. @Uatschitchun : What do you mean by "limit", i could find no such parameter in the documentation.

holtermp commented 4 years ago

Ok, just found it myself, it is a parameter when calling mopidyctl local scan: local scan [--limit LIMIT] [--force]

  Scan local media files and populate the local library.

    --limit LIMIT  Maximum number of tracks to scan
    --force        Force rescan of all media files
holtermp commented 4 years ago

It seemed to me that memory filled up very quickly when I had scan timeouts. And I had lots of them. I do not understand, how timeouts can occur on a local file system, but then I do not know how the mopidy scanner works. Anyway, I set the timeout to 10 seconds:

[local]
scan_timeout = 10000

And ran the scan in a loop of 1000 tracks each

#!/bin/bash
while :
do      
        mopidyctl local scan --limit=1000
        echo "         Press [CTRL+C] to stop.."
        sleep  5
done

That finally helped to complete the scan.

mrwrighty commented 4 years ago

Sadly the script didn't do it for me. I only have 11500 files and its bombing every 1000 or so. I have a timeout of 10000 and a limit of 1000 as per script, but no joy. Pi B3+

01tot10 commented 4 years ago

In proceedings to the topic:

I had to refresh my collection today, and tried out the limit-trick that was proposed in this thread. '--limit=1000' didn't quite prevent the crash in my instance of Mopidy, running on Raspberry Pi 3+, 1Gb of RAM, but I do have a good amount of processes running in the background. I continued with '--limit=500' for the rest of the scan and, at least with this batch, didn't crash the system anymore.

Anyways, just wanted to keep the thread alive and well, since the freeze takes down my whole server if I'm not careful and is quite a showstopper.

Trotter73 commented 4 years ago

Hi,

Yet another "Met to".. Running on a PiZero a scan would hang the Pi after an hour or so, the script above seems to be working great, I started low with a limit of 200 but after some monitoring took it up to 500, I could probably go more but just wanted something reliable.

It's taken a while, 16H or so but I've scanned nearly 28K files...

Big thanks to @holtermp !

tljohnsn commented 4 years ago

Thanks everyone who commented. I am having the same issue. I modified @holtermp 's script a bit:

#!/bin/bash                                                                                                                                                                 
media_dir=`sudo mopidyctl config |grep media_dir | cut -d = -f 2 | sed -e 's/\s//g'`
numfiles=`find /home/ftp/local -type f -name "*.mp3" |wc -l`
echo "found $numfiles files in /home/ftp/local"
loops=`echo $(( numfiles / 1000 ))`
for i in $(seq 0 $loops); do
    echo $i
    echo mopidyctl local scan --limit=1000
done