tmpfs and /tmp complications

[ndouglas]

Related to #1067 and (somewhat) #1382.

I'm importing a pretty large library (~2 TB) on a Synology NAS, which is taking a while. I've seen this particular issue crop up twice in the course of doing so.

Performance slows to a near standstill. It'll take twenty minutes to process a single album, and the time will gradually increase. System load reaches very high levels (10-11 on a 2-core machine). Beets is stuck in I/O. Top says kswapd is munching up a whole lot of CPU time.

The issue is resolved essentially instantaneously by removing all cover images from /tmp. Because #YOLO, I just rm *.jpg and boom, Beets starts churning through albums once more, at one a minute or so.

Note that Beets encounters exceptions somewhat often -- 503 status codes from musicbrainz, EOFs to SSL connections, etc. When this happens, I figure the temp files aren't cleaned up, so they just accumulate.

The problem seems to be that this is an issue with tmpfs and /tmp. I suspect that, although I have ~500MB free of RAM, the tmpfs RAM disk is causing swapping. This then results in a performance nightmare. I'm fuzzy on the details, but this is my best guess. Obviously this isn't an issue with Beets, but a configuration issue or a clash with the use of tmpfs.

My tentative suggestion is to allow specifying the temporary directory for cover images and so forth.

This may or may not be worth the time and effort, though -- either no one else has encountered this, or they didn't think it was worth reporting. I myself hesitated. But I did want to mention it and see what you thought.

[sampsyo]

Whoa, that's certainly a strange issue.

I assume you have fetchart enabled, right? Could you please paste your configuration?

It may be as simple as having forgotten to clean up downloaded images in fetchart, in which case we should fix that directly—no need to keep those around at all after the album finishes importing.

[ndouglas]

Yeah, sorry -- I was filling out the template and managed to accidentally delete it, so I didn't provide all of the relevant information.

library: /volume1/music/Beets/library.blb
directory: /volume1/music/
plugins: acousticbrainz badfiles discogs duplicates edit embedart fetchart ftintitle fuzzy info inline lastgenre lyrics mbcollection mbsync missing random smartplaylist the thumbnails types
paths:
        default: Main/%upper{%left{$albumartist_sort,1}}/$albumartist_sort/$original_year - $album%aunique{}/%if{$disc,$disc,01} $track $title
        "comp:true ^albumtype:soundtrack": Compilations/$original_year - $album%aunique{}/%if{$disc,$disc,01} $track $title
        "albumtype:soundtrack": Soundtracks/$original_year - $album%aunique{}/%if{$disc,$disc,01} $track $title
asciify_paths: yes
duplicates:
        album: no
embedart:
        auto: yes
        ifempty: no
fetchart:
        auto: yes
        enforce_ratio: yes
        sources: coverart itunes amazon albumart wikipedia
ftintitle:
        auto: yes
        drop: yes
import:
        move: yes
        write: yes
        log: /volume1/music/Beets/log.txt
        languages: en
        timid: no
lastgenre:
        auto: yes
        count: 5
        force: yes
lyrics:
        auto: yes
        force: yes
        sources: google lyricwiki lyrics.com musixmatch genius
        google_API_key: REDACTED
match:
        strong_rec_thresh: 0.0001
mbcollection:
        auto: yes
musicbrainz:
        user: REDACTED
        pass: REDACTED
original_date: yes
per_disc_numbering: yes
smartplaylist:
        relative_to: /volume1/music
        playlist_dir: /volume1/music/playlists
        playlists:
                - name: all.m3u
                  query: ''
                - name: 'ReleasedIn$year.m3u'
                  query: 'year::[0-9]{4}'
thumbnails:
        auto: yes
        force: yes
types:
        rating: int

[sampsyo]

Great. If you do have time for an experiment, can you try running with fetchart turned off and see whether the problem magically disappears?

[ndouglas]

Sure, but the problem itself takes a long time to manifest -- it's only happened when /tmp is almost full of cover images, and /tmp is 2GB so that takes a while. So it'll take at least that long to confirm with any confidence that it appears to work around the issue. (Even though I expect it to.)

[sampsyo]

OK! Thanks for giving it a try, even though reproducing this kind of thing is onerous.

[ndouglas]

I've been importing for a couple of days with fetchart disabled and haven't seen it. However, I've also been importing (different machine) with fetchart enabled and haven't seen it either.

I've been running a shell script that does this:

while true
do
    beet import -q ./
done

just so any fatal errors are retried and the machine doesn't sit idle while I'm asleep.

After a couple of days of that, my /tmp usage is still at 1%, and performance is just fine.

So I'm a bit puzzled.

I started with 2TB of music and I've imported and reimported it a couple of times -- I repented of some of the matches I chose in haste, and instead of trying to fix those specific releases I just nuked the library and started from scratch. So there's a whole lot of importing and reimporting of a really large library that's been going on over the past few weeks.

throws his hands up

I'm gonna go ahead and close this... I might comment again if I find something concrete and specific, but I'm not really expecting to. Maybe it might make a useful FAQ entry, but it's probably too narrow even for that.

Thanks <3

[sampsyo]

Wow, that certainly is confusing. Thanks for investigating further! Do let us know if you find out anything more.

[ndouglas]

Yeah, will do.

Just as a side note, I'm totally in love with Beets. This is the system I've been dreaming of.

I'm a developer, and have a few plugin ideas, so I expect you'll see me around.

[sampsyo]

Woohoo; thanks for the kind words!