Crashes on attempt to write tags

[gwern]

My installation of beets crashes while dealing with one particular track: https://dl.dropboxusercontent.com/u/243666993/kenichichiba-%E3%82%AF%E3%83%A9%E3%83%A0%E3%83%8E%E3%83%97%E6%96%B0%E4%B8%96%E7%B4%80%E3%83%95%E3%83%A5%E3%82%B8%E3%83%A7%E3%83%B3%E7%A0%B4-delta.ogg

A import session:

$ file music/random/kenichichiba-クラムノプ新世紀フュジョン破-delta.ogg
music/random/kenichichiba-クラムノプ新世紀フュジョン破-delta.ogg: Ogg data, Vorbis audio, stereo, 44100 Hz, ~256000 bps, created by: Xiph.Org libVorbis I
$ /usr/local/bin/beet -v import music/random/kenichichiba-クラムノプ新世紀フュジョン破-delta.ogg
user configuration: /home/gwern/.config/beets/config.yaml
Sending event: pluginload
Sending event: library_opened
data directory: /home/gwern/.config/beets
library database: /home/gwern/.config/beets/musiclibrary.blb
library directory: /home/gwern/music
Sending event: import_task_start
fingerprint matching '/home/gwern/music/random/kenichichiba-\xe3\x82\xaf\xe3\x83\xa9\xe3\x83\xa0\xe3\x83\x8e\xe3\x83\x97\xe6\x96\xb0\xe4\xb8\x96\xe7\xb4\x80\xe3\x83\x95\xe3\x83\xa5\xe3\x82\xb8\xe3\x83\xa7\xe3\x83\xb3\xe7\xa0\xb4-delta.ogg' failed: invalid fingerprint
Looking up: None
Tagging Kenichi Chiba - クラムノプ新世紀フュージョン破
No album IDs found.
Search terms: Kenichi Chiba - クラムノプ新世紀フュージョン破
Album might be VA: False
acoustid album candidates: 0
Evaluating 5 candidates.
Candidate: Kenichi Chiba - LIFE IS LIKE A LIVE!
Success. Distance: 0.419238
Candidate: Kenichi Chiba - LIFE IS LIKE A LIVE!
Success. Distance: 0.710808
Candidate: Chiba - Time Flies
Success. Distance: 0.812824
Candidate: リンダIII世 - 未来世紀eZ zoo
Success. Distance: 0.798599
Candidate: ASIAN KUNG-FU GENERATION - 新世紀のラブソング
Success. Distance: 0.736055

None (1 items)
Correcting tags from:
    Kenichi Chiba - クラムノプ新世紀フュージョン破
To:
    Kenichi Chiba - LIFE IS LIKE A LIVE!
URL:
    http://musicbrainz.org/release/698aed2f-05c9-4f4c-bb43-57a91ab0b202
(Similarity: 58.1%) (album, unmatched tracks) (2014, JP, Unitone)
Unmatched tracks:
 ! Delta (#4) (3:50)
Apply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? a
Sending event: import_task_choice
Sending event: import_task_apply
0 of 0 items replaced
Traceback (most recent call last):
  File "/usr/local/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.7', 'console_scripts', 'beet')()
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py", line 966, in main
    _raw_main(args)
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py", line 957, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/commands.py", line 884, in import_func
    import_files(lib, paths, query)
  File "/usr/local/lib/python2.7/dist-packages/beets/ui/commands.py", line 856, in import_files
    session.run()
  File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line 291, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py", line 299, in run
    out = self.coro.send(msg)
  File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py", line 181, in coro
    func(*(args + (task,)))
  File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line 1100, in apply_choices
    task.add(session.lib)
  File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line 616, in add
    self.album = lib.add_album(self.imported_items())
  File "/usr/local/lib/python2.7/dist-packages/beets/library.py", line 997, in add_album
    raise ValueError(u'need at least one item')
ValueError: need at least one item

Figuring out which track took a bit of debugging, because I have 421 tracks in that folder and the asynchronous of threading meant that the crash might happen while dealing with another file. In this case, the crash happens everytime so I could try files one by one; while importing my library, beets also segfaulted and crashed after a free(), but I can't easily reproduce them. Is there any way the trackbacks/crashes could be more informative and pinpoint which file killed beets?

[sampsyo]

That's odd; it looks like beets found a match but rejected the only track in it. :flushed: Investigating…

Is there any way the trackbacks/crashes could be more informative and pinpoint which file killed beets?

Alas, not really. This comes up often and there's no clear solution within beets' multithreaded context. The best we can do is keep putting up firewalls for fault isolation. If someone has an ingenious solution for making this easier to debug, I'd be interested to hear it.

[gwern]

That's odd; it looks like beets found a match but rejected the only track in it.

So you're able to replicate the crash on your system with the file I provided?

This comes up often and there's no clear solution within beets' multithreaded context.

Hm... perhaps each thread could append to a logfile each file it deals with and then a clause could catch all exceptions, print out the last n lines of the logfile, and exit? You wouldn't know exactly which file killed beets but you'd have a much smaller list to work with and wouldn't have to go back importing large directories.

[sampsyo]

So you're able to replicate the crash on your system with the file I provided?

Yes. And ugh, after debugging the non-obvious things, the obvious thing was the answer: that release has no tracks! That almost certainly seems like an error in the MB DB; I actually didn't know it was technically possible. Adding a check for this now.

And yes, some sort of background logging would be helpful—but per-thread logs seem like overkill. A separate issue worth discussing.

[gwern]

That almost certainly seems like an error in the MB DB; I actually didn't know it was technically possible.

One thing I've learned working with large user-generated datasets: everything is possible. Littlewood's law & curse of dimensionality in action. You just have to deal with the outliers as you find them...