Parse secondary album types from MusicBrainz

[sepo]

Problem

I've different radio plays. If i import one (e.g. https://musicbrainz.org/release-group/19e6209b-2ddc-30b8-9273-484bd075fe7b) to my library the ALBUMTYPE tag is always "other". The Musicbrainz site says "other + spokenword". I think the ALBUMTYPE should be "spokenword" instead. "other" is pretty useless.

Setup

• OS: Fedora 24
• Python version: 2.7.12
• beets version: 1.3.19
• Turning off plugins made problem go away (yes/no): no

My configuration (output of beet config) is:

directory: /mnt/Dateien/Musik
plugins: inline discogs chroma copyartifacts info badfiles edit scrub missing mbsync
paths:
    albumtype:soundtrack: Soundtracks/$album/$disc_and_track $title
    albumtype:audiobook: Hörbuch/$album/$disc_and_track $title
    albumtype:spokenword: Hörspiel/$albumartist/$album/$disc_and_track $title
    singleton:true: Tracks/$artist - $title
    comp:true: Compilations/$album%aunique{}/$disc_and_track $title
    default: Albums/$albumartist/$album%aunique{}/$disc_and_track $title
import:
    languages: de en
    detail: no
match:
    preferred:
        countries: ['DE', 'US', 'GB|UK']
        media: ['CD', 'Digital Media|File']
copyartifacts:
    extensions: .cue .CUE .LOG  .log .txt .TXT .jpeg .JPG .jpg .png .PNG
    print_ignored: yes
scrub:
    auto: yes
item_fields:
    disc_and_track: u'%02i.%02i' % (disc, track) if
                    disctotal > 1 else u'%02i' % (track)
edit:
    itemfields: album albumartist artist asin catalognum comments composer country day disc disc_and_track disctitle disctotal genre label language lyrics media month original_day original_month original_year title track tracktotal year    albumfields: album albumartist albumstatus albumtotal  
    albumtype asin catalognum country day disctotal genre label language month original_day original_month original_year year

[sampsyo]

Thanks for the report! It looks like release groups are now allowed to have "secondary" types. See the XML here: https://musicbrainz.org/ws/2/release-group/19e6209b-2ddc-30b8-9273-484bd075fe7b

We currently only parse the primary type, but we should consider getting the rest of them too.

[anshuman73]

Hi ! I'd like to work on this issue. Can you help me get started ? (a GCI student here)

We want both the primary tag and the secondary tag to be displayed or check if the primary is a non-useful tag (for eg, 'other') and accordingly use the secondary tag ?

(P.S. I'm unable to claim tasks at the moment due to parental consents issue, should be able to soon)

[sampsyo]

Hi, @anshuman73! Great!

Let's start by using all the album types. We can just just join them all into a single string using a comma or a semicolon for now. That should work well for a first implementation, and we can consider more nuanced policies in a later phase.

[anshuman73]

Sure, also, does beets use a local musicbrainz server or uses the api via http ? I couldn't find the code where the requests to musicbrainz is made. Any specific file you can point me to ?

[sampsyo]

There's no local server; it uses the public API.

The code in question here is beetsplug/mbcollection.py. You'll see the API calls go though the python-musicbrainzngs library, which is linked above.

[anshuman73]

oh, thanks ! I'll get to work right away.

[sampsyo]

@anshuman73 Ack! I answered that question incorrectly—I was thinking of a different thread.

The actually relevant code is in beets/autotag/mb.py. That's where we pull data out of the MB responses and format strings for metadata fields.

Sorry about that! :flushed: Please disregard my earlier comment.

[anshuman73]

@sampsyo No issues.

I believe this is where I'd be focusing at ?

https://github.com/beetbox/beets/blob/master/beets/autotag/mb.py#L280

[sampsyo]

Yep, exactly!

[anshuman73]

Also, is there any place where I can see example json/dictionary data given out ? just so I can figure out where the secondary tags are stored (as the value of what key)

[anshuman73]

Comparing it to the xml example given above, I cannot find a tag named "type", so I'm assuming it defaults to the primary-type tag ?

[sampsyo]

There's no place where it currently gets printed out, so I'd suggest temporarily inserting a print call to output the JSON data.

If you're interested in seeing (and experimenting with) how the XML is parsed precisely, the py-mb-ngs code might be the right place to look: https://github.com/alastair/python-musicbrainzngs/blob/master/musicbrainzngs/mbxml.py#L477

[anshuman73]

Ahh that's just perfect. Discloses the whole schema

[anshuman73]

@sampsyo I think I've written the required code, I just can't figure out how to pass a release result to test the function. Any suggestions ?

[sampsyo]

Sure! The test input is defined here: https://github.com/beetbox/beets/blob/9de27c6b70d460a43839728704332fc71587191c/test/test_mb.py#L31

You can add a test method to that class to demonstrate the new data extraction.

[anshuman73]

hmm, well it doesn't exactly have the primary type and secondary type fields. I just wanted to confirm the key names and whether secondary types are stored as a list or not. (though I checked the same from https://github.com/alastair/python-musicbrainzngs/blob/master/musicbrainzngs/mbxml.py)

[sampsyo]

Aha! Yes, you'll want to try running beets so it executes a real query in that case—and then check the data that gets returned by adding a print call.

[anshuman73]

Hmm, I'll build it from my local source by simply executing the setup.py ?

[sampsyo]

Good question! This wiki page has instructions for getting set up with a local installation: https://github.com/beetbox/beets/wiki/Hacking

In particular, I'd recommend pip install -e .. (It's sometimes convenient to use the --user flag too.)

[anshuman73]

o.O I never knew about such an easy pip method existing for local installation. Thanks !

[anshuman73]

Yay ! Everything's copacetic Submitting a PR in a min..

[mineo]

Since this (and #2294) have seen way more attention than my old #357, I'll just mention it here so it can get closed or something when this gets done.

[sampsyo]

Arg! Thanks for the reminder, @mineo—we should have noticed that a long time ago. As you suggested back on that ticket, the right way to resolve this is probably to store the types in separate tags.

[anshuman73]

@sampsyo So would you suggest me doing that ^ in the #2294 PR or add that when I'm tackling #1547 . I think it would be better to implement it while adding support for more fields (now that we've already logged the new types)

[sampsyo]

Good question. Since it has to do with which fields we set on the imported items, let's worry about it after #1547.

[sampsyo]

I just merged #2294, which should log the multiple album types to beets's debug output. If you're interested in finishing off this feature, please try importing a few albums with that debug output enabled to check what's going on with the difference between the current one-shot album type and the new data—that will help us decide how to handle the new album types.

[ghost]

My audiobooks are still being imported as albumtype "other" on the latest linuxserver release.

[squigglezworth]

Is there any plan on finishing this? Or is there any way of using the secondary types in a path format?

[tweitzel]

Oh hey you still need some samples, I think. Here's the file that led me to this issue number.

pear-2:autotag xarph$ beet -v import /Volumes/books/Calibre\ Library/David\ L.\ Craddock/Dungeon\ Hacks_\ How\ NetHack\,\ Angband\,\ and\ Other\ Roguelikes\ Changed\ the\ Course\ of\ Video\ Games\ \(7\)/Dungeon\ Hacks_\ How\ NetHack\,\ Angband\,\ and\ O\ -\ David\ L.\ Craddock.m4b 
user configuration: /Users/xarph/.config/beets/config.yaml
data directory: /Users/xarph/.config/beets
plugin paths: /Users/xarph/work/whatlastgenre/plugin/beets/beetsplug
Sending event: pluginload
lastgenre: Loading canonicalization tree /usr/local/lib/python3.7/site-packages/beets-1.5.0-py3.7.egg/beetsplug/lastgenre/genres-tree.yaml
inline: adding item field disc_and_track
library database: /Users/xarph/Music/beetslibrary.blb
library directory: /Users/xarph/Music/beets
Sending event: library_opened
Sending event: import_begin
Sending event: import_task_created
Sending event: import_task_start
Looking up: /Volumes/books/Calibre Library/David L. Craddock/Dungeon Hacks_ How NetHack, Angband, and Other Roguelikes Changed the Course of Video Games (7)/Dungeon Hacks_ How NetHack, Angband, and O - David L. Craddock.m4b
Tagging David L. Craddock read by Mike Rylander - Dungeon Hacks: How NetHack, Angband, and Other Roguelikes Changed the Course of Video Games
No album ID found.
Search terms: David L. Craddock read by Mike Rylander - Dungeon Hacks: How NetHack, Angband, and Other Roguelikes Changed the Course of Video Games
Album might be VA: False
Searching for MusicBrainz releases with: {'release': 'dungeon hacks: how nethack, angband, and other roguelikes changed the course of video games', 'artist': 'david l. craddock read by mike rylander', 'tracks': '1'}
Requesting MusicBrainz release 0886f9d5-15e8-4c1e-a2f4-de6784f24c04
primary MB release type: other
secondary MB release type(s): audiobook
Sending event: albuminfo_received
Candidate: David L. Craddock read by Mike Rylander - Dungeon Hacks: How NetHack, Angband, and Other Roguelikes Changed the Course of Video Games (0886f9d5-15e8-4c1e-a2f4-de6784f24c04)
Computing track assignment...
...done.
Success. Distance: 0.73
Requesting MusicBrainz release 8ef21a4c-43b1-48ea-94f3-149473b5f3be
primary MB release type: other
secondary MB release type(s): audiobook
Sending event: albuminfo_received
Candidate: David L. Craddock read by Mike Rylander - Stay Awhile and Listen: How Two Blizzards Unleashed Diablo and Forged a Video-Game Empire (8ef21a4c-43b1-48ea-94f3-149473b5f3be)
Computing track assignment...
...done.
Success. Distance: 0.86
Requesting MusicBrainz release 2628ed19-d337-49ac-9fa1-4b468fe32329
primary MB release type: album
secondary MB release type(s): spokenword, audiobook
Sending event: albuminfo_received
Candidate: Agatha Christie read by David Suchet - How does your garden grow? and other stories (2628ed19-d337-49ac-9fa1-4b468fe32329)
Computing track assignment...
...done.
Success. Distance: 0.91
Requesting MusicBrainz release 2c5bc106-7cce-43b9-ae35-ecb612a3d8e9
primary MB release type: album
secondary MB release type(s): mixtape/street
Sending event: albuminfo_received
Candidate: David Banner - Sex, Drugs & Video Games (2c5bc106-7cce-43b9-ae35-ecb612a3d8e9)
Computing track assignment...
...done.
Success. Distance: 0.86
Requesting MusicBrainz release 68fb8ed6-0665-49dd-adc2-c8515b6e252e
primary MB release type: broadcast
secondary MB release type(s): interview
Sending event: albuminfo_received
Candidate: Tim Pritlove - NetHack (68fb8ed6-0665-49dd-adc2-c8515b6e252e)
Computing track assignment...
...done.
Success. Distance: 0.83
chroma: acoustid album candidates: 0
Evaluating 5 candidates.

/Volumes/books/Calibre Library/David L. Craddock/Dungeon Hacks_ How NetHack, Angband, and Other Roguelikes Changed the Course of Video Games (7)/Dungeon Hacks_ How NetHack, Angband, and O - David L. Craddock.m4b (1 items)
Sending event: before_choose_candidate
Finding tags for album "David L. Craddock read by Mike Rylander - Dungeon Hacks: How NetHack, Angband, and Other Roguelikes Changed the Course of Video Games".
Candidates:
1. David L. Craddock read by Mike Rylander - Dungeon Hacks: How NetHack, Angband, and Other Roguelikes Changed the Course of Video Games (26.6%) (missing tracks, tracks, media) (Digital Media, 2015)
2. Tim Pritlove - NetHack (17.2%) (album, artist, tracks, ...) (Digital Media, 2013, XW, Metaebene Personal Media, CRE199)
3. David Banner - Sex, Drugs & Video Games (14.5%) (missing tracks, album, artist, ...) (Digital Media, 2012, US)
4. David L. Craddock read by Mike Rylander - Stay Awhile and Listen: How Two Blizzards Unleashed Diablo and Forged a Video-Game Empire (13.7%) (missing tracks, album, tracks, ...) (Digital Media, 2016)
5. Agatha Christie read by David Suchet - How does your garden grow? and other stories (9.5%) (missing tracks, artist, album, ...) (3xCD, 2002, GB, HarperCollins Publishers UK, 0-00-714536-5)
# selection (default 1), Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort, Print tracks, plaY, eDit,
edit Candidates? s
Sending event: import_task_choice
Sending event: import
Sending event: cli_exit

[tweitzel]

If you add this functionality I'd love to be able to force an album type on command line, since audiobooks actually being tagged as such in musicbrainz is rather rare, it seems.

pear-2:autotag xarph$ beet -v import /Volumes/books/Calibre\ Library/Henri\ Charriere/Papillon\ \(171\)/Papillon\ -\ Henri\ Charriere.m4b 
user configuration: /Users/xarph/.config/beets/config.yaml
data directory: /Users/xarph/.config/beets
plugin paths: /Users/xarph/work/whatlastgenre/plugin/beets/beetsplug
Sending event: pluginload
lastgenre: Loading canonicalization tree /usr/local/lib/python3.7/site-packages/beets-1.5.0-py3.7.egg/beetsplug/lastgenre/genres-tree.yaml
inline: adding item field disc_and_track
library database: /Users/xarph/Music/beetslibrary.blb
library directory: /Users/xarph/Music/beets
Sending event: library_opened
Sending event: import_begin
Sending event: import_task_created
Sending event: import_task_start
Looking up: /Volumes/books/Calibre Library/Henri Charriere/Papillon (171)/Papillon - Henri Charriere.m4b
Tagging Henri Charriere - Papillon (Unabridged)
No album ID found.
Search terms: Henri Charriere - Papillon (Unabridged)
Album might be VA: False
Searching for MusicBrainz releases with: {'release': 'papillon (unabridged)', 'artist': 'henri charriere', 'tracks': '1'}
Requesting MusicBrainz release a891885c-ddd6-4ae4-9cab-22dd270e0b60
primary MB release type: other
secondary MB release type(s): audiobook
Sending event: albuminfo_received
Candidate: Henri Charrière gelesen von Oliver Rohrbeck - Papillon (a891885c-ddd6-4ae4-9cab-22dd270e0b60)
Computing track assignment...
...done.
Success. Distance: 0.90
Requesting MusicBrainz release e2804d68-83a5-4102-bbb2-ec6b9ecc7167
primary MB release type: album
Sending event: albuminfo_received
Candidate: Henri Dès - Henri Dès, Volume 16: L'Hirondelle et le Papillon (e2804d68-83a5-4102-bbb2-ec6b9ecc7167)
Computing track assignment...
...done.
Success. Distance: 0.86
Requesting MusicBrainz release 49f61758-4db9-4381-9e24-d0059994109b
primary MB release type: single
Sending event: albuminfo_received
Candidate: JACKSON - Papillon (49f61758-4db9-4381-9e24-d0059994109b)
Computing track assignment...
...done.
Success. Distance: 0.44
Requesting MusicBrainz release 3ad990f5-c054-4f82-b9b6-8f57d6fcb76b
primary MB release type: single
Sending event: albuminfo_received
Candidate: JACKSON - Papillon (3ad990f5-c054-4f82-b9b6-8f57d6fcb76b)
Computing track assignment...
...done.
Success. Distance: 0.48
Requesting MusicBrainz release 9e5613cc-d7d6-47c2-a9f0-41a9b77805e8
primary MB release type: album
secondary MB release type(s): live
Sending event: albuminfo_received
Candidate: SixMileBridge - Unabridged (9e5613cc-d7d6-47c2-a9f0-41a9b77805e8)
Computing track assignment...
...done.
Success. Distance: 0.85
chroma: acoustid album candidates: 0
Evaluating 5 candidates.

/Volumes/books/Calibre Library/Henri Charriere/Papillon (171)/Papillon - Henri Charriere.m4b (1 items)
Sending event: before_choose_candidate
Finding tags for album "Henri Charriere - Papillon (Unabridged)".
Candidates:
1. JACKSON - Papillon (56.3%) (artist, tracks, album, ...) (Digital Media, 2017, US, Team Wang Co., Ltd)
2. JACKSON - Papillon (51.6%) (artist, tracks, album, ...) (Digital Media, 2017, CN, Team Wang Co., Ltd)
3. SixMileBridge - Unabridged (15.2%) (missing tracks, artist, tracks, ...)
4. Henri Dès - Henri Dès, Volume 16: L'Hirondelle et le Papillon (13.9%) (missing tracks, album, artist, ...) (CD, 2008, Productions Mary-Josée)
5. Henri Charrière gelesen von Oliver Rohrbeck - Papillon (9.6%) (missing tracks, artist, tracks, ...) (Digital Media, 2011, DE, Lauscherlounge)
# selection (default 1), Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort, eDit, edit Candidates, Print tracks,
plaY?