absubmit: Do not filter on supported formats

[bearcatsandor]

Problem

When running beet absubmit artist:"ace frehley" title:"rip it out" i get the error absubmit: Not analysing Ace Frehley - 1978 - Ace Frehley - Rip It Out - 3:39, file not in supported format.

However, i submited a bug about it to the essentia developers ( https://github.com/MTG/essentia/issues/569 ) and they claim that since essentia is based on ffmpeg that it should work just fine.

Can you tell me what is going on? I've placed a strace here: https://gist.github.com/4eb667a7eb81cdcfb16096b05240c651

Any ideas how i can fix this? The strace complains about my missing the files /home/bearcat/.config/beets/musiclibrary.blb-wal and /home/bearcat/.config/beets/musiclibrary.blb-journal but i've never seen mention of those files anywhere else.

Setup

• OS: Linux jaguarundi 4.9.5-gentoo #1 SMP Sat Jan 21 16:29:49 MST 2017 x86_64 AMD FX(tm)-6300 Six-Core Processor AuthenticAMD GNU/Linux
• Python version: 2.7.13, 3.4.6, 3.5.6 and 3.6.0
• beets version: git
• Turning off plugins made problem go away (yes/no): no

My configuration (output of beet config) is:

lyrics:
    bing_lang_from: []
    fallback: No lyrics found
    bing_client_secret: REDACTED
    google_API_key: REDACTED
    force: no
    google_engine_ID: REDACTED
    sources:
    - google
    - lyricwiki
    - lyrics.com
    - musixmatch
    auto: yes
    genius_api_key: REDACTED
    bing_lang_to:
duplicates:
    format: $albumartist - $album - $title
    count: yes
    album: no
    full: no
    keys: []
    move: ''
    tag: ''
    path: no
    copy: ''
    tiebreak: {}
    strict: no
    checksum: ''
    merge: no
    delete: no
play:
    command: gst-play-1.0 --gapless --volume=1.0
    use_folders: no
    raw: yes
    warning_threshold: 100
    relative_to:
library: ~/.config/beets/musiclibrary.blb

replace:
    '[\\/]': _
    ^\.: _
    '[\x00-\x1f]': _
    '[<>:"\?\*\|]': _
    \s+$: ''
    ' ': _
    \!: ''
original_date: yes
replaygain:
    backend: gstreamer
    auto: yes
    overwrite: yes
    targetlevel: 89

plugins: absubmit acousticbrainz badfiles bpd chroma convert copyartifacts duplicates edit embedart fetchart follow fromfilename ftintitle fuzzy hook importadded info inline lastgenre lyrics mbcollection mbsubmit mbsync mpdstats mpdupdate missing play random replaygain rewrite smartplaylist the thumbnails types web zero
follow:
    auto: yes
    email: bearcatsandor@gmail.com
    password: REDACTED
    userid: REDACTED
acousticbrainz:
    auto: yes
    force: yes

paths:
    default: '%lower{%the{$albumartist}/$original_year-$album%if{$albumdisambig,_[$albumdisambig]}/$disc-$track-$title}'
    singleton: '%lower{singletons/%the{$artist}-$title}'
    comp: '%lower{various_artists/%if{$original_year,$original_year,$year}-$album%if{$albumdisambig,_[$albumdisambig]}/$disc-$track-%the{$artist}-$title}'

format_item: $albumartist - $original_year - $album - $title - $length
art_filename: cover
zero:
    fields: comments
    comments:
    - EAC
    - LAME
    - from.+collection
    - ripped by
    update_database: yes
    auto: yes
    keep_fields: []
smartplaylist:
    relative_to: /home/hometheater/audio
    playlist_dir: ~/.config/mpd/playlists
    playlists: [{name: all.m3u, query: ''}, {name: beatles.m3u, query: 'artist:Beatles'}]
    auto: yes
missing:
    format: $albumartist - $album - $title
    count: yes
    total: yes
format_album: $albumartist - $original_year - $album - {$source}
lastgenre:
    fallback: ''
    canonical: ''
    source: track
    count: 4
    prefer_specific: no
    force: yes
    min_weight: 10
    auto: yes
    whitelist: yes
    separator: ', '

import:
    move: yes
    incremental: yes
    detail: yes
    timid: yes
    resume: ask
    log: /tmp/beets_import.log
    write: yes

sort_album: albumartist+ original_year+ album+

match:
    preferred:
        media:
        - Hybrid SACD
        - SACD
        - CD
        - Digital Media|File
mbcollection:
    auto: no
mpd:
    host: localhost
    port: 6600
    rating: no
    rating_mix: 0.75
    music_directory: /home/hometheater/audio
    password: REDACTED
fetchart:
    auto: yes
    google_key: REDACTED
    google_search: no
    cautious: no
    cover_names: cover front back art album folder
    minwidth: 500
    sources: filesystem fanarttv coverart itunes amazon albumart wikipedia google
    fanarttv_key: REDACTED
    store_source: yes
    google_engine: 001442825323518660753:hrh5ch1gjzm
    enforce_ratio: no
    maxwidth: 0
embedart:
    auto: yes
    maxwidth: 1000
    compare_threshold: 0
    ifempty: no
    remove_art_file: no

musicbrainz:
    user: bearcatsandor
    pass: REDACTED
    searchlimit: 20
fuzzy:
    threshold: 0.7
    prefix: '~'
sort_item: albumartist+ original_year+ album+ disc+ track+
convert:
    auto: no
    format: wavpack
    pretend: no
    quiet: no
    formats:
        flac:
            command: gst-launch-1.0 filesrc location="$source" ! decodebin ! audioconvert ! flacenc quality=8 ! flactag ! filesink location="$dest"
            extension: flac
        wavpack:
            command: gst-launch-1.0 filesrc location="$source" ! decodebin ! audioconvert ! wavpackenc extra-processing=1 md5=true ! apev2mux ! filesink location="$dest"
            extension: wv
        alac:
            command: ffmpeg -i $source -y -vn -acodec alac $dest
            extension: m4a
        aac:
            command: ffmpeg -i $source -y -vn -acodec libfaac -aq 100 $dest
            extension: m4a
        opus: ffmpeg -i $source -y -vn -acodec libopus -ab 96k $dest
        mp3: ffmpeg -i $source -y -vn -aq 2 $dest
        ogg: ffmpeg -i $source -y -vn -acodec libvorbis -aq 3 $dest
        wma: ffmpeg -i $source -y -vn -acodec wmav2 -vn $dest
    embed: yes
    dest: ~/tmp
    never_convert_lossy_files: no
    album_art_maxwidth: 0
    threads: 6
    copy_album_art: no
    tmpdir:
    max_bitrate: 500

    paths: {}
threaded: yes
bpd:
    port: 6600
    volume: 100
    host: ''
    password: REDACTED
per_disc_numbering: yes
ftintitle:
    format: (feat. {0})
    auto: yes
    drop: no
absubmit:
    auto: yes
    extractor: ''
album_fields:
    album_length: "total = 0\nfor item in items:\n    total += item.length\n    return total\n"

ui:
    color: yes
directory: /home/hometheater/audio
copyartifacts:
    print_ignored: no
    extensions: .*
pathfields: {}
item_fields: {}
hook:
    hooks: []
chroma:
    auto: yes
edit:
    itemfields: track title artist album
    albumfields: album albumartist
    ignore_fields: id path
thumbnails:
    auto: yes
    dolphin: no
    force: no
the:
    a: yes
    patterns: []
    the: yes
    strip: no
    format: '{0}, {1}'
mbsubmit:
    threshold: medium
    format: $track. $title - $artist ($length)
rewrite: {}
web:
    host: 127.0.0.1
    include_paths: no
    port: 8337
    cors: ''
importadded:
    preserve_write_mtimes: no
    preserve_mtimes: no

[sampsyo]

Thanks! If that's the case, then we shouldn't filter on our internal list of "supported" formats, here: https://github.com/beetbox/beets/blob/4d90c9645c4901835c8585ad0f8fc38345ae44ec/beetsplug/absubmit.py#L93-L95

@inytar, is there any particular reason you added that type filter rather than just letting the command fail on unsupported formats?

[inytar]

At first I was failing if the extractor gave an error, this was to make sure these files are just skipped instead of stopping the extractor from doing its work. As we now just log an error this can be removed. This will mean that you might see a lot of error messages in your log (I don't know if the extractor returns meaningful errors wich we could parse, then we could just log to info if the file type is not supported).

At the same time acousticbrainz is very specific on which file types they support: http://acousticbrainz.org/contribute, thus even if the upstream essentia supports these file types the acousticbrainz extractor might not. As mentioned in the Acousticbrainz faq, they prefer it if you use their static compiled extractor: http://acousticbrainz.org/faq

[sampsyo]

Huh; that's interesting! I wonder if they meant that list to be the definitive list of all formats they want to support, or just a hint saying that they know these ones work in FFmpeg. Assuming that this was just meant to be a hint, we can safely remove the check. Thanks for providing context, @inytar!

Also, back to the original problem—wv is in that list (and whitelisted in the plugin), so I'm not sure what the original problem was for @bearcatsandor.

[inytar]

Are we sure that the format field for wavpack is really wv? Otherwise that might be the bug in the code.

[sampsyo]

Oh, no—that would be the file extension. I think the format value (from MediaFile) would just be "WavPack."

[inytar]

Ah, that's my error, is there a list of formats?

[inytar]

Back to the part about parsing the error message from the extractor tool. If I try it on a opus file I get the following message:

$ ./streaming_extractor_music ~/Music/Allen,\ Lily/It\'s\ Not\ Me,\ It\'s\ You/01\ -\ \(Lily\ Allen\)\ -\ Everyone\'s\ at\ It.opus bla.json 
Process step: Read metadata
Process step: Compute md5 audio hash and codec
AudioLoader: Could not open file "/home/inytar/Music/Allen, Lily/It's Not Me, It's You/01 - (Lily Allen) - Everyone's at It.opus", error = End of file

With an exit code of 1. This does not seam very parsable. Would it be a problem if beets logs an error for every file that the extractor cannot parse?

[sampsyo]

Yep, the list is on the right-hand site of this map in MediaFile: https://github.com/beetbox/beets/blob/5ed2798a76daa2c547f04538dfdd2e699e487bba/beets/mediafile.py#L63-L77

Yeah, that error is not all that parse-able; you're right. That doesn't sound too verbose… in most cases, it probably doesn't matter to the user why the extractor failed; just that it failed.

[bearcatsandor]

If there is anything I can do to help, let me know. I'm glad to provide a short file or logs.

[sampsyo]

Hey, @bearcatsandor—we just merged a fix. Can you give the latest source a try to see if it works?

[bearcatsandor]

Well, uh. I think so but there's no way to tell. Doing beet absubmit artist:"caravan palace" (all wv files) doesn't return anything, but zsh reports: beet absubmit artist:"caravan palace" 269.75s user 6.28s system 91% cpu 5:01.75 total So, i didn't get any error messages but i think it ought to perhaps return something just for the user.

Looking at the tags in the file, no changes have been made. Looking at https://acousticbrainz.org/datasets/list i don't see anything from today, so how would we know if anything had actually happened?

[sampsyo]

Is there anything in the verbose log?

[bearcatsandor]

Yes. Doing a -vv i do see:

absubmit: Successfully submitted AcousticBrainz analysis for Caravan Palace - 2008 - Caravan Palace - Je m’amuse - 3:34. absubmit: Successfully submitted AcousticBrainz analysis for Caravan Palace - 2008 - Caravan Palace - Violente valse - 3:35. etc

So ...uh.. Now what? I don't see anywhere on AcousticBrains where this data is listed. Doing a beet acousticbrainz "caravan palace" tells me that it's acousticbrainz: getting data for: Caravan Palace - 2008 - Caravan Palace - La Caravane - 5:05 but there's nothing to reflect that in my tags.

[sampsyo]

As the docs for the plugins describe, the data is stored in flexible attributes. Is the data visible in beet info (via the info plugin), for example?

[bearcatsandor]

No. Here's an info dump of one of the files that i submitted and then did a `beet acousticbrainz' on:

https://gist.github.com/3b1922f4ee9a97c631f908a4cbc1e62a

[sampsyo]

Sorry; I should have been more specific: you'll want to use the library viewing functionality of info rather than the file viewer (which means using a query as an argument, not a filename). Does that work?

[bearcatsandor]

what i did was a `beet info caravan palace dragons'. That should have done it, yes?

[sampsyo]

Yes indeed. Weird! I don't currently see any clear reason why the plugin wouldn't be adding the relevant fields.

[bearcatsandor]

Anything else i can do? Provide a file?

[inytar]

Could you try using the acousticbrainz plugin to download the metadata? It might be that it takes acousticbrainz some time from upload to presenting it. The data is now online as can be checked on: https://acousticbrainz.org/api/v1/a7880efc-9085-4b77-b27d-8e3263f5e3fa/high-level. I am working on a branch which moves the absubmit plugin into the acousticbrainz plugin, this should solve the problem you are having.

[bearcatsandor]

Just did a beet info caravan palace dragons and what i get is: https://gist.github.com/eb7214b48fd9150da2b0921d026b0409

Maybe i'm just not understanding how this works. What i'm expecting is to see tags for gender, rhythm, danceability, mood, timbre, voice/instrumental (and hopefully initial key and tempo someday) in my files.

Having them merged will be great! Thanks for your work inytar.

[sampsyo]

Oh dear, I'm afraid I might have misled you—you'll need to use the -l flag to info to see fields in the library. Here's an example from a song I have run the AB plugin on:

$ beet info -l happy but you
[...]/01 I'm Happy but You Don't Like Me.mp3
            abitrate: 273265.9
acoustid_fingerprint: 
         acoustid_id: 
               added: 2013-09-29 21:32:37
               album: Asobi Seksu
            album_id: 1
         albumartist: Asobi Seksu
  albumartist_credit: Asobi Seksu
    albumartist_sort: Asobi Seksu
       albumdisambig: 
         albumstatus: Official
          albumtotal: 11
           albumtype: album
            arranger: 
              artist: Asobi Seksu
       artist_credit: Asobi Seksu
         artist_sort: Asobi Seksu
             artpath: /Users/asampson/code/beets/_etc/testlib/Asobi Seksu $/Asobi Seksu (273265.9)/Asobi Seksu.jpg
                asin: B000BB03J6
    average_loudness: 0.934070587158
                 bar: baz
            bitdepth: 0
             bitrate: 269kbps
                 bpm: 148.661605835
          catalognum: FFR-001
            channels: 2
 chords_changes_rate: 0.0360671952367
          chords_key: C
  chords_number_rate: 0.00197628466412
        chords_scale: major
            comments: 
                comp: False
            composer: 
             country: US
           danceable: 0.133124902844
         data_source: MusicBrainz
                 day: 18
                disc: 01
      disc_and_track: 
           disctitle: 
           disctotal: 01
             encoder: 
            filesize: 6347210
              format: MP3
              gender: male
               genre: Shoegaze
    genre_rosamerica: roc
            grouping: 
                  id: 8
         initial_key: C
        key_strength: 0.873910665512
               label: Friendly Fire Recordings
            language: mul
              length: 3:07
            lyricist: 
              lyrics: 
    mb_albumartistid: cc433a77-176b-4d02-857b-2bfb68b3219c
          mb_albumid: ca0fe9b2-9a52-4a83-9e78-364d38c87c71
         mb_artistid: cc433a77-176b-4d02-857b-2bfb68b3219c
   mb_releasegroupid: cd5fe4f5-1645-3c19-9c3a-071b401e6dfc
          mb_trackid: 59a09c1e-e06b-457e-96d5-668e2b75512f
               media: CD
             missing: 1
               month: 05
       mood_acoustic: 0.0202572327107
     mood_aggressive: 0.335941433907
     mood_electronic: 0.517917871475
          mood_happy: 0.880889296532
          mood_party: 0.682904958725
        mood_relaxed: 0.125917673111
            mood_sad: 0.128956481814
               mtime: 1969-12-31 19:00:00
        original_day: 18
      original_month: 05
       original_year: 2004
       rg_album_gain: 1.2
       rg_album_peak: 1.3
       rg_track_gain: 1.4
       rg_track_peak: 1.4
              rhythm: Tango
          samplerate: 44kHz
              script: Latn
           singleton: False
               title: I'm Happy but You Don't Like Me
               tonal: 0.581466734409
               track: 01
           track_alt: 1
          tracktotal: 11
  voice_instrumental: voice
                year: 2004

You can see a bunch of AB goodness in there.

[bearcatsandor]

OH MY GOD!! I haven't been so happy about something for a long time! Thank you both so much for your hard work on this. I mean, i can now queue up all my Metallica instrumentals and it's accurate? That's amazing!

[sampsyo]

Yay! Glad it's working.