Discogs fetch fails in oauth2?

[Julf]

Problem

Running this command in verbose (-vv) mode:

$ beet -vv et -vv import /tmp/music

Led to this problem:

Traceback (most recent call last):
 File "/usr/local/bin/beet", line 9, in <module>
   load_entry_point('beets==1.3.17', 'console_scripts', 'beet')()
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py",
line 1236, in main
   _raw_main(args)
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py",
line 1226, in _raw_main
   subcommand.func(lib, suboptions, subargs)
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/commands.py",
line 966, in import_func
   import_files(lib, paths, query)
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/commands.py",
line 943, in import_files
   session.run()
 File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line
323, in run
   pl.run_sequential()
 File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py",
line 379, in run_sequential
   list(self.pull())
 File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py",
line 456, in pull
   out = coro.send(msg)
 File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py",
line 184, in coro
   func(*(args + (task,)))
 File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line
1265, in lookup_candidates
   task.lookup_candidates()
 File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line
592, in lookup_candidates
   autotag.tag_album(self.items, search_ids=self.search_ids)
 File "/usr/local/lib/python2.7/dist-packages/beets/autotag/match.py",
line 439, in tag_album
   search_album, va_likely)
 File "/usr/local/lib/python2.7/dist-packages/beets/autotag/hooks.py",
line 580, in album_candidates
   out.extend(plugins.candidates(items, artist, album, va_likely))
 File "/usr/local/lib/python2.7/dist-packages/beets/plugins.py", line
357, in candidates
   out.extend(plugin.candidates(items, artist, album, va_likely))
                     File
"/usr/local/lib/python2.7/dist-packages/beetsplug/discogs.py", line 147,
in candidates
   return self.get_albums(query)
 File "/usr/local/lib/python2.7/dist-packages/beetsplug/discogs.py",
line 207, in get_albums
   type='release').page(1)
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/models.py", line
336, in page
   data = self.client._get(self._url_for_page(index))
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/client.py", line
110, in _get
   return self._request('GET', url)
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/client.py", line
97, in _request
   content, status_code = self._fetcher.fetch(self, method, url,
data=data, headers=headers)
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/fetchers.py",
line 73, in fetch
   resp, content = self.oauth_client.request(url, method, headers=headers)
 File "/usr/local/lib/python2.7/dist-packages/oauth2/__init__.py", line
682, in request
   connection_type=connection_type)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 1593, in request
   (response, content) = self._request(conn, authority, uri,
request_uri, method, body, headers, redirections, cachekey)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 1388, in _request
   redirections=redirections - 1)
 File "/usr/local/lib/python2.7/dist-packages/oauth2/__init__.py", line
662, in request
   req.sign_request(self.method, self.consumer, self.token)
 File "/usr/local/lib/python2.7/dist-packages/oauth2/__init__.py", line
493, in sign_request
   self['oauth_body_hash'] = base64.b64encode(sha(self.body).digest())
TypeError: must be string or buffer, not None
Traceback (most recent call last):
 File "/usr/local/bin/beet", line 9, in <module>
   load_entry_point('beets==1.3.17', 'console_scripts', 'beet')()
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py",
line 1236, in main
   _raw_main(args)
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/__init__.py",
line 1226, in _raw_main
   subcommand.func(lib, suboptions, subargs)
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/commands.py",
line 966, in import_func
   import_files(lib, paths, query)
 File "/usr/local/lib/python2.7/dist-packages/beets/ui/commands.py",
line 943, in import_files
   session.run()
 File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line
323, in run
   pl.run_sequential()
 File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py",
line 379, in run_sequential
   list(self.pull())
 File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py",
line 456, in pull
   out = coro.send(msg)
 File "/usr/local/lib/python2.7/dist-packages/beets/util/pipeline.py",
line 184, in coro
   func(*(args + (task,)))
 File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line
1265, in lookup_candidates
   task.lookup_candidates()
 File "/usr/local/lib/python2.7/dist-packages/beets/importer.py", line
592, in lookup_candidates
   autotag.tag_album(self.items, search_ids=self.search_ids)
 File "/usr/local/lib/python2.7/dist-packages/beets/autotag/match.py",
line 439, in tag_album
   search_album, va_likely)
 File "/usr/local/lib/python2.7/dist-packages/beets/autotag/hooks.py",
line 580, in album_candidates
   out.extend(plugins.candidates(items, artist, album, va_likely))
 File "/usr/local/lib/python2.7/dist-packages/beets/plugins.py", line
357, in candidates
   out.extend(plugin.candidates(items, artist, album, va_likely))
                     File
"/usr/local/lib/python2.7/dist-packages/beetsplug/discogs.py", line 147,
in candidates
   return self.get_albums(query)
 File "/usr/local/lib/python2.7/dist-packages/beetsplug/discogs.py",
line 207, in get_albums
   type='release').page(1)
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/models.py", line
336, in page
   data = self.client._get(self._url_for_page(index))
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/client.py", line
110, in _get
   return self._request('GET', url)
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/client.py", line
97, in _request
   content, status_code = self._fetcher.fetch(self, method, url,
data=data, headers=headers)
 File
"/usr/local/lib/python2.7/dist-packages/discogs_client/fetchers.py",
line 73, in fetch
   resp, content = self.oauth_client.request(url, method, headers=headers)
 File "/usr/local/lib/python2.7/dist-packages/oauth2/__init__.py", line
682, in request
   connection_type=connection_type)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 1593, in request
   (response, content) = self._request(conn, authority, uri,
request_uri, method, body, headers, redirections, cachekey)
 File "/usr/local/lib/python2.7/dist-packages/httplib2/__init__.py",
line 1388, in _request
   redirections=redirections - 1)
 File "/usr/local/lib/python2.7/dist-packages/oauth2/__init__.py", line
662, in request
   req.sign_request(self.method, self.consumer, self.token)
 File "/usr/local/lib/python2.7/dist-packages/oauth2/__init__.py", line
493, in sign_request
   self['oauth_body_hash'] = base64.b64encode(sha(self.body).digest())
TypeError: must be string or buffer, not None

Here's a link to the music files that trigger the bug (if relevant): http://musicbrainz.org/release/5fcf4c41-80bb-491e-9612-bba21ffa2802

Setup

• OS: Ubuntu 14.04
• Python version: 2.7.6
• beets version: 1.3.17
• Turning off plugins made problem go away (yes/no): Turning off discogs made crash go away, but music wasn't recognized

My configuration (output of beet config) is:

lyrics:
    sources: [lyricwiki, lyrics.com, musixmatch]
    google_API_key: REDACTED
    force: no
    auto: yes
    genius_api_key: REDACTED
    google_engine_ID: REDACTED
    fallback:
importfeeds:
    relative_to: /home/music/playlists
    dir: /home/music/playlists
    formats: m3u
    absolute_path: yes
    m3u_name: newfiles.m3u

ui:
    color: no
pluginpath: /home/julf/beetsplug
missing:
    format: $albumartist - $album - $title
    count: no
    total: no
color: no
embedart:
    ifempty: yes
    compare_threshold: 0
    auto: yes
    remove_art_file: no
    maxwidth: 0
replaygain:
    backend: audiotools
library: ~/data/musiclibrary.blb
badfiles:
    commands:
        flac: flac --test --warnings-as-errors --silent

plugins: lyrics lastgenre embedart fetchart echonest chroma mbsync inline importfeeds missing duplicates edit badfiles discogs

paths:
    default: '%asciify{$albumartist}/%asciify{$album%aunique{}}/%if{$multidisc,Disc $disc/}$track - %asciify{$title}'
    singleton: Non-Album/%asciify{$artist}/%asciify{$title}
    comp: Compilations/%asciify{$album%aunique{}}/%if{$multidisc,Disc $disc/}$track %asciify{$title}
    xcomp: Compilations/%asciify{$album%aunique{albumartist album disc, albumtype year label catalognum albumdisambig}}/%if{$multidisc,Disc $disc/}$track %asciify{$title}
convert:
    dest: /home/music/empeg/mp3s/
    embed: yes
    max_bitrate: 192
    format: mp3
    auto: yes
    never_convert_lossy_files: yes
    command: /home/music/progs/runlame $source $dest
    extension: mp3
threaded: no
pathfields:
    multidisc: 1 if disctotal > 1 else 0
logmetadatachanges:
    logfile: /home/music/metadatachanges.log
lastgenre:
    canonical: ''
    count: 1
    source: album
    force: yes
    min_weight: 10
    auto: yes
    whitelist: yes
    separator: ', '
    fallback:
timeout: 1
directory: /home/music/all

import:
    log: ~/beets.log
    incremental: yes
    write: yes
echonest:
    convert: no
    apikey: REDACTED
    auto: yes
    upload: yes
    truncate: yes
    energy: energy
    liveness: liveness
    tempo: bpm
    speechiness: speechiness
    acousticness: acousticness
    danceability: danceability
    valence: valence
chroma:
    auto: yes
album_fields: {}
item_fields: {}
fetchart:
    minwidth: 0
    sources:
    - coverart
    - itunes
    - amazon
    - albumart
    cautious: no
    google_engine: 001442825323518660753:hrh5ch1gjzm
    maxwidth: 0
    google_key: REDACTED
    auto: yes
    cover_names:
    - cover
    - front
    - art
    - album
    - folder
    remote_priority: no
    enforce_ratio: no
discogs:
    tokenfile: discogs_token.json
    apikey: REDACTED
    apisecret: REDACTED
    source_weight: 0.5
edit:
    itemfields: track title artist album
    albumfields: album albumartist
    ignore_fields: id path
duplicates:
    count: no
    full: no
    format: ''
    keys: []
    move: ''
    tag: ''
    path: no
    copy: ''
    tiebreak: {}
    album: no
    strict: no
    checksum: ''
    merge: no
    delete: no

[sampsyo]

Thanks for reporting! Can you help us reproduce this? I see you have a link to a MusicBrainz album, but you said this only matches in Discogs? Is there something I can do on my end to reproduce this, either using your files or by entering a manual search or something?

[Julf]

It is this one on Discogs: https://www.discogs.com/Husky-Rescue-The-Long-Lost-Friend-/release/6948755

More than happy to send you the actual files - what would be the best way?

[sampsyo]

Perhaps a Dropbox link?

[Julf]

Sent you a dropbox link by email - came across another album that also triggers the same issue.

[sampsyo]

Hmm... I gave it a try, but everything seemed to work fine (after I entered the Discogs URL):

$ beet imp /mnt/d/bad\ album
/mnt/d/bad album (11 items)
Finding tags for album " - ".
Candidates:
1. Various Artists - Bourbon Street Parade (31.0%) (tracks, album, source, ...) (Discogs, 2xVinyl, 1985, Netherlands, Dureco Benelux)
2. Tozovac - Uspesi (22.9%) (tracks, album, artist, ...) (Discogs, 2xCassette, 1980, Yugoslavia, Beograd Disk)
3. Jan Werich - Čochtan Vypravuje (20.8%) (tracks, unmatched tracks, album, ...) (Discogs, 4xShellac, Czechoslovakia, Supraphon)
4. Fiona - Together (5.7%) (unmatched tracks, tracks, album, ...) (Discogs, 2xVinyl, US, Jasfar Records)
5. Capcap... Cap - Woven (4.6%) (tracks, unmatched tracks, album, ...) (Discogs, Cassette, 2014, UK, Not On Label (Capcap... Cap Self-released))
# selection (default 1), Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? i
Enter release ID: https://www.discogs.com/Husky-Rescue-The-Long-Lost-Friend-/release/6948755
Correcting tags from:
    (unknown album)
To:
    Husky Rescue - The Long Lost Friend 
URL:
    https://www.discogs.com/Husky-Rescue-The-Long-Lost-Friend-/release/6948755
(Similarity: 20.2%) (tracks, missing tracks, album, artist, source) (Discogs, 2xCD, 2015, Europe, Catskills Records Ltd.)
CD 1: The Long Lost Friend
 * 11 - Rockland.flac (#0)                -> Under Friendly Fire (#2) (title)
 * 06 - Shine Like Neon Rays.flac (#0)    -> Colors (#4) (title)
 * 07 - Driving After You.flac (#0)       -> June (#5) (title)
 * 08 - Flash In The Dark.flac (#0)       -> Mountains Only Know (#6) (title)
 * 10 - Bad Girl.flac (#0)                -> The Long Lost Friend (#7) (title)
 * 05 - My Own Time.flac (#0) (2:46)      -> Min Lilla Eld (#9) (2:36) (title, length)
 * 04 - Lady Grey.flac (#0) (3:18)        -> Sunrider (#12) (3:34) (title, length)
CD 2: Hypnopompic
 * 03 - Oh My God.flac (#0) (3:24)        -> Pistachio Tree (#14) (3:43) (title, length)
 * 02 - Curvaceous Needs.flac (#0) (3:26) -> Wind In The Willows II (#18) (3:44) (title, length)
 * 01 - Old De Spain.flac (#0)            -> Deep Forest Green (Clouds Interpretation) (#23) (title)
 * 09 - My Dear.flac (#0) (3:04)          -> Fast Lane (ID Session Audio) (#24) (3:47) (title, length)
Missing tracks (13/24 - 54.2%):
 ! Restless Feet (#1) (4:18)
 ! River (#3) (5:07)
 ! Tree House (#8) (7:23)
 ! Deep Forest Green (#10) (3:59)
 ! Wind In The Willows (#11) (3:59)
 ! Skin Of Snow (#13) (3:53)
 ! Sunrise In The Mist (#15) (8:53)
 ! Robot Requiem (#16) (2:15)
 ! Werewolf & Fools (#17) (4:55)
 ! Jigsaw Puzzle (#19) (1:54)
 ! Far From The Storm (#20) (4:04)
 ! Onnellisuus (#21) (4:04)
 ! Tree House (Clouds Remix) (#22) (5:00)
Apply, More candidates, Skip, Use as-is, as Tracks, Group albums,
Enter search, enter Id, aBort? a
$

I'm stumped! Any chance deleting your ls ~/.config/beets/discogs_token.json and re-authorizing has an effect?

[Julf]

Deleted discogs_token.json - but now import doesn't prompt me to reauthorize, it just tells me "error: communication with Discogs failed"

[sampsyo]

Wow; that's even stranger. Anything in the verbose log?

[Julf]

Sending event: library_opened Sending event: import_begin discogs: connection error: 302: Invalid response from request token URL.

[sampsyo]

OK---I notice you have a custom Discogs API key in your configuration:

discogs:
    tokenfile: discogs_token.json
    apikey: REDACTED
    apisecret: REDACTED
    source_weight: 0.5

Does it help to remove that?

[Julf]

It is not in my config file.

[sampsyo]

That's odd. Could you perhaps paste your whole config file?

[Julf]

color: false
timeout: 1
pluginpath: /home/julf/beetsplug
threaded: false
directory: /home/music/all
library: ~/data/musiclibrary.blb
plugins: lyrics lastgenre embedart fetchart echonest chroma mbsync inline importfeeds missing duplicates edit badfiles discogs # beatport convert logmetadatachanges 

paths:
    default: %asciify{$albumartist}/%asciify{$album%aunique{}}/%if{$multidisc,Disc $disc/}$track - %asciify{$title}
    singleton: Non-Album/%asciify{$artist}/%asciify{$title}
    comp: Compilations/%asciify{$album%aunique{}}/%if{$multidisc,Disc $disc/}$track %asciify{$title}
    xcomp: Compilations/%asciify{$album%aunique{albumartist album disc, albumtype year label catalognum albumdisambig}}/%if{$multidisc,Disc $disc/}$track %asciify{$title}
import:
    log: ~/beets.log
    incremental: yes
    write: yes
lastgenre:
    canonical: ''
logmetadatachanges:
    logfile: /home/music/metadatachanges.log
importfeeds:
    formats: m3u
    dir: /home/music/playlists
    absolute_path: yes
    m3u_name: newfiles.m3u
pathfields:
    multidisc: 1 if disctotal > 1 else 0
echonest:
    convert: no
    apikey: #############
missing:
    format: $albumartist - $album - $title
    count: no
    total: no
convert:
    dest: /home/music/empeg/mp3s/
    embed: yes
    max_bitrate: 192
    format: mp3
    auto: yes
    never_convert_lossy_files: yes
    command: /home/music/progs/runlame $source $dest
    extension: mp3
replaygain:
    backend: audiotools
embedart: 
    #compare_threshold: 30
    ifempty: yes
badfiles:
    commands:
        flac: flac --test --warnings-as-errors --silent

[Julf]

And just to be sure, I ran "find /home/ -type f -print0 | xargs -0 grep discogs_token.json". Only file with a mention of discogs_token.json was the plugin itself.

[sampsyo]

Thanks for all the extra investigation. I'm mystified about both:

• Why discogs seems completely broken for you, while it works for me and you haven't changed any of the config.
• Why beet config is showing a bunch of default values even though you didn't override them.

One last crazy idea: I've noticed that some authentication servers depend on the time being set correctly. Can you check your system clock to make sure it's right? (This sometimes goes wrong in virtual machines and such.)

[Julf]

I am mystified too. Time is set correctly (real, physical server (as opposed to VM) and synchronized to several NTP servers.

[sampsyo]

Scrounging for other ideas: perhaps the discogs_client module needs an update? You can try with pip install -U discogs_client.

[Julf]

That seems to have done it! Now it asked for a new authorization key and doesn't crash any more. I guess I was under the naive impression that pip install -U beets updates the modules as well. Anyway, thanks, really appreciate the help!

[sampsyo]

Aha! Well, glad that's fixed. I should have thought of that earlier!

FWIW, pip install -U beets upgrades beets and its direct dependencies, but it doesn't do anything about plugin dependencies.

[Julf]

Is there any handy way to upgrade all plugins, or does that have to be done plugin-by-plugin?

[jackwilsdon]

You can update any of the plugins found in the extra_requires section of setup.py.

You can do this like so:

pip install -U beets[plugin_name]

And to do multiple:

pip install -U beets[first_plugin,second_plugin,...]

Here's the command to update all plugins (although be warned, it will install plugins that are not already installed and may fail for metasync (see install log for #1943)):

pip install -U beets[fetchart,chroma,discogs,echonest,lastgenre,mpdstats,web,import,thumbnails,metasync]

And in your case, here is how you can update just discogs (although I see you have already done it):

pip install -U beets[discogs]

[thedevilisinthedetails]

Thanks @jackwilsdon !! I guess i had to update a few more plugins as well. This issue can be closed

[jackwilsdon]

You commented on the wrong issue, haha! I'll close #1993 anyway, glad it's sorted out 😄.