Problem with non-ascii characters when using the discogs plugin and oauth

[fhchl]

Just found this bug: when trying to import with the awesome discogs plugin I get the following error:

beet -v import /Volumes/Speicher/Music/iTunes/iTunes\ Media/Music/Nikolai\ Lugansky/Rachmaninov_\ Préludes,\ Moments\ Musicaux
user configuration: /Users/Franz/.config/beets/config.yaml
data directory: /Users/Franz/.config/beets
Sending event: pluginload
library database: /Users/Franz/fun/data/musiclibrary.blb
library directory: /Users/Franz/fun/music
Sending event: library_opened
Sending event: import_task_start
Looking up: /Volumes/Speicher/Music/iTunes/iTunes Media/Music/Nikolai Lugansky/Rachmaninov_ Préludes, Moments Musicaux
Tagging Nikolai Lugansky - Rachmaninov: Préludes, Moments Musicaux
No album IDs found.
Search terms: Nikolai Lugansky - Rachmaninov: Préludes, Moments Musicaux
Album might be VA: False
Traceback (most recent call last):
  File "/Users/Franz/anaconda/bin/beet", line 9, in <module>
    load_entry_point('beets==1.3.9', 'console_scripts', 'beet')()
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/ui/__init__.py", line 935, in main
    _raw_main(args)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/ui/__init__.py", line 925, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/ui/commands.py", line 872, in import_func
    import_files(lib, paths, query)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/ui/commands.py", line 844, in import_files
    session.run()
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/importer.py", line 295, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/util/pipeline.py", line 299, in run
    out = self.coro.send(msg)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/util/pipeline.py", line 181, in coro
    func(*(args + (task,)))
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/importer.py", line 1156, in lookup_candidates
    task.lookup_candidates()
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/importer.py", line 539, in lookup_candidates
    autotag.tag_album(self.items)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/autotag/match.py", line 432, in tag_album
    search_album, va_likely)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/autotag/hooks.py", line 557, in album_candidates
    out.extend(plugins.candidates(items, artist, album, va_likely))
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beets/plugins.py", line 301, in candidates
    out.extend(plugin.candidates(items, artist, album, va_likely))
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beetsplug/discogs.py", line 110, in candidates
    return self.get_albums(query)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/beets-1.3.9-py2.7.egg/beetsplug/discogs.py", line 156, in get_albums
    releases = self.discogs_client.search(query, type='release').page(1)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/discogs_client/models.py", line 336, in page
    data = self.client._get(self._url_for_page(index))
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/discogs_client/client.py", line 110, in _get
    return self._request('GET', url)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/discogs_client/client.py", line 97, in _request
    content, status_code = self._fetcher.fetch(self, method, url, data=data, headers=headers)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/discogs_client/fetchers.py", line 73, in fetch
    resp, content = self.oauth_client.request(url, method, headers=headers)
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/oauth2/__init__.py", line 676, in request
    uri = req.to_url()
  File "/Users/Franz/anaconda/lib/python2.7/site-packages/oauth2/__init__.py", line 440, in to_url
    urllib.urlencode(query, True), fragment)
  File "/Users/Franz/anaconda/lib/python2.7/urllib.py", line 1357, in urlencode
    l.append(k + '=' + quote_plus(str(elt)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 31-32: ordinal not in range(128)

My beets options are set like this:

plugins: discogs chroma keyfinder fetchart embedart # fromfilename

group_albums: yes
asciify_paths: yes

# Plugin options
lastgenre:
    fallback: ''

keyfinder:
    bin: /Applications/KeyFinder.app/Contents/MacOS/KeyFinder
    auto: no
    overwrite: no

fetchart:
    auto: no

chroma:
    auto: no

I'm using the most recent version and the asciify_path option has no effect on the error.

[sampsyo]

Thanks for the report. I can confirm this:

>>> from beetsplug import discogs
>>> p = discogs.DiscogsPlugin()
>>> p.get_albums(u'caf\xe9')
[ugly traceback]

Alas, I think this is a bug in discogs-client rather than beets. Investigating.

[sampsyo]

Bug filed: https://github.com/discogs/discogs_client/issues/40

Unfortunately, I don't think there's much we can do on our end. We can consider isolating the fault so beets at least avoids crashing, though.

[ghost]

i'm reopening this so we can track whether this gets a proper fix, since the discogs client now supports utf-8 queries. maybe we can check the version, so we still convert to ascii if their version is too old

[nicksellen]

I just ran into this error TypeError: sequence item 0: expected str instance, bytes found. Previously beets had been working fine.

I changed the code (at https://github.com/beetbox/beets/blob/c4347960eab78be2ad54f4e35b32928c47e287d1/beetsplug/discogs.py#L245-L248) from:

query = re.sub(r'(?u)\W+', ' ', query).encode('ascii', "replace")
query = re.sub(br'(?i)\b(CD|disc)\s*\d+', b'', query)

To:

query = re.sub(r'(?u)\W+', ' ', query)
query = re.sub(r'(?i)\b(CD|disc)\s*\d+', '', query)

And that resolved it.

Not sure why I encountered this again, as all the issues seem to be from years ago, and I have very new versions of everything. Here is the output from pip freeze and I'm running latest git version of beets which I installed with a fresh virtualenv and this update script. My python version is 3.9.2. I'm running arch.

I guess that workaround to encoding to ascii can be removed by now?

[sampsyo]

Huh; that's odd—from this snippet, I can't yet see any reason why we'd need that encode; it looks objectively wrong. Thanks for doing the investigation!!

Any chance you would be willing to open a quick PR with this change so we can check it out?

[nicksellen]

@sampsyo sure! Here you go --> https://github.com/beetbox/beets/pull/3885