discogs: Rate limiting - Githubissues

arawden commented 5 years ago

Problem

I'm trying to run beets to import my music library, using the discogs plugin.

I run

$ beet import /mnt/music

Which yields:

Traceback (most recent call last):
  File "/usr/share/beets/beetsplug/discogs.py", line 226, in get_master_year
    year = result.fetch('year')
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 245, in fetch
    self.refresh()
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 211, in refresh
    data = self.client._get(self.data['resource_url'])
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 123, in _get
    return self._request('GET', url)
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 120, in _request
    raise HTTPError(body['message'], status_code)
discogs_client.exceptions.HTTPError: 429: You are making requests too quickly.

Some googling reveals that Discogs rate limits requests to about 3 per minute.

Running this command in verbose (-vv) mode:

$ beet -vv import /mnt/music

Causes the application to hang:

Candidate: Arctic Monkeys - Whatever People Say I Am, That’s What I’m Not (aa94ffcc-1c9b-39d6-953f-7aa803d8146c)
Computing track assignment...
...done.
Success. Distance: 0.01
discogs: Searching for master release 76279
discogs: Searching for master release 76279
discogs: Searching for master release 76279

Setup

OS: Ubuntu 18.10
Python version: 3.6.7
beets version: 1.4.7
Turning off plugins made problem go away (yes/no): Problem is with plugin

My configuration (output of beet config) is:

directory: /mnt/music
library: /mnt/music/beets.db
ignore: [0bleep, beats]
max_filename_length: 255
original_date: yes
per_disc_numbering: yes

ui:
    color: yes

import:
    copy: no
    move: yes
    timid: yes

paths:
    default: $albumartist/$album ($year) %aunique{}/$discn$track $title

plugins: inline edit discogs embedart fetchart replaygain scrub web
item_fields:
    discn: str(disc).lstrip('0') + '.' if disctotal > 1 else ''
discogs:
    source_weight: 0.0
    user_token: REDACTED
    apikey: REDACTED
    apisecret: REDACTED
    tokenfile: discogs_token.json
replaygain:
    backend: gstreamer
    overwrite: no
    auto: yes
    targetlevel: 89
    r128: [Opus]
web:
    host: 172.16.0.4
    port: 8000
    cors: '*'
    cors_supports_credentials: no
    reverse_proxy: no
    include_paths: no
edit:
    albumfields: album albumartist
    itemfields: track title artist album
    ignore_fields: id path
pathfields: {}
album_fields: {}
embedart:
    maxwidth: 0
    auto: yes
    compare_threshold: 0
    ifempty: no
    remove_art_file: no
scrub:
    auto: yes
fetchart:
    auto: yes
    minwidth: 0
    maxwidth: 0
    enforce_ratio: no
    cautious: no
    cover_names:
    - cover
    - front
    - art
    - album
    google_key: REDACTED
    google_engine: 001442825323518660753:hrh5ch1gjzm
    fanarttv_key: REDACTED
    store_source: no

sampsyo commented 5 years ago

Thanks for pointing this out! Looks like we need to add rate limiting to the Discogs plugin (like we already have for MusicBrainz and a few other APIs).

Causes the application to hang:

I think the importer prompt is just being buried by the verbose output. (Try hitting enter.)

arawden commented 5 years ago

That was definitely it! I'm going to add the full traceback in case you need it:

Sending event: database_change
Reimported item added 1542237203.1288793 from item 216 for /mnt/music/Aphex Twin/Drukqs (2001)/2.15 Nanou 2.mp3
Reimported item flexible attributes dict_keys(['data_source', 'track_alt']) from item 216 for /mnt/music/Aphex Twin/Drukqs (2001)/2.15 Nanou 2.mp3
Sending event: database_change
Traceback (most recent call last):
  File "/usr/share/beets/beetsplug/discogs.py", line 226, in get_master_year
    year = result.fetch('year')
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 245, in fetch
    self.refresh()
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/models.py", line 211, in refresh
    data = self.client._get(self.data['resource_url'])
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 123, in _get
    return self._request('GET', url)
  File "/home/avr/.local/lib/python3.6/site-packages/discogs_client/client.py", line 120, in _request
    raise HTTPError(body['message'], status_code)
discogs_client.exceptions.HTTPError: 429: You are making requests too quickly.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/bin/beet", line 11, in <module>
    load_entry_point('beets==1.4.7', 'console_scripts', 'beet')()
  File "/usr/share/beets/beets/ui/__init__.py", line 1256, in main
    _raw_main(args)
  File "/usr/share/beets/beets/ui/__init__.py", line 1243, in _raw_main
    subcommand.func(lib, suboptions, subargs)
  File "/usr/share/beets/beets/ui/commands.py", line 943, in import_func
    import_files(lib, paths, query)
  File "/usr/share/beets/beets/ui/commands.py", line 913, in import_files
    session.run()
  File "/usr/share/beets/beets/importer.py", line 329, in run
    pl.run_parallel(QUEUE_SIZE)
  File "/usr/share/beets/beets/util/pipeline.py", line 445, in run_parallel
    six.reraise(exc_info[0], exc_info[1], exc_info[2])
  File "/home/avr/.local/lib/python3.6/site-packages/six.py", line 693, in reraise
    raise value
  File "/usr/share/beets/beets/util/pipeline.py", line 312, in run
    out = self.coro.send(msg)
  File "/usr/share/beets/beets/util/pipeline.py", line 194, in coro
    func(*(args + (task,)))
  File "/usr/share/beets/beets/importer.py", line 1351, in lookup_candidates
    task.lookup_candidates()
  File "/usr/share/beets/beets/importer.py", line 641, in lookup_candidates
    autotag.tag_album(self.items, search_ids=self.search_ids)
  File "/usr/share/beets/beets/autotag/match.py", line 460, in tag_album
    va_likely):
  File "/usr/share/beets/beets/plugins.py", line 561, in decorated
    for v in generator(*args, **kwargs):
  File "/usr/share/beets/beets/autotag/hooks.py", line 616, in album_candidates
    for candidate in plugins.candidates(items, artist, album, va_likely):
  File "/usr/share/beets/beets/plugins.py", line 371, in candidates
    for candidate in plugin.candidates(items, artist, album, va_likely):
  File "/usr/share/beets/beetsplug/discogs.py", line 151, in candidates
    return self.get_albums(query)
  File "/usr/share/beets/beetsplug/discogs.py", line 216, in get_albums
    return [album for album in map(self.get_album_info, releases[:5])
  File "/usr/share/beets/beetsplug/discogs.py", line 216, in <listcomp>
    return [album for album in map(self.get_album_info, releases[:5])
  File "/usr/share/beets/beetsplug/discogs.py", line 303, in get_album_info
    original_year = self.get_master_year(master_id) if master_id else year
  File "/usr/share/beets/beetsplug/discogs.py", line 230, in get_master_year
    self._log.debug(u'API Error: {0} (query: {1})', e, result._uri)
AttributeError: 'Master' object has no attribute '_uri'

arawden commented 5 years ago

It looks like the Discogs client exposes a rate limit in the header of the response

Perhaps there is a way to check this before performing the next query

sampsyo commented 5 years ago

Neat! That should make it pretty straightforward.

jattenberg commented 5 years ago

experiencing the same issue.

sampsyo commented 5 years ago

Fixed in #3138.

beetbox / beets

discogs: Rate limiting #3081

Problem

Setup