beetbox / beets

music library manager and MusicBrainz tagger
http://beets.io/
MIT License
12.89k stars 1.82k forks source link

discogs: Apply rate limiting to all requests #3878

Closed insolentgeek closed 2 years ago

insolentgeek commented 3 years ago

I have disabled Musicbrainz as discussed in #400 while I initially import my entire collection while in timid mode. I was running into exceptions every couple minutes as a result of HTTP 429 errors: discogs_client.exceptions.HTTPError: 429: You are making requests too quickly.

I found that there is a mechanism to limit requests to their API but that it is not evenly applied. Such as when setting an ID for a release, it was just being sent without any regard to the delay. I wrapped that call but still was running into issues; after finding a couple more calls to the API and wrapping them, I was able to keep the 429 errors from happening with such regularity. I then changed the rate limiter to 50 instead of 60 and haven't seen a 429 since.

Here are the lines I wrapped with self.request_start() and self.request_finished(): https://github.com/beetbox/beets/blob/master/beetsplug/discogs.py#L218 https://github.com/beetbox/beets/blob/master/beetsplug/discogs.py#L268 https://github.com/beetbox/beets/blob/master/beetsplug/discogs.py#L294

sampsyo commented 3 years ago

Sounds good! Any chance you'd be interested in putting together a PR that does exactly that?

insolentgeek commented 3 years ago

I have become very busy with other projects, I don't see me doing a PR anytime soon.

JOJ0 commented 2 years ago

Hi, just stumbled accross this issue accidentally. We implemented rate limiting in python3-discogs-client around June 2021. Is this still an issue?

https://python3-discogs-client.readthedocs.io/en/latest/requests_rate_limit.html

sampsyo commented 2 years ago

Thank you for pointing this out! (And for maintaining the library!!) We removed all rate limiting on our side recently in https://github.com/beetbox/beets/pull/4118.

JOJ0 commented 2 years ago

Very welcome, and a big big thank you back! beets is such an amazing tool! Still learning how to use it for my ideas but I am getting there and love it already 🤩