search

joalla / discogs_client

Continuation of the "Official Python Client for the Discogs API"
https://python3-discogs-client.readthedocs.io
Other
310 stars 50 forks source link

Meta issue for JSONDecode errors we see often lately #163

Open JOJ0 opened 2 weeks ago

JOJ0 commented 2 weeks ago

Several users have reported similar randomly happening JSONDecode errors when retrieving arbitrary stuff via python3-discogs-client. I experienced it myself and post my experience soon. I'd like to summarize discussion and improvement suggestions in this issue here.

In my opinion it looks like api.discogs.com responds with unexpected data but only sometimes. I would guess it is a problem on their end for whatever reason, and they might be aware of and working on it already, or not..... In the end we have to live with it and find a workaround.

A second reason is that our backoff mechanism could be faulty or not compatible anymore since something might have changed at Discogs. It worked very well for us so far and it's in place since ...about 4 years already....https://github.com/joalla/discogs_client/pull/34

Related reports so far:

Did any of you solve the issue simply with try/except catchin jsondecode errors? If yes, share you experience, maybe a simple fix like that can be included in python3-discogs-client already.

Can this problem be reproduced when using api.discogs.com simply with Python requests module only?

JOJ0 commented 1 week ago

This is what I do to catch such errors in an application using discogs_client. For example this loops through my sales inventory and imports into an sqlite database. Instead of crashing I'd rather have missing entries. This also includes an error counter, since I currently want to learn how often these errors happen.

            for listing in self.collection.me.inventory:
                try:
                    self.collection.create_sales_entry({
                        "d_sales_release_id": listing.release.id,
                        "d_sales_listing_id": listing.id,
                        "d_sales_release_url": listing.release.url,
                    })
                except JSONDecodeError as e:
                    log.error("Catched a JSONDecodeError. Not retrying! %s", e)
                    decode_err += 1
                except Exception as e:
                    log.error("Catched an Exception. Not retrying! %s", e)
                    other_err += 1

        print(f"Discogs JSONDecode errors : {decode_err}.")
        print(f"Other errors : {other_err}.")

I'm not 100% sure since this error is hard to reproduce but I'm pretty sure this must catch JSONDecodeErrors, no matter where they happen!

JOJ0 commented 1 week ago

@AnssiAhola do you think we could include the catching of the JSONDecodeError (only! not other exceptions!) into the core of the client?

I'm not sure if simply wrapping the "main fetch wrapper":

https://github.com/joalla/discogs_client/blob/98059c0d00a3c65afa8f1e514d3a6f1327cade3c/discogs_client/fetchers.py#L57-L61

in a try except is the best idea or an approach like the backoff decorator is a better solution: https://github.com/joalla/discogs_client/blob/98059c0d00a3c65afa8f1e514d3a6f1327cade3c/discogs_client/utils.py#L44-L48

or is all this a bad idea anyway? And of course we should add tests for it!