alex9smith / gdelt-doc-api

A Python client for the GDELT 2.0 Doc API
MIT License
91 stars 20 forks source link

Recommended fix for rate limit response #22

Closed pavelbrn closed 1 year ago

pavelbrn commented 1 year ago

I have been getting a brand new rate limit error from the GDELT API, even though I wasn't putting in a large amount of requests. I decided to check if the same request would work through my browser and it did.

Here is a small patch for the api_client.py script. I added this to line 150 in def _query() . It turns out the API now needs a user agents before returning any data(at least on my end). The following changes fixed my issue:

        headers = {
         "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36"
        }

        response = requests.get(
            f"https://api.gdeltproject.org/api/v2/doc/doc?query={query_string}&mode={mode}&format=json", headers=headers
        )

Perhaps an option to add a "User-Agent" would be useful for a future version?

Awesome project by the way!

alex9smith commented 1 year ago

Hey, thanks very much for investigating! I suspect the polite thing to do would be to set a user agent that includes the library name and version number. I'll try to release a new version of the client with that change soon.