alex9smith / gdelt-doc-api

A Python client for the GDELT 2.0 Doc API
MIT License
100 stars 23 forks source link

Queries fail with `TypeError: argument of type 'int' is not iterable` #18

Open alex9smith opened 2 years ago

alex9smith commented 2 years ago

There have been several recent bug reports (#11 , #15 , #17 ) where queries fail with the error

File ~/code/gdelt-doc-api/gdeltdoc/api_client.py:79, in GdeltDoc.article_search(self, filters)
     64 """
     65 Make a query against the `ArtList` API to return a DataFrame of news articles that
     66 match the supplied filters.
   (...)
     76     A pandas DataFrame of the articles returned from the API.
     77 """
     78 articles = self._query("artlist", filters.query_string)
---> 79 if "articles" in articles:
     80     return pd.DataFrame(articles["articles"])
     81 else:

TypeError: argument of type 'int' is not iterable

These are all because the query string is invalid and the API returns an error message. The load_json helper isn't handling this case correctly, and it returns a long integer and not a Python dictionary (possibly the Unicode representation of the error string).

Then the line if "articles" in articles: fails because it's looking for a string in an int.

This isn't a great user experience.

The error message needs to be better when the query string is invalid.

I also need to address the root cause of these bugs - the library is generating invalid query strings. Test coverage of the various filter options is pretty poor at the moment so increasing that will hopefully find any remaining bugs.

alex9smith commented 2 years ago

I've just released 1.4.0 which includes a check for API errors like the ones causing these and passes the error message from the API to the user which at least will be more helpful.

Still need to check over all the other query parameters and sort out better test coverage