Change in API?

[DSoudis]

I have a piece of code that run several times untill yesterday but today gives me a JSONDecodeError. Could it be that the API Changed?

Thanks!

Code example: import wbgapi as w

wb.data.DataFrame('SP.POP.TOTL', time = 'all',l abels=True, skipAggs=True, columns='series')

Output:

JSONDecodeError Traceback (most recent call last) /app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/requests/models.py in json(self, kwargs) 959 try: --> 960 return complexjson.loads(self.content.decode(encoding), kwargs) 961 except UnicodeDecodeError:

/usr/local/lib/python3.9/json/init.py in loads(s, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw) 345 parse_constant is None and object_pairs_hook is None and not kw): --> 346 return _default_decoder.decode(s) 347 if cls is None:

/usr/local/lib/python3.9/json/decoder.py in decode(self, s, _w) 336 """ --> 337 obj, end = self.raw_decode(s, idx=_w(s, 0).end()) 338 end = _w(s, end).end()

/usr/local/lib/python3.9/json/decoder.py in raw_decode(self, s, idx) 354 except StopIteration as err: --> 355 raise JSONDecodeError("Expecting value", s, err.value) from None 356 return obj, end

JSONDecodeError: Expecting value: line 2 column 9 (char 9)

During handling of the above exception, another exception occurred:

JSONDecodeError Traceback (most recent call last) /app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/wbgapi/init.py in _queryAPI(url) 532 try: --> 533 result = response.json() 534 except:

/app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/requests/models.py in json(self, **kwargs) 967 except JSONDecodeError as e: --> 968 raise RequestsJSONDecodeError(e.msg, e.doc, e.pos) 969

JSONDecodeError: Expecting value: line 2 column 9 (char 9)

During handling of the above exception, another exception occurred:

APIResponseError Traceback (most recent call last)

in () 11 12 # Get data and tidy ---> 13 macro_data_df = wb.data.DataFrame('SP.POP.TOTL', time = 'all', 14 labels=True, skipAggs=True, columns='series') 15 /app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/wbgapi/data.py in DataFrame(series, economy, time, index, columns, mrv, mrnev, skipBlanks, labels, skipAggs, numericTimeKeys, timeColumns, params, db, **dimensions) 332 df2 = frame(index) 333 --> 334 for row in fetch(series, economy, time, mrv=mrv, mrnev=mrnev, skipBlanks=skipBlanks, labels=True, skipAggs=skipAggs, numericTimeKeys=numericTimeKeys, params=params, db=db, **dimensions): 335 column_key = row[columns]['id'] 336 if len(index) == 1: /app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/wbgapi/data.py in fetch(series, economy, time, mrv, mrnev, skipBlanks, labels, skipAggs, numericTimeKeys, params, db, **dimensions) 90 aggs = w.economy.aggregates() 91 ---> 92 for row in w.refetch(url, keys, params=params_, **values): 93 if skipBlanks and row['value'] is None: 94 continue /app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/wbgapi/__init__.py in refetch(url, variables, **kwargs) 320 try: 321 for url2 in _refetch_url(url, variables[0], variables[1:], **kwargs): --> 322 for row in fetch(url2, params, concepts, lang): 323 yield row 324 except URLError: /app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/wbgapi/__init__.py in fetch(url, params, concepts, lang) 279 280 url_ = '{}/{}/{}?{}'.format(endpoint, lang, url, urllib.parse.urlencode(params_)) --> 281 (hdr,result) = _queryAPI(url_) 282 283 if totalRecords is None: /app/dataiku/DSS_DATA_DIR/code-envs/python/medinfo_env/lib/python3.9/site-packages/wbgapi/__init__.py in _queryAPI(url) 533 result = response.json() 534 except: --> 535 raise APIResponseError(url, 'JSON decoding error') 536 537 hdr = _responseHeader(url, result) APIResponseError: APIError: JSON decoding error (https://api.worldbank.org/v2/en/sources/2/series/SP.POP.TOTL/country/all/time/all?per_page=1000&page=5&format=json)

[gauravrpjain]

Facing the same issue, for all data fetching functions including wb.data.fetch and wb.data.Dataframe.

[tgherzog]

I’ll look at this when I have computer access again in a couple of weeks. Conversations with my Bank colleagues suggest it might be a performance issue on the API end independent of the package. Can you post updates on whether this is still occurring?

[DSoudis]

Tried again today. It worked initially, but the same error occurred at some point. It seems the issue might indeed be from API side.

[tgherzog]

Erratic performance is almost always an API server issue. If it keeps misbehaving, I suggest contacting the World Bank at data@worldbank.org. I’ll test it myself when I’m back next week.

[gauravrpjain]

I also tried again today, it is working fine. I will test it over the next few days to see if it continues to work fine.
@tgherzog, I agree with your assessment that it is possibly an API server issue from the World Bank side.

[tgherzog]

This seems to be working with no changes to the module, so I'm assuming this is a performance issue for the API itself and closing this ticket.