WP API errors are not handled

[nyurik]

In a fairly common scenario when MediaWiki API returs an error, the app should pause for a second and repeat the API call, but no more than 3 times total.

Also, there should be a log entry notifying of the api error (last line with the dict is good)

2013-10-01 08:50:31+0000 [HTTP11ClientProtocol,client] Unhandled Error

  File "/var/praekelt/vumi-go/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia.py", line 361, in consume_user_message
    session = yield pfunc(msg, config, session)
  File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1068, in _inlineCallbacks
    result = result.throwExceptionIntoGenerator(g)
  File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/var/praekelt/vumi-go/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia.py", line 387, in process_message_searching
    results = yield self.get_wikipedia_api(config).search(query)
  File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1070, in _inlineCallbacks
    result = g.send(result)
  File "/var/praekelt/vumi-go/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia_api.py", line 187, in search
    raise APIError(response)
vumi_wikipedia.wikipedia_api.APIError: {u'servedby': u'mw1116', u'error': {u'info': u'Pool queue is full', u'code': u'srsearch-error'}} 

[smn]

Another similar error:

APIError: {u'servedby': u'mw1201', u'error': {u'info': u'The search backend returned an error: ', u'code': u'srsearch-error'}}

[smn]

Filed https://bugzilla.wikimedia.org/show_bug.cgi?id=60032 for this. On our side we need to handle these more gracefully and possible be more strict on our timeouts so we still have time to respond to the user within the USSD session window informing them of the problems reaching Wikipedia.

[nyurik]

Thanks Simon. Also FYI, we have just deployed the new search engine, currently enabled as beta, on EN wiki. We should move to it as a primary system fairly soon, assuming everything works ok. I suspect it will be migrated transparently in the API, so no changes in code, but should be much faster with fewer issues (one would hope, right?)

On Tue, Jan 14, 2014 at 4:06 AM, Simon de Haan notifications@github.comwrote:

Filed https://bugzilla.wikimedia.org/show_bug.cgi?id=60032 for this. On our side we need to handle these more gracefully and possible be more strict on our timeouts so we still have time to respond to the user within the USSD session window informing them of the problems reaching Wikipedia.

— Reply to this email directly or view it on GitHubhttps://github.com/praekelt/vumi-wikipedia/issues/28#issuecomment-32248554 .

[hodgestar]

More of the same error:

2014-01-27 18:57:08+0000 [HTTP11ClientProtocol,client] Unhandled Error
        Traceback (most recent call last):
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
            current.result = callback(current.result, *args, **kw)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1155, in gotResult
            _inlineCallbacks(r, g, deferred)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
        --- <exception caught here> ---
          File "/var/praekelt/vumi-go/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia.py", line 377, in consume_user_message
            session = yield pfunc(msg, config, session)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1097, in _inlineCallbacks
            result = result.throwExceptionIntoGenerator(g)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/python/failure.py", line 389, in throwExceptionIntoGenerator
            return g.throw(self.type, self.value, self.tb)
          File "/var/praekelt/vumi-go/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia.py", line 404, in process_message_searching
            results = yield self.get_wikipedia_api(config).search(query)
          File "/var/praekelt/vumi-go/ve/local/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
            result = g.send(result)
          File "/var/praekelt/vumi-go/ve/src/vumi-wikipedia/vumi_wikipedia/wikipedia_api.py", line 187, in search
            raise APIError(response)
        vumi_wikipedia.wikipedia_api.APIError: {u'servedby': u'mw1147', u'error': {u'info': u'The search backend returned an error: ', u'code': u'srsearch-error'}}

[smn]

I think #36 deals with this, can we now close this?

[smn]

bump

[jerith]

I think #36 deals with this, can we now close this?

This ticket is about retrying failed calls, #36 was about improved logging.

[smn]

ah, thanks.