search

mediawiki-utilities / python-mwapi

Simple Python Wrapper around MediaWiki API
http://pythonhosted.org/mwapi
MIT License
31 stars 11 forks source link

Handle WikimediaError by retrying #26

Open halfak opened 8 years ago

halfak commented 8 years ago

Sometimes we get a response that looks like the following and this will raise a ValueError because it cannot be parsed as JSON:

<!DOCTYPE html>
<html lang=en>
<meta charset=utf-8>
<title>Wikimedia Error</title>
<style>
* { margin: 0; padding: 0; }
body { background: #fff; font: 15px/1.6 sans-serif; color: #333; }
.content { margin: 7% auto 0; padding: 2em 1em 1em; max-width: 560px; }
.footer { clear: both; margin-top: 14%; border-top: 1px solid #e5e5e5; background: #f9f9f9;

It seems that this can happen for many reason, but we should probably re-try the request a maximum number of times before giving up instead of just raising the error.

intracer commented 8 years ago

Are there any phabricator issues? Better to fix MediaWiki too

halfak commented 8 years ago

I've been filing them in phab as I find them. We've addressed a few that are related to query structures that result in inefficient queries. Many of these errors seem to be intermittent issues that are resolved on a retry. I think that a min retry that reports the error after a small number of failures will help us differentiate between the intermittent issues and problematic queries/revisions.