Automatically resume and slow down on timeouts

[GoogleCodeExporter]

wiki-site.com has some nasty configuration for which it stops responding, or 
sends error 403, if you do too many requests. I've tried 60, now 90 seconds 
delay and will probably find one which works with them in end, but it would be 
nice to have a better handling of this error than the current one:

Traceback (most recent call last):
  File "dumpgenerator.py", line 1195, in <module>
    main()
  File "dumpgenerator.py", line 1152, in main
    config, other = getParameters(params=params)
  File "dumpgenerator.py", line 908, in getParameters
    if checkAPI(config['api']):
  File "dumpgenerator.py", line 931, in checkAPI
    f = urllib2.urlopen(req)
  File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.6/urllib2.py", line 391, in open
    response = self._open(req, data)
  File "/usr/lib/python2.6/urllib2.py", line 409, in _open
    '_open', req)
  File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.6/urllib2.py", line 1170, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.6/urllib2.py", line 1145, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 110] Connection timed out>
No </mediawiki> tag found: dump failed, needs fixing; resume didn't work. 
Exiting.

Original issue reported on code.google.com by nemow...@gmail.com on 5 Nov 2013 at 7:35

• Blocking: #33

[GoogleCodeExporter]

Original comment by nemow...@gmail.com on 5 Nov 2013 at 7:42

• Added labels: Priority-Low, Type-Enhancement
• Removed labels: Priority-Medium, Type-Defect

[GoogleCodeExporter]

Original comment by nemow...@gmail.com on 5 Nov 2013 at 7:43

• Now blocking: #33

[GoogleCodeExporter]

Well, and maybe also
    urllib2.HTTPError: HTTP Error 503: Service Temporarily Unavailable
which I get sometimes from them too.

Original comment by nemow...@gmail.com on 8 Nov 2013 at 10:40