search

WikiTeam / wikiteam

Tools for downloading and preserving wikis. We archive wikis, from Wikipedia to tiniest wikis. As of 2024, WikiTeam has preserved more than 600,000 wikis.
https://github.com/WikiTeam
GNU General Public License v3.0
714 stars 148 forks source link

Automatically resume and slow down on timeouts #70

Open emijrp opened 10 years ago

emijrp commented 10 years ago

From nemow...@gmail.com on November 05, 2013 20:35:07

wiki-site.com has some nasty configuration for which it stops responding, or sends error 403, if you do too many requests. I've tried 60, now 90 seconds delay and will probably find one which works with them in end, but it would be nice to have a better handling of this error than the current one:

Traceback (most recent call last): File "dumpgenerator.py", line 1195, in main() File "dumpgenerator.py", line 1152, in main config, other = getParameters(params=params) File "dumpgenerator.py", line 908, in getParameters if checkAPI(config['api']): File "dumpgenerator.py", line 931, in checkAPI f = urllib2.urlopen(req) File "/usr/lib/python2.6/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.6/urllib2.py", line 391, in open response = self._open(req, data) File "/usr/lib/python2.6/urllib2.py", line 409, in _open '_open', req) File "/usr/lib/python2.6/urllib2.py", line 369, in _call_chain result = func(*args) File "/usr/lib/python2.6/urllib2.py", line 1170, in http_open return self.do_open(httplib.HTTPConnection, req) File "/usr/lib/python2.6/urllib2.py", line 1145, in do_open raise URLError(err) urllib2.URLError: <urlopen error [Errno 110] Connection timed out> No tag found: dump failed, needs fixing; resume didn't work. Exiting.

Original issue: http://code.google.com/p/wikiteam/issues/detail?id=70

emijrp commented 10 years ago

From nemow...@gmail.com on November 05, 2013 11:42:53

Labels: -Type-Defect -Priority-Medium Type-Enhancement Priority-Low

emijrp commented 10 years ago

From nemow...@gmail.com on November 05, 2013 11:43:10

Blocking: wikiteam:33

emijrp commented 10 years ago

From nemow...@gmail.com on November 08, 2013 14:40:39

Well, and maybe also urllib2.HTTPError: HTTP Error 503: Service Temporarily Unavailable which I get sometimes from them too.

nemobis commented 6 years ago

This should be largely fixed with https://github.com/WikiTeam/wikiteam/commit/142b48cc6926a0ed4b3e0a97a1040932d0ec7c87 but we'll see how well requests does.