termie / lp2gh

launchpad to github migration tool
19 stars 13 forks source link

Incremental back-off to circumbent Github abuse detection #4

Open timfel opened 8 years ago

timfel commented 8 years ago

When migrating larger projects, Github's abuse detection triggers at some point. To still allow the migration to go on, incrementally back off, and also check the rate limits for the API to check how long we'll have to wait.

gunchleoc commented 4 years ago

I tried to use this patch (without using the new script, since I already had the JSON data) to migrate > 3000 issues and it failed before finishing second pass for all issues.

second pass on issue Ingame README needs review
Traceback (most recent call last):
  File "./bin/lp2gh-import-bugs", line 30, in <module>
    o = bugs.import_(repo, data, milestones_map=milestones_map)
  File "<path>/lp2gh/lp2gh/bugs.py", line 267, in import_
    limit_retry(e, repo, lambda: comments.append(body='%s\n%s' % (by_line, msg['content'])))
  File "<path>/lp2gh/lp2gh/bugs.py", line 49, in limit_retry
    return try_block()
  File "<path>/lp2gh/lp2gh/bugs.py", line 267, in <lambda>
    limit_retry(e, repo, lambda: comments.append(body='%s\n%s' % (by_line, msg['content'])))
  File "<path>/lp2gh/github3/client.py", line 70, in append
    rv = self.client.post(self.url, **kw)
  File "<path>/lp2gh/github3/request.py", line 41, in post
    return self._opener.open(req)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 1241, in https_open
    context=self._context)
  File "/usr/lib/python2.7/urllib2.py", line 1201, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1121, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 438, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 402, in _read_status
    raise BadStatusLine("No status line received - the server has closed the connection")
httplib.BadStatusLine: No status line received - the server has closed the connection