Error when using the github backend

[geazzy]

hi everybody,

I have been had some troubles when using the github backend to get some informations.

Whe I run the following comand: bicho --db-user-out=* --db-password-out=* --db-database-out=DATABASENAME -b github -u https://api.github.com/repos/maqetta/maqetta/issues --backend-user=* --backend-password=****\ --debug

Sometimes I got the following error: DBG: [29/Nov/2013-22:00:24] https://api.github.com/repos/maqetta/maqetta/issues/3264 closed updated_at 2012-09-06T16:58:53Z (ratelimit = 4039) Error in function analyzeBug with URL: ' 'https://api.github.com/repos/maqetta/maqetta/issues and Bug: {u'body': u'This commit 18008340d7 broke the "proposed new CSS rule" feature (as described in the documentation page about adding a new rule to app.css). I did a checkout of the previous commit (3a3ef472121972b7af5cf9a378468b2ccf4fd525) and the feature is working with that commit.', u'labels': [{u'url': u'https://api.github.com/repos/maqetta/maqetta/labels/Properties+View', u'color': u'444444', u'name': u'Properties View'}, {u'url': u'https://api.github.com/repos/maqetta/maqetta/labels/Priority+High', u'color': u'F7AF07', u'name': u'Priority High'}, {u'url': u'https://api.github.com/repos/maqetta/maqetta/labels/Regression', u'color': u'FF9999', u'name': u'Regression'}, {u'url': u'https://api.github.com/repos/maqetta/maqetta/labels/Severity+2', u'color': u'e10c02', u'name': u'Severity 2'}], u'title': u'Proposed new rule feature broken', u'url': u'https://api.github.com/repos/maqetta/maqetta/issues/3264', u'labels_url': u'https://api.github.com/repos/maqetta/maqetta/issues/3264/labels{/name}', u'created_at': u'2012-09-05T23:12:43Z', u'events_url': u'https://api.github.com/repos/maqetta/maqetta/issues/3264/events', u'comments_url': u'https://api.github.com/repos/maqetta/maqetta/issues/3264/comments', u'html_url': u'https://github.com/maqetta/maqetta/issues/3264', u'comments': 2, u'number': 3264, u'updated_at': u'2012-09-06T16:58:53Z', u'assignee': {u'following_url': u'https://api.github.com/users/billreed63/following{/other_user}', u'events_url': u'https://api.github.com/users/billreed63/events{/privacy}', u'organizations_url': u'https://api.github.com/users/billreed63/orgs', u'url': u'https://api.github.com/users/billreed63', u'gists_url': u'https://api.github.com/users/billreed63/gists{/gist_id}', u'html_url': u'https://github.com/billreed63', u'subscriptions_url': u'https://api.github.com/users/billreed63/subscriptions', u'avatar_url': u'https://1.gravatar.com/avatar/e95349cbf47ddb62c3c8c73536fad7fa?d=https%3A%2F%2Fidenticons.github.com%2F5d24bde446d37dc7bb486e053d65928b.png&r=x', u'repos_url': u'https://api.github.com/users/billreed63/repos', u'received_events_url': u'https://api.github.com/users/billreed63/received_events', u'gravatar_id': u'e95349cbf47ddb62c3c8c73536fad7fa', u'starred_url': u'https://api.github.com/users/billreed63/starred{/owner}{/repo}', u'site_admin': False, u'login': u'billreed63', u'type': u'User', u'id': 717822, u'followers_url': u'https://api.github.com/users/billreed63/followers'}, u'state': u'closed', u'user': {u'following_url': u'https://api.github.com/users/JonFerraiolo/following{/other_user}', u'events_url': u'https://api.github.com/users/JonFerraiolo/events{/privacy}', u'organizations_url': u'https://api.github.com/users/JonFerraiolo/orgs', u'url': u'https://api.github.com/users/JonFerraiolo', u'gists_url': u'https://api.github.com/users/JonFerraiolo/gists{/gist_id}', u'html_url': u'https://github.com/JonFerraiolo', u'subscriptions_url': u'https://api.github.com/users/JonFerraiolo/subscriptions', u'avatar_url': u'https://1.gravatar.com/avatar/7d36042b898a8cb379cab49328f3f231?d=https%3A%2F%2Fidenticons.github.com%2F81f7e9edef99b4f38a38225c41fa8fab.png&r=x', u'repos_url': u'https://api.github.com/users/JonFerraiolo/repos', u'received_events_url': u'https://api.github.com/users/JonFerraiolo/received_events', u'gravatar_id': u'7d36042b898a8cb379cab49328f3f231', u'starred_url': u'https://api.github.com/users/JonFerraiolo/starred{/owner}{/repo}', u'site_admin': False, u'login': u'JonFerraiolo', u'type': u'User', u'id': 706721, u'followers_url': u'https://api.github.com/users/JonFerraiolo/followers'}, u'milestone': {u'description': u'', u'title': u'M7', u'url': u'https://api.github.com/repos/maqetta/maqetta/milestones/16', u'labels_url': u'https://api.github.com/repos/maqetta/maqetta/milestones/16/labels', u'created_at': u'2012-03-09T05:46:28Z', u'creator': {u'following_url': u'https://api.github.com/users/jhpedemonte/following{/other_user}', u'events_url': u'https://api.github.com/users/jhpedemonte/events{/privacy}', u'organizations_url': u'https://api.github.com/users/jhpedemonte/orgs', u'url': u'https://api.github.com/users/jhpedemonte', u'gists_url': u'https://api.github.com/users/jhpedemonte/gists{/gist_id}', u'html_url': u'https://github.com/jhpedemonte', u'subscriptions_url': u'https://api.github.com/users/jhpedemonte/subscriptions', u'avatar_url': u'https://0.gravatar.com/avatar/7aea57b6d1c49e9daf17743533d06e76?d=https%3A%2F%2Fidenticons.github.com%2Fc11d51e7fd3611d3345c752bebe19ea6.png&r=x', u'repos_url': u'https://api.github.com/users/jhpedemonte/repos', u'received_events_url': u'https://api.github.com/users/jhpedemonte/received_events', u'gravatar_id': u'7aea57b6d1c49e9daf17743533d06e76', u'starred_url': u'https://api.github.com/users/jhpedemonte/starred{/owner}{/repo}', u'site_admin': False, u'login': u'jhpedemonte', u'type': u'User', u'id': 711365, u'followers_url': u'https://api.github.com/users/jhpedemonte/followers'}, u'number': 16, u'updated_at': u'2013-03-01T17:32:18Z', u'due_on': u'2012-09-10T07:00:00Z', u'state': u'closed', u'closed_issues': 844, u'open_issues': 0, u'id': 94209}, u'closed_at': u'2012-09-06T16:58:53Z', u'pull_request': {u'diff_url': None, u'html_url': None, u'patch_url': None}, u'id': 6675608} Traceback (most recent call last): File "/usr/local/bin/bicho", line 25, in retval = Bicho.main.main() File "/usr/local/lib/python2.7/dist-packages/Bicho/main.py", line 56, in main backend.run() File "/usr/local/lib/python2.7/dist-packages/Bicho/backends/github.py", line 597, in run issue_data = self.analyze_bug(bug) File "/usr/local/lib/python2.7/dist-packages/Bicho/backends/github.py", line 437, in analyze_bug entries = self.get_batch_activities(bug['number']) File "/usr/local/lib/python2.7/dist-packages/Bicho/backends/github.py", line 485, in get_batch_activities result = urllib2.urlopen(request) File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen return _opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 406, in open response = meth(req, response) File "/usr/lib/python2.7/urllib2.py", line 519, in http_response 'http', request, response, code, msg, hdrs) File "/usr/lib/python2.7/urllib2.py", line 444, in error return self._call_chain(_args) File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain result = func(_args) File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default raise HTTPError(req.get_full_url(), code, msg, hdrs, fp) urllib2.HTTPError: HTTP Error 502: Bad Gateway

Has anyone had the same problem?

[geazzy]

Sorry, but i forgot to indicate the milestone... I'm using the Bicho 1.0.

[sduenas]

I think your problem is related to the rate limit defined by Github's API. You can only make up to ~5000 request per hour when you are authenticated and around 60 queries without authentication. See http://developer.github.com/v3/#rate-limiting for more info.

In Bicho backend-user and backend-password are the login parameters but probably for any reason are not working in Github backend. Can you check if you are using a valid username and password?

[geazzy]

I think it is another problem... because I tryed with my email and password that I use in my personal account. I had tried with Token (Basic Authentication) instead of my password, like this page has recommended https://help.github.com/articles/creating-an-access-token-for-command-line-use.

I have done some modifications to try to mine all the repository. In the file backends/github.py I modified the line 602, I changed the "raise" to "pass"... Although I got the same error, the program continue and I think that it get all the issues of repository, or almost all, because I got some errors yet.

Here is the log of my test: http://geazzy.com/bichoGitLog.txt

In this test I used the token instead of my password.

[jgbarah]

I've tried to reproduce your error, but I couldn't. I've run

bicho -d 1 --db-user-out=jgb --db-password-out=XXX --db-database-out=test_bicho -b github -u https://api.github.com/repos/maqetta/maqetta/issues --backend-user=jgbarah --backend-password=XXX --debug

And I've got (apparently) all issues. The log ends as:

DBG: [06/Jan/2014-23:50:56] Getting ticket number 3983
Done. 3971 bugs analyzed

I've found some warnings, such as:

...
DBG: [06/Jan/2014-23:50:17] https://api.github.com/repos/maqetta/maqetta/issues/3972 closed updated_at 2013-06-10T13:26:09Z (ratelimit = 4614)
ERROR: 
(1267, "Illegal mix of collations (latin1_swedish_ci,IMPLICIT) and (utf8_general_ci,COERCIBLE) for operation '='")
DBG: [06/Jan/2014-23:50:19] Getting ticket number 3972
...
DBG: [06/Jan/2014-23:50:55] https://api.github.com/repos/maqetta/maqetta/issues/3983 closed updated_at 2014-01-02T19:59:52Z (ratelimit = 4614)
/usr/lib/python2.7/dist-packages/storm/database.py:371: Warning: Incorrect string value: '\xE5\x9D\x91\xE8\xB4\xA7...' for column 'description' at row 1
  return function(*args, **kwargs)

From your error, I suspect temporary problems either at your IP path to github, or internal at the github platform. The HTTP 502 error is usually associated with over capacity, for example, see http://pcsupport.about.com/od/findbyerrormessage/a/502error.htm

So, for now I'm closing this one as not reproducible. Please, reopen if you have further info, or can figure out other ways to reproduce it. If you want the db I've produced, just ask: I'm keeping it for a while, just in case.