ForbiddenError: 403 Resource not accessible by integration

[jamielennox]

I'm seeing a bunch of errors saying that there is an operation that could not be performed by integration. This is on zuul.v2.opentechsjc.bonnyci.org:

2017-05-01 20:30:41,193 ERROR zuul.GithubWebhookListener: Exception when handling event:
Traceback (most recent call last):
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 117, in __dispatch_event
event = method(json_body)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 87, in wrapped_f
return r.call(f, *args, **kw)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 235, in call
do = self.iter(result=result, exc_info=exc_info)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 205, in iter
raise RetryError(fut).reraise()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 284, in reraise
raise self.last_attempt.result()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
return self.__get_result()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 238, in call
result = fn(*args, **kwargs)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 186, in _event_issue_comment
pr_body = self._issue_to_pull_request(body)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 87, in wrapped_f
return r.call(f, *args, **kw)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 235, in call
do = self.iter(result=result, exc_info=exc_info)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 205, in iter
raise RetryError(fut).reraise()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 284, in reraise
raise self.last_attempt.result()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
return self.__get_result()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 238, in call
result = fn(*args, **kwargs)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 240, in _issue_to_pull_request
pr_body = self.connection.getPull(owner, project, number)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 87, in wrapped_f
return r.call(f, *args, **kw)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 235, in call
do = self.iter(result=result, exc_info=exc_info)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 205, in iter
raise RetryError(fut).reraise()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 284, in reraise
raise self.last_attempt.result()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result
return self.__get_result()
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/__init__.py", line 238, in call
result = fn(*args, **kwargs)
File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 571, in getPull
pr = github.pull_request(owner, project, number)
File "/opt/venvs/zuul/src/github3.py/github3/github.py", line 1032, in pull_request
json = self._json(self._get(url), 200)
File "/opt/venvs/zuul/src/github3.py/github3/models.py", line 215, in _json
if self._boolean(response, status_code, 404) and response.content:
File "/opt/venvs/zuul/src/github3.py/github3/models.py", line 237, in _boolean
raise exceptions.error_for(response)
ForbiddenError: 403 Resource not accessible by integration 

I haven't had a chance to look at why yet. It appears to have started a while ago, but is ramping up: https://elk.bonnyci.org/goto/be0bd4b5f1d99b99ef0081cb9770ac02

I think that tenacity might have been a bad idea. There is already a retries= parameter we can pass to requests to retry connection errors internally, but this is forcing it to retry anything that raises and exception and there are a number of legitimate reasons to raise an exception.

[omgjlk]

Looking at github3.py, it appears that the pull_request method does an anonymous hit on the API to get the pull request details. https://github.com/sigmavirus24/github3.py/blob/b58ff53ce9607f71aeb06f46eefe991f83c5e83e/github3/github.py#L1012 seems like it should work, might have to set up an attempt outside of zuul to walk through, or put a rpdb debug statement in to catch when a 403 happens so we can walk back through what happened.

[omgjlk]

I just learned that @SpamapS still had his repo(s) set up to use the web hook and user method, and was generating events. I wonder if this was causing zuul to try to use the integration API to fetch his PR, and since the integration isn't installed there, was getting a denial. I'm not sure if the timestamps line up, but seems plausible. Clint is going the integration route now, so maybe we'll see a tail off on this.

I feel like we should be able to match up logging from events coming in to attempts to grab the PR from API, I just don't know how to do that easily without hand parsing log files.

[SpamapS]

I moved gearmand to integration configuration now. This has not been seen since May 26, however, I think nothing has been working since May 26 because of SSL init errors on Zuul

[SpamapS]

A restart masked the SSL errors (maybe requests holds its cacert dir open and then we replaced it? Dunno). The original 403's are not happening. I'm going to close this as a v2 issue we can reopen if v3 shows it.

[SpamapS]

Actually I just saw this.

2017-06-08 21:17:00,675 DEBUG zuul.GithubWebhookListener: Github Webhook Received. 2017-06-08 21:17:00,677 DEBUG zuul.GithubWebhookListener: Payload Signature: sha1=95b8e9187c15b05e306348f3a4bddb3062897a9a 2017-06-08 21:17:00,677 DEBUG zuul.GithubWebhookListener: Request Signature: sha1=95b8e9187c15b05e306348f3a4bddb3062897a9a 2017-06-08 21:17:00,678 DEBUG zuul.GithubWebhookListener: X-Github-Event: issue_comment 2017-06-08 21:17:05,478 ERROR zuul.GithubWebhookListener: Exception when handling event: Traceback (most recent call last): File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 117, in dispatch_event event = method(json_body) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 87, in wrapped_f return r.call(f, *args, **kw) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 235, in call do = self.iter(result=result, exc_info=exc_info) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 205, in iter raise RetryError(fut).reraise() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 284, in reraise raise self.last_attempt.result() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result return self.__get_result() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 238, in call result = fn(*args, **kwargs) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 186, in _event_issue_comment pr_body = self._issue_to_pull_request(body) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 87, in wrapped_f return r.call(f, *args, **kw) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 235, in call do = self.iter(result=result, exc_info=exc_info) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 205, in iter raise RetryError(fut).reraise() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 284, in reraise raise self.last_attempt.result() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result return self.__get_result() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 238, in call result = fn(*args, *kwargs) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 240, in _issue_to_pull_request pr_body = self.connection.getPull(owner, project, number) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 87, in wrapped_f return r.call(f, args, **kw) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 235, in call do = self.iter(result=result, exc_info=exc_info) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 205, in iter raise RetryError(fut).reraise() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 284, in reraise raise self.last_attempt.result() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result return self.get_result() File "/opt/venvs/zuul/local/lib/python2.7/site-packages/tenacity/init.py", line 238, in call result = fn(*args, **kwargs) File "/opt/venvs/zuul/local/lib/python2.7/site-packages/zuul/connection/github.py", line 571, in getPull pr = github.pull_request(owner, project, number) File "/opt/venvs/zuul/src/github3.py/github3/github.py", line 1032, in pull_request json = self._json(self._get(url), 200) File "/opt/venvs/zuul/src/github3.py/github3/models.py", line 215, in _json if self._boolean(response, status_code, 404) and response.content: File "/opt/venvs/zuul/src/github3.py/github3/models.py", line 237, in _boolean raise exceptions.error_for(response) ForbiddenError: 403 Resource not accessible by integration

zuul.opentechsjc.bonnyci.org:443 192.30.252.42 - - [08/Jun/2017:21:17:00 +0000] "POST /connection/github/payload HTTP/1.1" 200 3164 "-" "GitHub-Hookshot/43e4729"

Corresponds to cf4c2e00-4c8f-11e7-88d9-2cf864adb69a in https://github.com/organizations/BonnyCI/settings/apps/bonnyci/advanced

Oddly enough, that is a comment on another issue in projman, so my theory is that non-integrated repos that aren't setup, are causing this, and that it is a real issue we should make sure isn't happening in v3.