ChaosBot is down

[syvb]

Dates and comment scores aren't updating. Appears to be that the Github API isn't responding correctly (from logs). For the commit with ID: "92857c62f631ae73627e2eaac8e650a6cb8a0bd9" in chaosbot/chaos, the web UI returns commit information:  https://github.com/chaosbot/Chaos/commit/92857c62f631ae73627e2eaac8e650a6cb8a0bd9 but the API does not: https://api.github.com/repos/chaosbot/Chaos/statuses/92857c62f631ae73627e2eaac8e650a6cb8a0bd9 Also, I can't find any documentation for calling the API with a GET request, only the POST message. The only way to change the endpoint used is to open a PR, which @chaosbot will ignore, because the API doesn't work. Maybe Github will fix the API issue, or maybe we will need some manual intervention to let @chaosbot keep running.

For another example, notice how #338 is stuck at 3:14 time remaining.

[phil-r]

Travis statuses are not returned by github api for some reason ;(

[amoffat]

Hotfix here https://github.com/chaosbot/Chaos/commit/295545f87bf8fe9f5dfc56597dcb53b7e223ebd9, needs more investigation. Props to @phil-r and @Smittyvb for looking into it so far

[amoffat]

Some additional details on the original PR https://github.com/chaosbot/Chaos/pull/331#issuecomment-304549940

[phil-r]

it looks like it's down completely now ;(

[PlasmaPower]

Or is it? It appears to be not down, but just very broken.

[amoffat]

the current killer is running out of api requests for a few different reasons:

05-29 00:54 github_api   DEBUG    requests remaining: 5, reset in: 26s, cooldown sleep: 14.79s
05-29 00:55 github_api   INFO     requesting GET to '/repos/chaosbot/Chaos/statuses/8421c56baa717912c7154c47fde9660abe4cf6a3'
Traceback (most recent call last):
  File "chaos.py", line 60, in <module>
    main()
  File "chaos.py", line 55, in main
    schedule.run_pending()
  File "/root/.virtualenvs/chaos/lib/python3.6/site-packages/schedule/__init__.py", line 462, in run_pending
    default_scheduler.run_pending()
  File "/root/.virtualenvs/chaos/lib/python3.6/site-packages/schedule/__init__.py", line 75, in run_pending
    self._run_job(job)
  File "/root/.virtualenvs/chaos/lib/python3.6/site-packages/schedule/__init__.py", line 129, in _run_job
    ret = job.run()
  File "/root/.virtualenvs/chaos/lib/python3.6/site-packages/schedule/__init__.py", line 377, in run
    ret = self.job_func()
  File "/root/workspace/Chaos/cron/poll_pull_requests.py", line 28, in poll_pull_requests
    for pr in prs:
  File "/root/workspace/Chaos/github_api/prs.py", line 182, in get_ready_prs
    build_passed = has_build_passed(api, pr["statuses_url"])
  File "/root/workspace/Chaos/github_api/prs.py", line 147, in has_build_passed
    statuses = api("get", statuses_path)
  File "/root/workspace/Chaos/github_api/__init__.py", line 67, in __call__
    resp.raise_for_status()
  File "/root/.virtualenvs/chaos/lib/python3.6/site-packages/requests/models.py", line 928, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 403 Client Error: Forbidden for url: https://api.github.com/repos/chaosbot/Chaos/statuses/8421c56baa717912c7154c47fde9660abe4cf6a3

We had 5 requests remaining, so the api client slept a long time, but still the request failed. This is because github I think is just giving us a rough number of remaining requests. So we'll run out of requests, fail on a call, process dies, supervisor restarts the process, but the first thing we do is make an api request, which fails. Supervisor then gives up trying to bring up chaos.

But the underlying issue is why we are going through api requests so quickly...it looks like the caching patching got misplaced. We were making an api request for every voter/user, every time we looked at a PR, and were going through them too quickly.

https://github.com/chaosbot/Chaos/pull/352 addresses both of these

[chaosbot]

This issue hasn't been active for a while.To keep it open, react with :-1: on the vote close post.

[chaosbot]

Command has been run

[chaosbot]

/vote close

Time remaining: -1:08 - Vote status: passing

[chaosbot]

:no_entry: The issue has been closed after a vote.