search

palantir / policy-bot

A GitHub App that enforces approval policies on pull requests
Apache License 2.0
758 stars 103 forks source link

Question: Intermittent issue with resolving the repository #189

Closed zelin-l closed 4 years ago

zelin-l commented 4 years ago

Hello, When using the app we have recently been getting this error intermittently: "failed to compute disapproval status: failed to get last disapprover: failed to load pull request data: Could not resolve to a Repository with the name XXX"

Most of the time there are no issues but this error every once in a while. policy-bot is given access to the github repo in the error above.

Currently policy bot is deployed in kubernetes in two different clusters with two replicas each, if that helps.

Would anyone have any ideas what the issue could be? Much appreciated :)

bluekeyes commented 4 years ago

I believe GitHub returns this error in response to a GraphQL queries, but since policy-bot always sends the same query, it's surprising that the error is intermittent.

It might help to have some additional information about your situation:

  1. Is the repository always the same or does it happen for multiple repositories?
  2. Are you using GitHub Enterprise or GitHub.com? If GitHub Enterprise, which version?
  3. Which version of policy-bot are you using?
  4. If you have access to the server logs, can you share the log for this error from the server? If you have historical logs, does the error always happen with the same webhook event type or does it happen for multiple types of events?
zelin-l commented 4 years ago

Thanks for the quick response!

With regards to the questions:

  1. We run policy bot on several repositories. One of the repositories is a large monolith (150k+ commits) but others are not. But we have seen this across all the repositories
  2. We are on github teams (I could not find a version)
  3. We are using v1.15.0 for the bot (palantirtechnologies/policy-bot:1.15.0)
  4. From what I see this error happens about 1 in 100-200 calls. I don't see the webhook in the error logs (see below) but most of the calls look like this from what I see in the logs:

{"level":"info","rid":"brcpjc3qm95d35078al0","method":"POST","path":"/api/github/hook","status":200,"size":0,"elapsed":1115.395009,"time":"2020-06-05T01:02:41.410011685Z","message":"http_request"}

Here are the server logs for some of the errors (they all have the repository resolution error). I've X'ed out the repository names. They result in the PR being blocked. However, it seems like a comment would restart the process and can get the policy-bot to pass it. These 4 errors are from the same repo:

1 {"level":"warn","rid":"brcpa2rqm959ta97es7g","github_event_type":"pull_request_review","github_delivery_id":"7b8fa100-a6c5-11ea-9157-8d77308a5961","github_installation_id":813502,"github_repository_owner":"XXXXXXXXXXXX","github_repository_name":"XXXXXX","github_pr_num":10859,"github_sha":"c17b49e08366f8e9a25adff8f6b2a4c7ee8d1a7a","error":"failed to compute disapproval status: failed to get last disapprover: failed to load pull request data: Could not resolve to a Repository with the name 'XXXXXXXXXXXX/XXXXXX'.\ngithub.com/palantir/policy-bot/pull.(GitHubContext).loadPagedData\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:394\ngithub.com/palantir/policy-bot/pull.(GitHubContext).Comments\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:250\ngithub.com/palantir/policy-bot/policy/common.(Methods).Candidates\n\t/go/src/github.com/palantir/policy-bot/policy/common/methods.go:57\ngithub.com/palantir/policy-bot/policy/disapproval.(Policy).lastActor\n\t/go/src/github.com/palantir/policy-bot/policy/disapproval/disapprove.go:148\ngithub.com/palantir/policy-bot/policy/disapproval.(Policy).IsDisapproved\n\t/go/src/github.com/palantir/policy-bot/policy/disapproval/disapprove.go:111\ngithub.com/palantir/policy-bot/policy/disapproval.(Policy).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/disapproval/disapprove.go:92\ngithub.com/palantir/policy-bot/policy.evaluator.Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/policy.go:76\ngithub.com/palantir/policy-bot/server/handler.(Base).EvaluateFetchedConfig\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:175\ngithub.com/palantir/policy-bot/server/handler.(Base).Evaluate\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:150\ngithub.com/palantir/policy-bot/server/handler.(PullRequestReview).Handle\n\t/go/src/github.com/palantir/policy-bot/server/handler/pull_request_review.go:45\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp.(eventDispatcher).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp/dispatcher.go:178\ngithub.com/palantir/policy-bot/vendor/goji%2eio.dispatch.ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/dispatch.go:17\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Recover.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:107\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Catch.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.AccessHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:180\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.RequestIDHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:169\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp.NewMetricsHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp/middleware.go:55\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.NewHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:30\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/goji%2eio.(Mux).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/mux.go:74\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","time":"2020-06-05T00:42:52.241141934Z","message":"Error evaluating policy defined by XXXXXXXXXXXX/XXXXXX ref=master"}

2 {"level":"error","rid":"brcoqbrqm95d35077vrg","github_event_type":"pull_request","github_delivery_id":"cce96180-a6c0-11ea-887c-fc02ff8f5234","error":"Unable to select random request reviewers: failed to list repository collaborators: failed to load pull request data: Could not resolve to a Repository with the name 'XXXXXXXXXXXX/XXXXXX'.\ngithub.com/palantir/policy-bot/pull.(GitHubContext).loadCollaboratorData\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:444\ngithub.com/palantir/policy-bot/pull.(GitHubContext).RepositoryCollaborators\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:268\ngithub.com/palantir/policy-bot/policy/reviewer.SelectReviewers\n\t/go/src/github.com/palantir/policy-bot/policy/reviewer/reviewer.go:164\ngithub.com/palantir/policy-bot/server/handler.(Base).EvaluateFetchedConfig\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:212\ngithub.com/palantir/policy-bot/server/handler.(Base).Evaluate\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:150\ngithub.com/palantir/policy-bot/server/handler.(PullRequest).Handle\n\t/go/src/github.com/palantir/policy-bot/server/handler/pull_request.go:52\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp.(eventDispatcher).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp/dispatcher.go:178\ngithub.com/palantir/policy-bot/vendor/goji%2eio.dispatch.ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/dispatch.go:17\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Recover.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:107\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Catch.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.AccessHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:180\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.RequestIDHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:169\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp.NewMetricsHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp/middleware.go:55\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.NewHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:30\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/goji%2eio.(Mux).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/mux.go:74\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(conn).serve\n\t/usr/local/go/src/net/http/server.go:1890\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1357","time":"2020-06-05T00:09:25.498541399Z","message":"Unexpected error handling webhook request"}

3 {"level":"warn","rid":"brcorcjqm959ta97elsg","github_event_type":"pull_request","github_delivery_id":"1afe6500-a6c1-11ea-9c0a-aa4fcffb55df","github_installation_id":813502,"github_repository_owner":"XXXXXXXXXXXX","github_repository_name":"XXXXXX","github_pr_num":11226,"github_sha":"c913107002e85ccc18a9fd6a456916282485ddeb","error":"failed to compute approval status: failed to list commits: failed to load commits: Could not resolve to a Repository with the name 'XXXXXXXXXXXX/XXXXXX'.\ngithub.com/palantir/policy-bot/pull.(GitHubContext).loadRawCommits\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:529\ngithub.com/palantir/policy-bot/pull.(GitHubContext).loadCommits\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:462\ngithub.com/palantir/policy-bot/pull.(GitHubContext).Commits\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:234\ngithub.com/palantir/policy-bot/policy/approval.(Rule).filteredCommits\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:254\ngithub.com/palantir/policy-bot/policy/approval.(Rule).filteredCandidates\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:225\ngithub.com/palantir/policy-bot/policy/approval.(Rule).IsApproved\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:139\ngithub.com/palantir/policy-bot/policy/approval.(Rule).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:103\ngithub.com/palantir/policy-bot/policy/approval.(RuleRequirement).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:53\ngithub.com/palantir/policy-bot/policy/approval.(AndRequirement).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:120\ngithub.com/palantir/policy-bot/policy/approval.(AndRequirement).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:120\ngithub.com/palantir/policy-bot/policy/approval.(evaluator).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:33\ngithub.com/palantir/policy-bot/policy.evaluator.Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/policy.go:77\ngithub.com/palantir/policy-bot/server/handler.(Base).EvaluateFetchedConfig\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:175\ngithub.com/palantir/policy-bot/server/handler.(Base).Evaluate\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:150\ngithub.com/palantir/policy-bot/server/handler.(PullRequest).Handle\n\t/go/src/github.com/palantir/policy-bot/server/handler/pull_request.go:52\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp.(eventDispatcher).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp/dispatcher.go:178\ngithub.com/palantir/policy-bot/vendor/goji%2eio.dispatch.ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/dispatch.go:17\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Recover.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:107\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Catch.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.AccessHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:180\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.RequestIDHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:169\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp.NewMetricsHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp/middleware.go:55\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.NewHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:30\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/goji%2eio.(Mux).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/mux.go:74\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890","time":"2020-06-05T00:11:31.820741356Z","message":"Error evaluating policy defined by XXXXXXXXXXXX/XXXXXX ref=master"}

4 {"level":"warn","rid":"brcol33qm959ta97ej00","github_event_type":"pull_request","github_delivery_id":"39fbf780-a6bf-11ea-8965-503df7f0cfbc","github_installation_id":813502,"github_repository_owner":"XXXXXXXXXXXX","github_repository_name":"XXXXXX","github_pr_num":10581,"github_sha":"569768b14f7e7b1b78db07c3f60092948b04f179","error":"failed to compute approval status: failed to list commits: failed to load commits: Could not resolve to a Repository with the name 'XXXXXXXXXXXX/XXXXXX'.\ngithub.com/palantir/policy-bot/pull.(GitHubContext).loadRawCommits\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:529\ngithub.com/palantir/policy-bot/pull.(GitHubContext).loadCommits\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:462\ngithub.com/palantir/policy-bot/pull.(GitHubContext).Commits\n\t/go/src/github.com/palantir/policy-bot/pull/github.go:234\ngithub.com/palantir/policy-bot/policy/approval.(Rule).filteredCommits\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:254\ngithub.com/palantir/policy-bot/policy/approval.(Rule).filteredCandidates\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:225\ngithub.com/palantir/policy-bot/policy/approval.(Rule).IsApproved\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:139\ngithub.com/palantir/policy-bot/policy/approval.(Rule).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/approve.go:103\ngithub.com/palantir/policy-bot/policy/approval.(RuleRequirement).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:53\ngithub.com/palantir/policy-bot/policy/approval.(AndRequirement).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:120\ngithub.com/palantir/policy-bot/policy/approval.(AndRequirement).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:120\ngithub.com/palantir/policy-bot/policy/approval.(evaluator).Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/approval/evaluate.go:33\ngithub.com/palantir/policy-bot/policy.evaluator.Evaluate\n\t/go/src/github.com/palantir/policy-bot/policy/policy.go:77\ngithub.com/palantir/policy-bot/server/handler.(Base).EvaluateFetchedConfig\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:175\ngithub.com/palantir/policy-bot/server/handler.(Base).Evaluate\n\t/go/src/github.com/palantir/policy-bot/server/handler/base.go:150\ngithub.com/palantir/policy-bot/server/handler.(PullRequest).Handle\n\t/go/src/github.com/palantir/policy-bot/server/handler/pull_request.go:52\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp.(eventDispatcher).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-githubapp/githubapp/dispatcher.go:178\ngithub.com/palantir/policy-bot/vendor/goji%2eio.dispatch.ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/dispatch.go:17\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Recover.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:107\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear.Catch.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/bluekeyes/hatpear/hatpear.go:60\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.AccessHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:180\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.RequestIDHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:169\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp.NewMetricsHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/palantir/go-baseapp/baseapp/middleware.go:55\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog.NewHandler.func1.1\n\t/go/src/github.com/palantir/policy-bot/vendor/github.com/rs/zerolog/hlog/hlog.go:30\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2007\ngithub.com/palantir/policy-bot/vendor/goji%2eio.(Mux).ServeHTTP\n\t/go/src/github.com/palantir/policy-bot/vendor/goji.io/mux.go:74\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2802\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1890","time":"2020-06-04T23:58:05.937020131Z","message":"Error evaluating policy defined by XXXXXXXXXXXX/XXXXXX ref=master"}

zelin-l commented 4 years ago

Hm the error has just gotten much more frequent (fails 4/5 times now). The repository is private. I'm thinking if this could be an authorization issue with github? I'm thinking this error would not show up if the repo is public

zelin-l commented 4 years ago

We are now suspecting that it could be something on github's side (either rate limiting or errors). The error rate is fluctuating

bluekeyes commented 4 years ago

I agree this looks like a GitHub issue, particularly since it happens for 3 different GraphQL queries based on the logs you shared. If you'd like to try it outside of the context of policy-bot, here's a (untested) GraphQL query you can run manually. It's extracted from the loadRawCommits function that caused the error in logs 3 and 4:

query {
  repository(owner: "owner", name: "name") {
    pullRequest(number: 123) {
      commits(first: 100) {
        pageInfo {
          endCursor
          hasNextPage
        }
        nodes {
          commit {
            oid 
            author {
              __typename
              login
            }
            committer {
              __typename
              login
            }
            committedViaWeb
            pushedDate
            parents(first: 3) {
              nodes {
                oid
              }
            }
          }
        }
      }
    }
  }
}

You'll have to replace the owner, name and number values with the correct ones for your pull request.

In general, I expect rate limiting errors to have a different error code and message, but its possible that the v4 API behaves in a strange way.

zelin-l commented 4 years ago

Great thanks! I will try that in a bit. An update: we were forced to turn off the policy-bot for the big monolith repo to unblock PRs. Afterwards the policy-bot started to work 100% for all the other smaller repos.

Sent in a ticket to github but that will probably take some time to get a response. Will keep you updated.

zelin-l commented 4 years ago

Update: it does seem that this was something on Github's side but we will keep monitoring. Still haven't heard a response to the ticket. Will close this issue.

Thanks for the help!