intermittent 404 Not Found on /pulls github calls

jacekn commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
If you are interested in working on this issue or have submitted a pull request, please leave a comment.

Overview of the Issue

I'm experiencing intermittent 404s on PRs. Atlantis leaves the following comment in the PR:

GET https://api.github.com/repos/myorg/myrepo/pulls/999/files?per_page=300: 404 Not Found []

Reproduction Steps

This happens intermittently but the steps are to:

open new PR
atlantis drops the comment with 404

Logs

```json {"level":"info","ts":"2021-11-19T11:50:25.342Z","caller":"events/events_controller.go:318","msg":"identified event as type \"opened\"","json":{}} {"level":"info","ts":"2021-11-19T11:50:25.342Z","caller":"events/events_controller.go:346","msg":"executing autoplan","json":{}} {"level":"error","ts":"2021-11-19T11:50:25.766Z","caller":"events/pull_updater.go:14","msg":"GET https://api.github.com/repos/myorg/myrepo/pulls/999/files?per_page=300: 404 Not Found []","json":{"repo":"myorg/myrepo","pull":"999"},"stacktrace":"github.com/runatlantis/atlantis/server/events.(*PullUpdater).updatePull\n\t/home/circleci/project/server/events/pull_updater.go:14\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).runAutoplan\n\t/home/circleci/project/server/events/plan_command_runner.go:77\ngithub.com/runatlantis/atlantis/server/events.(*PlanCommandRunner).Run\n\t/home/circleci/project/server/events/plan_command_runner.go:221\ngithub.com/runatlantis/atlantis/server/events.(*DefaultCommandRunner).RunAutoplanCommand\n\t/home/circleci/project/server/events/command_runner.go:162"} ```

Environment details

Atlantis v0.17.5

Atlantis server-side config file:

    repo-allowlist: github.com/myorg/myrepo
    allow-fork-prs: true

    default-tf-version: 0.13.7

    # See https://www.runatlantis.io/docs/checkout-strategy.html#merge
    checkout-strategy: merge

    gh-user: xxx

    data-dir: /atlantis
    atlantis-url: http://yyy

Repo atlantis.yaml file (atlantis is actually disabled for the

version: 3
automerge: true
delete_source_branch_on_merge: true
projects:
- dir: mydir1
  terraform_version: v0.13.7
  autoplan:
    when_modified: ["**", "../modules/**.tf"]
    enabled: true
- dir: mydir2
  autoplan:
    enabled: false

Environment variables:

    - name: ATLANTIS_PORT
      value: "4141"

Additional Context

Similar to

bmbferreira commented 2 years ago

I think this might be related with the failure on e2e tests that happened the other day after merging https://github.com/runatlantis/atlantis/pull/1884

kapilt commented 2 years ago

it feels like part of the issue might be the lack of exponential backoff in the retry fix https://github.com/runatlantis/atlantis/pull/1131/files

current code https://github.com/runatlantis/atlantis/blob/master/server/events/vcs/github_client.go#L333

it does look like its doing geometirc backoff now, but its unclear if we need to switch to exponential to resolve.

jacekn commented 2 years ago

Maybe it's worth also adding extra logging to the section to record the fact that retries are happening?

virgofx commented 2 years ago

Confirmed that we are experiencing this issue as well in 0.17.5. Would love for this to get fixed ASAP.

jamesclair commented 2 years ago

This issue is happening in 0.17.4 as well.

iainlane commented 2 years ago

Looks like something similar to #1131 is needed but in GetModifiedFiles

https://github.com/runatlantis/atlantis/blob/1b30fe04564fc301c27d25ec4e4caf98b4816e64/server/events/vcs/github_client.go#L123-L126

jamengual commented 2 years ago

is this still an issue with v0.19.8?

V1Ct0RHMz commented 1 year ago

It's happening with me on v0.20.1

darkside720 commented 1 year ago

its happening on 0.22 as well

runatlantis / atlantis