Kodiak unable to update branch due to changed workflow file (maybe bug?)

paulgessinger commented 1 year ago

We're using kodiak over on https://github.com/acts-project/acts. It has been working fine for us for quite a while, but now we're getting issues with kodiak failing to update PR branches.

The error message reads:

API call 'pull_request/update_branch' failed with HTTP status '403' and response: 
'b'{"message":"refusing to allow a GitHub App to create or update workflow .github/workflows/analysis.yml without workflows permission",
"documentation_url":"[https://docs.github.com/rest/reference/pulls#update-a-pull-request-branch"}\](https://docs.github.com/rest/reference/pulls#update-a-pull-request-branch%22%7D%5C)''

I did change the branch protection rules a bit, but I think I changed them back with the errors still persisting. I'm attaching screenshots of the branch protection settings down below.

The PR in question does not actually touch the workflow file mentioned in the error message, but I believe the target branch has a commit that changes it, which kodiak tries to merge into the source branch.

Any idea what I might be doing wrong?

chdsbd commented 1 year ago

Hi @paulgessinger, thanks for reaching out

I think you may need to accept a new permission for your Kodiak installation. Try navigating to your installation settings and see if Kodiak is requesting new permissions: https://docs.github.com/en/organizations/managing-programmatic-access-to-your-organization/reviewing-your-organizations-installed-integrations

paulgessinger commented 1 year ago

Hey @chdsbd, thanks for the tip. I had checked this already and didn't see any request for changed permissions. The relevant page says "Read and write access to checks, code, issues, pull requests, and workflows" for kodiak, which would seem fine to me.

chdsbd commented 1 year ago

@paulgessinger okay, this definitely sounds like a bug then. Do you have the pull request where that issue occurred?

chdsbd commented 1 year ago

@paulgessinger It looks like it's occurring on this PR: https://github.com/acts-project/acts/pull/1512

One suspect issue is that PR is from a forked repository. Did you allow edits from maintainers in that PR?

It's possible this is a GitHub API bug.

paulgessinger commented 1 year ago

Hey @chdsbd sorry for not digging up a PR earlier. Indeed the "allow edits" option is set:

chdsbd commented 1 year ago

@paulgessinger Okay, I think this is a GitHub API bug. Because your installation has the workflow write permission. Additionally, the PR isn't even touching a workflow file. I'm going to open an issue with GitHub, but it might be good for you to do the same if you don't mind. https://support.github.com/contact/bug-report

paulgessinger commented 1 year ago

Thanks for looking at this. I'll create a bug report!

chdsbd commented 1 year ago

My ticket from late October was closed without this being escalated. I got another ping via #857 about this issue, so I've created another GitHub Support ticket. Hopefully I can get that issue escalated to someone that can help from GitHub.

chdsbd commented 1 year ago

This is still an issue and I'm still trying to get GitHub Support to recognize this as a bug to escalate to an engineer. I think I've given them a better reproduction step so hopefully we'll get a better response.

I'm sorry these bug has been outstanding for so long.

I created an example repository here: https://github.com/uncle-boris/github-bug/pull/3

Here's steps to reproduce:

create a new repository with account A. Enable a branch protection requirement on main to require pull requests to be up to date before merging.
with a second account, B, fork that repository owned by A and open a pull request to add a new file.
create a .github/ workflow in the repository with account A
The pull request opened by B should now be out of date. Attempt to update that pull request using a GitHub App. The API request will fail saying {"message":"refusing to allow a GitHub App to create or update workflow .github/workflows/blank.yml without workflows permission","documentation_url":"https://docs.github.com/rest/reference/pulls#update-a-pull-request-branch"}

Here's the curl request:

curl --request PUT \
  --url https://api.github.com/repos/uncle-boris/github-bug/pulls/3/update-branch \
  --header 'Accept: application/vnd.github.antiope-preview+json,application/vnd.github.merge-info-preview+json' \
  --header 'Authorization: Bearer ghs_redacted_token' \
  -vvv
*   Trying 140.82.113.5:443...
* Connected to api.github.com (140.82.113.5) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-CHACHA20-POLY1305-SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=GitHub, Inc.; CN=*.github.com
*  start date: Mar 16 00:00:00 2022 GMT
*  expire date: Mar 16 23:59:59 2023 GMT
*  subjectAltName: host "api.github.com" matched cert's "*.github.com"
*  issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: PUT]
* h2h3 [:path: /repos/uncle-boris/github-bug/pulls/3/update-branch]
* h2h3 [:scheme: https]
* h2h3 [:authority: api.github.com]
* h2h3 [user-agent: curl/7.84.0]
* h2h3 [accept: application/vnd.github.antiope-preview+json,application/vnd.github.merge-info-preview+json]
* h2h3 [authorization: Bearer ghs_redacted_token]
* Using Stream ID: 1 (easy handle 0x131011e00)
> PUT /repos/uncle-boris/github-bug/pulls/3/update-branch HTTP/2
> Host: api.github.com
> user-agent: curl/7.84.0
> accept: application/vnd.github.antiope-preview+json,application/vnd.github.merge-info-preview+json
> authorization: Bearer ghs_redacted_token
> 
< HTTP/2 403 
< server: GitHub.com
< date: Sat, 04 Feb 2023 20:52:31 GMT
< content-type: application/json; charset=utf-8
< content-length: 240
< x-github-media-type: github.v3; param=antiope-preview; format=json, github.merge-info-preview; format=json
< x-github-api-version-selected: 2022-11-28
< x-ratelimit-limit: 5000
< x-ratelimit-remaining: 4995
< x-ratelimit-reset: 1675547512
< x-ratelimit-used: 5
< x-ratelimit-resource: core
< access-control-expose-headers: ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Used, X-RateLimit-Resource, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, X-GitHub-SSO, X-GitHub-Request-Id, Deprecation, Sunset
< access-control-allow-origin: *
< strict-transport-security: max-age=31536000; includeSubdomains; preload
< x-frame-options: deny
< x-content-type-options: nosniff
< x-xss-protection: 0
< referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
< content-security-policy: default-src 'none'
< vary: Accept-Encoding, Accept, X-Requested-With
< x-github-request-id: C97F:5B18:4E88E6:A253A8:63DEC58F
< 
{
  "message": "refusing to allow a GitHub App to create or update workflow `.github/workflows/blank.yml` without `workflows` permission",
  "documentation_url": "https://docs.github.com/rest/reference/pulls#update-a-pull-request-branch"
}
* Connection #0 to host api.github.com left intact

chdsbd commented 1 year ago

Here's maybe more streamlined steps.

create a repository
install a GitHub App on the repository that has permission to Repository Contents, Repository Issues, Repository Pull Requests
with a separate GitHub account, fork the repository, and open a PR to add an arbitrary file.
with the original account that owns the repository, add a GitHub workflow. Now the forked repository is out of date because the original repository has a new workflow file.
with the GitHub App, call the /update-branch api to attempt to update the pull request branch. Verify you recieve an error about {"message":"refusing to allow a GitHub App to create or update workflow .github/workflows/blank.yml without workflows permission","documentation_url":"https://docs.github.com/rest/reference/pulls#update-a-pull-request-branch"}

It seems that if the pull request from the fork is behind the main repository, and the difference includes a new workflow file on the main branch, we get an error updating the branch.

chdsbd commented 1 year ago

There is no permission we can request that will allow the GitHub App to update the branch when the difference includes .github/workflow changes. 😞

I tried giving a GitHub App access to every permission and the app still couldn't update the branch.

chdsbd commented 1 year ago

GitHub says this is intended behavior

I heard from the engineering team and I understand that the behavior works as it should.

Adding workflows in this way is undesired because it would allow injecting malicious workflow files which directly run to expose secrets of the org/user holding the forked repository.

As mentioned in my last reply, this was implemented in response to some security concerns.

Please let me know if there is anything else we can help with.

paulgessinger commented 1 year ago

Ok I guess there is no path to 'fix' this then. But at least good to know it's understood.

chdsbd commented 1 year ago

I raised the issue a bit more and GitHub Support said they'd relay my concerns to the engineering team and let me know if there's any response. I'll update this issue if I hear anything

chdsbd / kodiak

Kodiak unable to update branch due to changed workflow file (maybe bug?) #853