PRs from forked repos failing to checkout on bitbucket.org

[idiotwithalaptop]

I ran into an interesting shortcoming recently that may need some attention.

Part of the standard practice I have been using when working with git repositories is that I always fork a copy of the repo, make my changes there, keep it up to date using git fetch remote and then contribute back using a pull request.

Turns out that when I fork / update / raise a PR on bitbucket.org, the buildkite agent was unable to checkout the changes. Here is an example of from the logs:

$ cd /var/lib/buildkite-agent/builds/build-123/account/project
# Host "bitbucket.org" already in list of known hosts at "/var/lib/buildkite-agent/.ssh/known_hosts"
$ git clone -v -- git@bitbucket.org:account/project.git .
Cloning into '.'...
remote: Counting objects: 531, done.
remote: Compressing objects: 100% (238/238), done.
remote: Total 531 (delta 248), reused 436 (delta 204)
Receiving objects: 100% (531/531), 21.45 MiB | 5.89 MiB/s, done.
Resolving deltas: 100% (248/248), done.
$ git submodule foreach --recursive git clean -ffxdq
$ git clean -ffxdq
$ git fetch -v origin ABCDEF123456
fatal: Couldn't find remote ref ABCDEF123456
$ git fetch -v --prune origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
From bitbucket.org:account/project
 = [up to date]      master     -> origin/master
$ git checkout -f ABCDEF123456
error: pathspec 'ABCDEF123456' did not match any file(s) known to git.
⚠️ Warning: Checkout failed! Error running `/usr/bin/git checkout -f ABCDEF123456`: exit status 1 (Attempt 1/3 Retrying in 2s)

Looking into this, turns out this is because unlike with github.com, bitbucket.org's refspecs & branches for pull requests are internal only. To the best of my knowledge there are 3 options:

• Only merge PRs from branches 👎
• using git clone --mirror 😨
• or make code change to the build agent to create a remote back to the origin of the PR (using envionment var BUILDKITE_PULL_REQUEST_REPO) and then attempt checkout. ❓

If you are interested in the 3rd option, I written a possible solution and would be more than happy to raise a PR.

Cheers, Ryan

[azz]

There's usually a question of trust when building pull request from forks. E.g. could a pull request be opened from a (malicious) public repository to a private repository to extract secrets from the build environment?

Does buildkite have any security controls for managing this?

[idiotwithalaptop]

@azz I would have thought that was more a concern of the origin repository, rather than buildkite. The origin repository should have the ability to control who can raise PRs and fork their code and also choose where those changes come from.

[azz]

@idiotwithalaptop I would hope so. I'm not sure how bitbucket.org handles public-private pull requests. I found this documentation from TravisCI.

Regardless this is tangential to the issue you've raised.

[lox]

Does buildkite have any security controls for managing this?

We certainly do for Github:

I'll have to investigate for Bitbucket.

@idiotwithalaptop it looks like recently Bitbucket started supporting git refs for pull requests: https://www.atlassian.com/git/articles/pull-request-proficiency-fetching-abilities-unlocked

We support setting a custom refspec in the agent via a special environment BUILDKITE_REFSPEC. You should be able to set this via a pre-checkout hook on your agent to something like:

export BUILDKITE_REFSPEC='+refs/pull-requests/*/from:refs/remotes/origin/pr/*'

I think that should do the trick! I'll get a bitbucket setup and do some testing and see if there is a way we can do that for you automatically.

[idiotwithalaptop]

@lox I noticed that blog a few days ago too, not sure how recent it is though as it still references atlassian.stash.com and the featured article for https://www.atlassian.com/git/articles dates back to 2014. I did try following what it said by using the refspec mentioned, but had no success unfortunately.

There is an open issue on Bitbucket about this (see https://bitbucket.org/site/master/issues/5814/reify-pull-requests-by-making-them-a-ref), but it's been open since 2012 so I'm not holding my breath for a fix anytime soon.