Open tianon opened 5 months ago
The following flag to git fetch
would likely also be a sufficient solution, but fetching explicitly by the commit provided still seems worthwhile/important:
-p, --prune
Before fetching, remove any remote-tracking references that no
longer exist on the remote. Tags are not subject to pruning if they
are fetched only because of the default tag auto-following or due
to a --tags option. However, if tags are fetched due to an explicit
refspec (either on the command line or in the remote configuration,
for example if the remote was cloned with the --mirror option),
then they are also subject to pruning. Supplying --prune-tags is a
shorthand for providing the tag refspec.
See the PRUNING section below for more details.
$ docker buildx build ... https://github.com/tianon/dockerfiles.git#f23bacded613589161013e51fce6a06b8a9a350f:docker-master
...
#1 [internal] load git source https://github.com/tianon/dockerfiles.git#f23bacded613589161013e51fce6a06b8a9a350f:docker-master
fatal: Not a valid object name f23bacded613589161013e51fce6a06b8a9a350f^{commit}
From [https://github.com/tianon/dockerfiles](https://github.com/tianon/dockerfiles%1B[0m)
18c4789a..f23bacde master -> origin/master
error: cannot lock ref 'refs/remotes/origin/update/prometheus': 'refs/remotes/origin/update/prometheus/node-exporter' exists; cannot create 'refs/remotes/origin/update/prometheus'
! [new branch] update/prometheus -> origin/update/prometheus (unable to update local ref)
error: some local refs could not be updated; try running
'git remote prune origin' to remove any old, conflicting branches
...
(still affected periodically -- eventually my cache expires and the builds succeed :see_no_evil:)
Reproducing this is unfortunately really annoying -- you've got to create a repository with a branch name like
foo
, do a build (so that branch gets pulled into the BuildKit cache for the repo), then delete that remotefoo
branch and create a newfoo/bar
branch and do a build again with a new commit (so that it isn't just fully cached right away and tries to fetch again). It will then fail with an error similar to the above.I'd suggest a two-part remedy to this:
most importantly, GitHub (and some others) support direct fetching of commits such that we don't have to do this "fetch all the branches of the remote and hope one of them has the commit we're asking for" dance -- if we're given an explicit commit to fetch, we should start by trying to fetch that exact commit individually and only fetch all branches if that fails (this gets especially important if branches that aren't being fetched are large and unused/unnecessary for the current build :see_no_evil:)
before we fetch branches but after we've set up the remote, we should probably
git remote prune origin
as suggested in the error message :eyes:Edit: sorry, for context in my above error message, I have an automated process that creates branches like
update/foo
for updates to theDockerfile
I maintain in directoryfoo
, and it means that at some point I hadupdate/prometheus
and now haveupdate/prometheus/blackbox-exporter
, but Git doesn't allow those two to co-exist simultaneously because one implies the other must be a directory instead a file because of implementation details in Git itself -- the build that's failing wasn't touching either of those branches, however, so it's honestly surprising for them to be fetched at all, let alone break the build :smile: