Git push has some rejections

[kazoompa]

Hi,

I recently used BFG to remove some old files and change a few texts in my test files. The clean operations went all OK without a problem. When I tried to push the changes I had the following error message at the end:

{code} error: failed to push some refs to '' {code}

These are some of the rejected messages:

{code} ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hidden ref) ! [remote rejected] refs/pull/1/merge -> refs/pull/1/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hidden ref) ! [remote rejected] refs/pull/10/merge -> refs/pull/10/merge (deny updating a hidden ref) ! [remote rejected] refs/pull/11/head -> refs/pull/11/head (deny updating a hidden ref) {code}

Am I in trouble? Did I just mess up my repository? Is there anyway to correct this error?

I would really appreciate any help, thanks.

[rtyley]

You're not in trouble! However this is something I could cover with better documentation - issue #16 relates to this. The refs beginning 'refs/pull' are synthetic read-only refs created by GitHub - you can't update (and therefore 'clean') them, because they reflect branches that may well actually come from other repositories - ones that submitted pull-requests to you.

So, while you've pushed all your real refs, the pull requests don't get updated. There's no real way to fix them with your existing repo with the GitHub admin tools. The severity of an issue depends on whether you were trying to remove any 'private' data:

• Private data : the pull requests still have that private data in their history - they will remain as a way for other people to retrieve that data. The only ways to recover from this are to a) delete the repo using the GitHub admin tools, and then recreate it, or b) contact GitHub support and ask them to delete the PRs for you (they're pretty good about it)
• Nothing Private : the pull requests still reference your old history - if any of them are still open, you don't want to merge them into your new fresh history! Get the submitters to rebase them on your new fresh history before merging.

[isomorphisms]

So if you get this error does that mean BFG has fixed the problem, or not?

[algarecu]

Hey there, could this be possibly related to the fact that BFG has duplicated most of my commit history after deleting a file from it? I have the bfg-report for reference.

[rtyley]

Hey there, could this be possibly related to the fact that BFG has duplicated most of my commit history after deleting a file from it? I have the bfg-report for reference.

In order to change Git history (ie to change a repo to make it look like that file was never added) brand new commits have to be created. This is how Git works - every commit-id reflects a hash of it's history - change the history, change the commit id. To delete a file, you have to create new 'clean' commit history, and throw away the old 'dirty' history.

So yes, your commit history has been duplicated, but that's essential. And this can be awkward if you have old pull requests lying around, pointing to the old history.

[takecare]

does this mean that bfg can't "clean" your history on any github repository that has had pull requests "touching" any file you're trying to remove?

what i mean is, you look at the history (on github) and you can see that the history has been re-written and the file(s) you're cleaning out cannot be seen on the commits but if you go through the pull request history or if you still have the hash for the commit the file was introduced in (as an example), you can still see it (and its contents). unless i'm wrong.

so in order for someone to remove a file for sure they'd have to:

1. run bfg, re-writing history
2. push (get this rejected error but history has been re-written)
3. ask github staff to delete all the PRs involving this file (?)
4. what about the commits?

i have a public repo in which (i believe) this issue is easily seen, in case i didn't explain myself appropriately.

[jochenwezel]

I just try to run bfg without deleting any files, without any open push requests, and it fails, too:

This is what I did:

D:\temp-cleanup-cwm>git clone --mirror https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
Cloning into bare repository 'cammIntegrationPortal.git'...
remote: Counting objects: 5979, done.
remote: Compressing objects: 100% (3/3), done.
remote: Total 5979 (delta 1), reused 0 (delta 0), pack-reused 5976 eceiving objects: 100% (5979/5979), 14.99 MiB | 2.27
Receiving objects: 100% (5979/5979), 15.78 MiB | 2.27 MiB/s, done.

Resolving deltas: 100% (3801/3801), done.
Checking connectivity... done.

D:\temp-cleanup-cwm>cd temp-cleanup-cwm
Das System kann den angegebenen Pfad nicht finden.

D:\temp-cleanup-cwm>cd cammIntegrationPortal.git

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 5979, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5871/5871), done.
Writing objects: 100% (5979/5979), done.
Total 5979 (delta 3984), reused 1806 (delta 0)

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git push
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/CompuMasterGmbH/cammIntegrationPortal.git'

D:\temp-cleanup-cwm\cammIntegrationPortal.git>

After that I tried to follow the suggestions at http://stackoverflow.com/questions/34265266/remote-rejected-errors-after-mirroring-a-git-repository and did following:

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git config -e

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git remote update
Fetching origin

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git push
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/CompuMasterGmbH/cammIntegrationPortal.git'

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git reflog expire --expire=now --all && git gc --prune=now --aggressive
Counting objects: 5979, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (5871/5871), done.
Writing objects: 100% (5979/5979), done.
Total 5979 (delta 3984), reused 1992 (delta 0)

D:\temp-cleanup-cwm\cammIntegrationPortal.git>git push
Total 0 (delta 0), reused 0 (delta 0)
To https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
 ! [remote rejected] refs/pull/2/head -> refs/pull/2/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/3/head -> refs/pull/3/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/6/head -> refs/pull/6/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/7/head -> refs/pull/7/head (deny updating a hidden ref)
 ! [remote rejected] refs/pull/8/head -> refs/pull/8/head (deny updating a hidden ref)
error: failed to push some refs to 'https://github.com/CompuMasterGmbH/cammIntegrationPortal.git'

D:\temp-cleanup-cwm\cammIntegrationPortal.git>

with the following config with updated fetches:

[core]
        repositoryformatversion = 0
        filemode = false
        bare = true
        symlinks = false
        ignorecase = true
        hideDotFiles = dotGitOnly
[remote "origin"]
        url = https://github.com/CompuMasterGmbH/cammIntegrationPortal.git
        fetch = +refs/heads/*:refs/heads/*
        fetch = +refs/tags/*:refs/tags/*
        fetch = +refs/change/*:refs/change/*
        mirror = true

But still without effect (see output above) :-(

The repository in this case is:

No (open) push requests, only 1 additional branch - so it should be very simply, isn't it?

[javabrett]

No (open) push requests, only 1 additional branch - so it should be very simply, isn't it?

I think you mean Pull Requests. You are distinguishing open PRs from closed ones, however GitHub is retaining a (read-only) ref for each PR, regardless of whether it is open or closed. This allows you to later view the PR and the diffs it represents, even if that history would have been otherwise lost/garbage-collected if it were not merged into an ongoing branch.

Since you originally followed the BFG documentation and used --mirror, your config would have had both mirror = true and a complete-wildcard fetch refspec of fetch = +refs/*:refs/*. You have since amended to fetch config, but the mirror config remains. This means your git push will behave like it has the --mirror switch added. Assuming you had already fetched and rewritten all your refs/pull refs, this will account for the read-only error.

How you deal with this depends on what you are using BFG for and what you want to achieve. If you want sensitive or large files completely removed from the GitHub repo, then you need to contact GitHub support according to https://help.github.com/articles/remove-sensitive-data/ (emphasis mine):

This article tells you how to make commits with sensitive data unreachable from any branches or tags in your GitHub repository. However, it's important to note that those commits may still be accessible in any clones or forks of your repository, directly via their SHA-1 hashes in cached views on GitHub, and through any pull requests that reference them. You can't do anything about existing clones or forks of your repository, but you can permanently remove all of your repository's cached views and pull requests on GitHub by contacting GitHub Support.

In other words, GitHub Support might be able to provide further info on how you can discard the old history referenced by PRs.

If you just want bulk removed from what default (non-mirror) clones see, then you don't have to worry about these, as they aren't fetched by default, but be aware that the size of the repo on GitHub will actually increase slightly as a result.

[markkolich]

Since this is a top search result for "deny updating a hidden ref", I figured I'd leave this here to help out the next person...

If you've used BFG to clean up a repo, and you're trying to force push back to GitHub, make sure you're not trying to force push back to a protected branch. I wasted several hours staring at [remote rejected] ... (deny updating a hidden ref) thinking the discussion in this issue was the cause of my grief, only to realize it's a total red herring.

Buried deep in the deny updating a hidden ref noise, was a "protected branch hook declined" failure right in my terminal.

If you're in the same boat, disable/delete branch protection, and try another git push -f --mirror:

[mark (master)]~/workspace/foobar.mirror$ git push -f --mirror
...
remote: Resolving deltas: 100% (34037/34037), done.
To https://github.com/foo/bar.git
 + a5fb13391...9b66d3db1 master -> master (forced update)    <<<---!!!!!!
 + 0978357d5...f987b9bd2 release -> release (forced update)    <<<---!!!!!!
...
 ! [remote rejected]     refs/pull/100/head -> refs/pull/100/head (deny updating a hidden ref)

Note the familiar (forced update) from Git indicating the git push -f --mirror worked fine. Subsequent clones of this repo after the force push also confirmed success.

TL;DR — I could be totally mistaken, but the [remote rejected] ... (deny updating a hidden ref) complaints appear to be just warnings (for the reasons covered earlier in this thread) and do not seem to impact the ultimate success of a force push.

@rtyley perhaps something to clarify in future documentation?

[antoniocascais]

We faced the same problem described in this issue. We had an existent PR in our Repo, so we assume that was the root cause of the push rejection.

Here's what we did (let's say that our repo was named awesome-tool):

1) Created a new empty repo with name awesome-tool-tmp; 2) In our cloned repo, we changed the URL with git set-url to the url of the new awesome-tool-tmp repo; 3) git push (at this point, both awesome-tool and awesome-tool-tmp have the same content and commits in master; 4) used the tool as described here https://rtyley.github.io/bfg-repo-cleaner/. git push now worked (because there was no PR in the tmp repo, we assume) 5) renamed awesome-tool to awesome-tool-tmp2 6) renamed awesome-tool-tmp to awesome-tool 7) deleted awesome-tool-tmp2 (the repo with the passwords) 8) asked the ppl to delete their folders and clone the repo again

And we were able to successfully remove passwords from git history.

Might not be the best solution for everyone, but in our case was enough.

[rodrigolece]

I think that I accidentally found a solution (I'm not sure I can explain why it works). Same thing, I had PRs (closed) on GitHub and git push was being rejected.

1. I cloned the local copy of my repo (using git clone --mirror but providing the path to my local directory).
2. I cleaned this copy using BFG.
3. git remote was showing origin as pointing to my local repo, so I removed origin and added it but this time pointing to my repo hosted on GitHub.
4. From the clean mirror copy, I forced a push using git push -f --set-upstream origin master.

As far as I can tell, my repo in GitHub is now clean. Git pulling from my original repo doesn't work, but now I can easily delete that folder and clone a new copy.

Any ideas for/against doing it this way?

[calebTree]

Mirror cloning from a local copy path apparently doesn't include pull request refs. This can be verified using git ls-remote. BFG still doesn't apparently seem to be built to clean pull requests at all. Here is a tl;dr blog I found with some more details https://www.secjuice.com/github-complete-cleaning-sensitive-secrets/.

[avatar-lavventura]

@rodrigolece

3. I removed origin

How did you removed origin? (git remote rm origin)? and how did you added it but this time pointing to your repo hosted on GitHub?

[rodrigolece]

@avatar-lavventura that's exactly right, git remote remove origin followed by git remote add origin <URL>

[je-hal]

The error deny updating a hidden ref also came up for me, although the deleted file wasn't part of any PR.

What I did to test bfg:

1. I created a new file and pushed it to my GitHub repo.
2. I deleted the new file and pushed the commit to my GitHub repo.
3. I followed the bfg documentation to delete the file. The error above came up multiple times.

Result:

• The commits (steps 1 and 2) were rewritten, they don't contain any changes, but are still in the history with their commit message.
• bfg reported that 5 object IDs were changed.

My explanation: For some reason, bfg not only had to change the commit IDs related to the first two steps, but also other ones (why?). The commit IDs of the other ones seem to contain file changes of files that are in closed PRs, but that doesn't bother me, because I only need to remove a file that wasn't part of a PR.

Just wanted to add this: I don't believe you always need to delete PRs for a total clean-up. Sometimes, this message comes up as a side-effect of rewriting depending commits that are part of PRs.

[ykla]

I think that I accidentally found a solution (I'm not sure I can explain why it works). Same thing, I had PRs (closed) on GitHub and git push was being rejected.

1. I cloned the local copy of my repo (using git clone --mirror but providing the path to my local directory).
2. I cleaned this copy using BFG.
3. git remote was showing origin as pointing to my local repo, so I removed origin and added it but this time pointing to my repo hosted on GitHub.
4. From the clean mirror copy, I forced a push using git push -f --set-upstream origin master.

As far as I can tell, my repo in GitHub is now clean. Git pulling from my original repo doesn't work, but now I can easily delete that folder and clone a new copy.

Any ideas for/against doing it this way?

If you have any merged branches, it just looks like the push worked, which is the same as using git clone --bare. But if you push and then clone back, you'll see that nothing has changed.

C:\Users\ykla\Desktop\SEP-CN\test>git clone --mirror  https://github.com/taophilosophy/SEP-CN.git C:\Users\ykla\Desktop\SEP-CN/test
C:\Users\ykla\Desktop\SEP-CN\test>java -jar C:\Users\ykla\Desktop\SEP-CN\bfg.jar --strip-blobs-bigger-than 2M test

Using repo : C:\Users\ykla\Desktop\SEP-CN\test

Scanning packfile for large blobs: 36652
Scanning packfile for large blobs completed in 225 ms.
Found 4 blob ids for large blobs - biggest=10114104 smallest=9900680
Total size (unpacked)=39948080
Found 2635 objects to protect
Found 23 commit-pointing refs : HEAD, refs/heads/main, refs/pull/1/head, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 167e5a0d (protected by 'HEAD')

Cleaning
--------

Found 3070 commits
Cleaning commits:       100% (3070/3070)
Cleaning commits completed in 445 ms.

Updating 6 Refs
---------------

        Ref                Before     After
        --------------------------------------
        refs/pull/1/head | 893e91f0 | cb5430e5
        refs/pull/2/head | e6d30510 | 640191ab
        refs/pull/3/head | bb15af62 | 51ee9aab
        refs/pull/4/head | 3cf332d1 | f3f0f301
        refs/pull/5/head | 5afaa66c | 1fd7ed94
        refs/pull/7/head | 0f682ae9 | 90b45a77

Updating references:    100% (6/6)
...Ref update completed in 26 ms.

Commit Tree-Dirt History
------------------------

        Earliest                                              Latest
        |                                                          |
        ..............................................DD....D.......

        D = dirty commits (file tree fixed)
        m = modified commits (commit message or parents changed)
        . = clean commits (no changes to file tree)

                                Before     After
        -------------------------------------------
        First modified commit | 893e91f0 | cb5430e5
        Last dirty commit     | 0f682ae9 | 90b45a77

Deleted files
-------------

        Filename                               Git id
        ---------------------------------------------------------------------------
        autocorrect-node.linux-x64-gnu.node  | 61db0d39 (9.6 MB), 4f04acea (9.4 MB)
        autocorrect-node.linux-x64-musl.node | c5bf586b (9.4 MB), aef00e88 (9.6 MB)

In total, 18 object ids were changed. Full details are logged here:

        C:\Users\ykla\Desktop\SEP-CN\test.bfg-report\2024-05-24\22-40-42

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive
C:\Users\ykla\Desktop\SEP-CN\test>git reflog expire --expire=now --all && git gc --prune=now --aggressive
C:\Users\ykla\Desktop\SEP-CN\test>git remote
origin
C:\Users\ykla\Desktop\SEP-CN\test>git remote remove origin
Note: A branch outside the refs/remotes/ hierarchy was not removed;
to delete it, use:
  git branch -d main
C:\Users\ykla\Desktop\SEP-CN\test>git remote add origin https://github.com/taophilosophy/SEP-CN.git

C:\Users\ykla\Desktop\SEP-CN\test>git push -f --set-upstream origin main
branch 'main' set up to track 'origin/main'.
Everything up-to-date

But if you delete the directory test and clone it again, you'll see that everything is back to the way it was. This only changes locally.

It doesn't look like there's any difference between this command and git clone --bare, but neither works.

C:\Users\ykla\Desktop\SEP-CN>java -jar C:\Users\ykla\Desktop\SEP-CN\bfg.jar --strip-blobs-bigger-than 2M test

Using repo : C:\Users\ykla\Desktop\SEP-CN\test

Scanning packfile for large blobs: 36652
Scanning packfile for large blobs completed in 222 ms.
Found 4 blob ids for large blobs - biggest=10114104 smallest=9900680
Total size (unpacked)=39948080
Found 2635 objects to protect
Found 23 commit-pointing refs : HEAD, refs/heads/main, refs/pull/1/head, ...

Protected commits
-----------------

These are your protected commits, and so their contents will NOT be altered:

 * commit 167e5a0d (protected by 'HEAD')

Cleaning
--------

Found 3070 commits
Cleaning commits:       100% (3070/3070)
Cleaning commits completed in 460 ms.

Updating 6 Refs
---------------

        Ref                Before     After
        --------------------------------------
        refs/pull/1/head | 893e91f0 | cb5430e5
        refs/pull/2/head | e6d30510 | 640191ab
        refs/pull/3/head | bb15af62 | 51ee9aab
        refs/pull/4/head | 3cf332d1 | f3f0f301
        refs/pull/5/head | 5afaa66c | 1fd7ed94
        refs/pull/7/head | 0f682ae9 | 90b45a77

Updating references:    100% (6/6)
...Ref update completed in 25 ms.

Commit Tree-Dirt History
------------------------

        Earliest                                              Latest
        |                                                          |
        ..............................................DD....D.......

        D = dirty commits (file tree fixed)
        m = modified commits (commit message or parents changed)
        . = clean commits (no changes to file tree)

                                Before     After
        -------------------------------------------
        First modified commit | 893e91f0 | cb5430e5
        Last dirty commit     | 0f682ae9 | 90b45a77

Deleted files
-------------

        Filename                               Git id
        ---------------------------------------------------------------------------
        autocorrect-node.linux-x64-gnu.node  | 61db0d39 (9.6 MB), 4f04acea (9.4 MB)
        autocorrect-node.linux-x64-musl.node | c5bf586b (9.4 MB), aef00e88 (9.6 MB)

In total, 18 object ids were changed. Full details are logged here:

        C:\Users\ykla\Desktop\SEP-CN\test.bfg-report\2024-05-24\23-06-23

BFG run is complete! When ready, run: git reflog expire --expire=now --all && git gc --prune=now --aggressive