How to deal with fatal: bad object HEAD in git

swyxio commented 1 year ago

category: note slug: solve-git-bad-object-head

as a cheapo who uses Apple iCloud (the world's worst sync service) as a sync service for my Obsidian Second Brain, I have recently run into this issue a lot:

$ git status 
fatal: bad object HEAD
$ git log
fatal: bad object refs/heads/main

The top Stackoverflow answer is 10 yrs old and tells you to rm -rf .git, which ofc is not what you want to do it "right".

I've had to fix this a few times, and only recently worked it out with ChatGPT. Saving here for posterity:

$ git fetch origin
# THIS USUALLY DOES IT FOR ME - then i can commit and merge and push normally

# hopefully this is good enough for git status to start working again
$ git pull

when it is more fucked than normal

# confirm fuckedness

$ git fsck
bad sha1 file: .git/objects/00/084ad8182051c580009e8ccc22e224d97a4823 2
bad sha1 file: .git/objects/02/8027ce244a5390478159b510bd6767d648e31f 2
bad sha1 file: .git/objects/03/8c235cd6cc3e93596b25d7fb0ab14d9a999ec7 2
bad sha1 file: .git/objects/05/72e5490252d8fd25230669000747a7032a14df 2
bad sha1 file: .git/objects/07/ecafe1981e587022b562562ce02fd0adc45beb 2
bad sha1 file: .git/objects/09/e5b5f51f4f9683155f7a81d4fad1321e4085ee 2
bad sha1 file: .git/objects/0a/47019b78b8fd69a1b1a2d96b2be442fbceaf49 2
bad sha1 file: .git/objects/0c/b9400cec84abac87519225f7e5ceca987e7f66 2
bad sha1 file: .git/objects/10/2a21b02d2f08c777bfe0066de28afe0df3290c 2
bad sha1 file: .git/objects/14/e8edad8d4898bc04c4a00ce0df268cf359df63 2
bad sha1 file: .git/objects/14/65567afb6ee576ceb3e3061ffdf9c4ccd47095 2
bad sha1 file: .git/objects/1a/ac8cf84058d474b8845fe4ff410a2a1bc5a25a 2
bad sha1 file: .git/objects/1b/5cf76bb3a880fcf70083fda0aa4d0faccb5864 2

# etc etc etc

If you see a lot of "2"'s, thats icloud fucking you up. solve with

#  remove all files in .git with a space and then a number in their name, likely duplicates created by icloud
find .git -type f -name '* [0-9]*' -delete

Then you can git fetch origin again.

If you have

$ git log
fatal: bad object refs/heads/main

then run

git update-ref -d refs/heads/main

misc notes

# NOTE: THIS WILL DESTROY ANY UNCOMMITTED/UNSAVED CHANGES - take care before running it
$ git reset --hard origin/main

This may show you some bad objects:

remote: Enumerating objects: 3899, done.
remote: Counting objects: 100% (888/888), done.
remote: Compressing objects: 100% (364/364), done.
remote: Total 3899 (delta 554), reused 597 (delta 523), pack-reused 3011
Receiving objects: 100% (3899/3899), 1.32 MiB | 9.92 MiB/s, done.
error: corrupt loose object 'dc3711cb3ba71eefe40c8e49cc9c9fb311d29553'
fatal: loose object dc3711cb3ba71eefe40c8e49cc9c9fb311d29553 (stored in .git/objects/dc/3711cb3ba71eefe40c8e49cc9c9fb311d29553) is corrupt
fatal: fetch-pack: invalid index-pack output
fatal: Could not parse object 'origin/main'.

So the fix is this:

rm -f .git/objects/dc/3711cb3ba71eefe40c8e49cc9c9fb311d29553
git fetch origin

(July 31 update) corrupt loose object

I also had this issue today:

$ git fsck

error: corrupt loose object '25726bffc0e97532098679d0ed04095710452d29'
error: unable to unpack contents of .git/objects/25/726bffc0e97532098679d0ed04095710452d29
error: 25726bffc0e97532098679d0ed04095710452d29: object corrupt or missing: .git/objects/25/726bffc0e97532098679d0ed04095710452d29
error: corrupt loose object '5b4e2ac94c33715545674545e501a705ea20924b'
error: unable to unpack contents of .git/objects/5b/4e2ac94c33715545674545e501a705ea20924b
error: 5b4e2ac94c33715545674545e501a705ea20924b: object corrupt or missing: .git/objects/5b/4e2ac94c33715545674545e501a705ea20924b
error: corrupt loose object 'ad7ceee5b187af409cbd2219bc4b578faf425471'
error: unable to unpack contents of .git/objects/ad/7ceee5b187af409cbd2219bc4b578faf425471
error: ad7ceee5b187af409cbd2219bc4b578faf425471: object corrupt or missing: .git/objects/ad/7ceee5b187af409cbd2219bc4b578faf425471
error: corrupt loose object 'fdacf1b391398e4548fa3a9f09b0f6714881907c'
error: unable to unpack contents of .git/objects/fd/acf1b391398e4548fa3a9f09b0f6714881907c
error: fdacf1b391398e4548fa3a9f09b0f6714881907c: object corrupt or missing: .git/objects/fd/acf1b391398e4548fa3a9f09b0f6714881907c
Checking object directories: 100% (256/256), done.
Checking objects: 100% (27396/27396), done.
missing blob 60e0633917e44a958f890039cff7297f99d8e8eb
dangling commit d885e9e214a1f84ce5feb8fdcdf51a27249e689e
dangling commit a38bc134bc0ec9741e7abbfa8ee200a0c82253d6
missing blob fdacf1b391398e4548fa3a9f09b0f6714881907c
missing blob 5b4e2ac94c33715545674545e501a705ea20924b
missing blob 25726bffc0e97532098679d0ed04095710452d29
missing blob f2f68aa9deda03f256574d468395e840144d9d05
missing blob ad7ceee5b187af409cbd2219bc4b578faf425471

i got a lot of recommendations with exotic options but ultimately the fix was this

git reset <last good remote commit> # NOT hard!
git stash # to store stuff that has been changed since last good remote commit
git fetch origin # pull down everything since last commit
git stash pop # put back the new stuff
git add . && git commit -m "add back new stuff" # new stuff
git push origin main # done!

architchandra commented 1 year ago

Oh wow! I also recently wrote about the same issue with Obsidian on iCloud: https://architchandra.com/articles/a-side-effect-of-storing-a-git-repository-in-icloud-drive. 🙌🏽

Really like how you've tackled everything through the command line. I hope it would be okay if I try some of your learnings and incorporate them into my article later. Thanks!

I love your content, btw. You're an inspiration. :)

swyxio commented 1 year ago

you're welcome! thanks for the nice note!

robrecord commented 1 week ago

Thanks for the tips.

The ultimate solution to your particular issue to not store .git repositories in iCloud. They are designed to pushed and pulled to external remotes and that's the way to back them up/sync them. You don't need iCloud in this case, and neither of these things were designed to be used the way you're trying to use them.

The reason iCloud messes with git repositories is because they are highly specialized and operate in a different way a human would interact with files. File sync services don't expect this.

There might be a way to hold the repo outside the folder, outside of iCloud's view, using a git worktree, but when checking out different repo states this could cause syncing slowdowns and unneccesary syncs.

Just don't do it :-) you are asking for trouble. It's not iCloud, it's the same with Dropbox for example.

architchandra commented 1 week ago

@robrecord - You're absolutely right, but this is for when we still want to do it 😅. Storing Git repos in cloud-sync services should ideally be avoided, but it might still be the best bet for some people who don't want to pay extra for using Git features on mobile devices (eg. you need to install paid apps to do anything meaningful Git related stuff on iOS).

Just talking about Obsidian, I've been using iCloud Drive to keep notes in sync on my desktop and phone for quite a while now without much of a hassle, esp. since I removed the workspace.json file from version control.

robrecord commented 1 day ago

Can’t say fairer than that! :-)

If needed, I’m thinking you could, as mentioned, store the git repo outside the synced folders - happy to shed light if that’s of interest.

jonasfroeller commented 7 hours ago

@robrecord I personally store my code on both Synology Drive and GitHub because I like having an extra backup outside of a Git system. It’s also convenient to automatically sync secrets between devices.

swyxio / swyxdotio