Git Error when cloning OF repo

[armadillu]

I get an error trying to clone OF.

$git clone https://github.com/openframeworks/openFrameworks.git --verbose
Cloning into 'openFrameworks'...
POST git-upload-pack (gzip 1391 to 749 bytes)
remote: Counting objects: 134186, done.
error: object bb1f763543cc01b7f5904fcfb636888c3ad73def: zeroPaddedFilemode: contains zero-padded file modes
fatal: Error in object

$git --version
git version 2.12.2

I think it is due to me updating my .gitconfig recently with:

[Transfer]
fsckobjects = true

Not sure how serious this is, I couldn't find much on it... But I thought I should report just in case.

[bakercp]

This is discussed here:

https://github.com/github/hub/issues/1404

From their analysis, it is an issue with our repository and requires a rewrite of the history.

Perhaps this can be closed when / if we remove all the old binary libs from the repo with something like bfg. Workaround is to make sure you don't use fsckobjects.

[bakercp]

For reference here are related issues:

1805 #4804

[danoli3]

Just chiming in here... I literally have not had stable internet for 8 months and on numerous attempts have failed to checkout the repo (behind corporate firewalls, 4g connections, public wifi, impossible). This is not standard for a GitHub repo and I have no real issues with any others.

Observing a full checkout on a stable 20mbit/ internet connection just now it took me 7 hours. (It wasn't just download, it was creating the whole git structure for numerous binary releases (1.4gb of garbage)).

Issues that have been holding this back:

• It breaks current PR requests. (unless manually cherry picked).
• Forks that are not reset to the new HEAD will be left behind (notify oF developer mail to mitigate active developers continuing down the wrong branch)
• Git LFS is not an option as it does not support forks / public repos.

Now one has to think of the developer the checkout problems for those other than the few who are actively already checked this repo source out in order to contribute.

I've shown oF to people in the past; before my own issues, and I've watched them try and check out the repo and give up after an hour of waiting and then go on to download the releases from the oF site. They use it, make some changes, but these developers never try and contribute because they can't check it out, can't get involved ... This is costing oF a lot of potential developer time it will never see... to only hold onto the past?

How can we fix this?

One of the core must do this. I'd suggest doing a cold run on a fork to test first like I did here: See https://github.com/openframeworks/openFrameworks/issues/1805#issuecomment-165859086

Command to run with BFG (everything I did in 2015 was detailed in #1805 ):

git clone --mirror https://github.com/danoli3/openFrameworks.git
git clone openFrameworks.git oFBFG
cd oFBFG
## Remove binaries from Master / Head for complete history removal of binaries
 find . -name "*.a" -exec rm -rf {} \;
 find . -name "*.so" -exec rm -rf {} \;
 find . -name "*.dll" -exec rm -rf {} \;
 find . -name "*.app" -exec rm -rf {} \;
 find . -name "*.exe" -exec rm -rf {} \;
 find . -name "*.lib" -exec rm -rf {} \;
# add all binary types to remove
git commit -m "Removal of Binaries"
cd ../
# run BFG to remove binaries from all history!

java -jar bfg.jar --delete-files '*.{a,exe,lib,so,dll,app}' openFrameworks.git

cd /Users/danoli3/Documents/bfg/openFrameworks.git
git reflog expire --expire=now --all && git gc --prune=now --aggressive
cd ../

# done now just git push -f
#  git push -f

[bilderbuchi]

Just as a mitigation strategy for people downloading oF: a shallow clone of master will not be too big, and iirc you can submit PRS or merge even from shallow clones with a reasonably new git version.

[danoli3]

While a shallow clone will work in the initial checkout using:

git clone --depth 1 https://github.com/openFrameworks/openFrameworks.git openFrameworks

This requires the developer to have advanced knowledge of git in using a GUI Git Program, not natively used in GitHub Application and or require them to use bash/terminal.

Adding to that, once a shallow clone is complete, any pulls from any branch living this bloated binary tree will from any GUI Application, SourceTree, GitHub, TortoiseGit will fill up the cache with the whole history including the binaries regardless and same problem applies.

[danoli3]

Also you literally cannot push a commit from a shallow clone... so Developers who shallow depth clone cannot Pull Request. This is massive.

To https://github.com/danoli3/openFrameworks.git
 ! [remote rejected] replace-wget-with-curl -> replace-wget-with-curl (shallow update not allowed)

For Reference: The Command to Unshallow depth the repo is:

 git fetch origin --unshallow 

[bilderbuchi]

Also you literally cannot push a commit from a shallow clone... so Developers who shallow depth clone cannot Pull Request. This is massive.

Sorry but this is not true. What is your git version? I just tried this successfully, with a "typical" workflow of creating a feature based on current master:

bilderbuchi@osprey:~$ mkdir temp
bilderbuchi@osprey:~$ cd temp
bilderbuchi@osprey:~/temp$ git clone --depth 1 git@github.com:bilderbuchi/ofxStiwiGui.git
Cloning into 'ofxStiwiGui'...
remote: Counting objects: 61, done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 61 (delta 27), reused 45 (delta 20), pack-reused 0
Receiving objects: 100% (61/61), 167.95 KiB | 0 bytes/s, done.
Resolving deltas: 100% (27/27), done.
Checking connectivity... done.
bilderbuchi@osprey:~/temp$ cd ofxStiwiGui/
bilderbuchi@osprey:~/temp/ofxStiwiGui$ git checkout -b feature-branch
Switched to a new branch 'feature-branch'
bilderbuchi@osprey:~/temp/ofxStiwiGui$ touch bla.txt
bilderbuchi@osprey:~/temp/ofxStiwiGui$ git add bla.txt 
bilderbuchi@osprey:~/temp/ofxStiwiGui$ git commit -m "Some commit."
[feature-branch 9d992de] Some commit.
 1 file changed, 0 insertions(+), 0 deletions(-)
 create mode 100644 bla.txt
bilderbuchi@osprey:~/temp/ofxStiwiGui$ git status
On branch feature-branch
nothing to commit, working directory clean
bilderbuchi@osprey:~/temp/ofxStiwiGui$ git push origin feature-branch 
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (2/2), done.
Writing objects: 100% (3/3), 273 bytes | 0 bytes/s, done.
Total 3 (delta 1), reused 0 (delta 0)
remote: Resolving deltas: 100% (1/1), completed with 1 local object.
To git@github.com:bilderbuchi/ofxStiwiGui.git
 * [new branch]      feature-branch -> feature-branch
bilderbuchi@osprey:~/temp/ofxStiwiGui$ git --version
git version 2.7.4

[arturoc]

@danoli3 as @bilderbuchi says the recommended way to clone OF now if you don't have a very good connection is to use a shallow clone, latest git versions should allow to push without problem from such a clone.

We removed all binaries already so a shallow clone should be really small. Ideally we would remove them from the whole history but that would break every PR which would be really problematic.

Not sure there's a solution for this that won't break PRs or any other commit references. Perhaps in some time when all valid PRs reference to a commit after we removed all binaries there might be another solution?

Or perhaps we could create a github bot that takes a list of old hash new hash correspondences an updates every reference in PRs and issues

[bilderbuchi]

It's not only about open PRs, if you rewrite the public history also every one of the (currently) 1911 forks on Github, and all those on users' harddisks or wherever on a computer lose the ability to sync/pull/push from the OF root repo, and therefore need to be force-updated.

[danoli3]

It’s 911 more forks now than it was when we tried to fix this 2 years ago and it’s only going to get worse.

And seriously @bilderbuchi, I know you are also invested in this and still hope for a better less hash changing option, but as far as we can tell, we just need to rip the bandaid off sooner than later

What we suggest to do is GitHubs own recommendation for this situation. https://help.github.com/articles/what-is-my-disk-quota/

I don’t know what else to say. It needs to be done. It’s massive and requires so many processes and safeguard in place yes, but that’s what Apothecary and all the build scripts and all the other work was leading to... to removig the binaries from the repo.

So stage one was completed and so much better as well with automated releases as everything with built binaries from a controlled safe bot.

I’m all for a PR bot if that’s possible. That would definitely make this all easier!

I’m yet to be back at my build machine to test the shallow clone test case. I’ll get back to you with the results tomorrow.

On Mon, 9 Oct 2017 at 11:50 pm, Christoph Buchner notifications@github.com wrote:

It's not only about open PRs, if you rewrite the public history also every one of the (currently) 1911 forks on Github, and all those on users' harddisks or wherever on a computer lose the ability to sync/pull/push from the OF root repo, and therefore need to be force-updated.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/openframeworks/openFrameworks/issues/5562#issuecomment-335148423, or mute the thread https://github.com/notifications/unsubscribe-auth/AAytHL4pzm3aoH3yeru8uXNIfVLaxnw2ks5sqhazgaJpZM4NDbqt .

[bilderbuchi]

I am sure that if you volunteer as the main contact point for when people's forks/PRS/workflows break after doing this, @arturoc will be very glad.

This problem just has no "nice" solution, which is why I always try to educate git learners to be very strict about what gets into a repo.

Also, you misread the situation - I have not been particularly invested in this since I withdrew from OF work quite long ago now. I can't avoid wearing my "git hat", though, and try to check and correct overly broad statements like "shallow repos can't push/PR". ;-)

[danoli3]

So it seems a shallow clone will not work for anything except for master.

Master seems to resolve just fine submitting PR's from a shallow clone, but as soon as you deal with another branch, especially one that branches off before the shallow clone depth, you get rejected if you do manage to check out that branch using bash. Super weird git magic

[bilderbuchi]

Well, it's not a surprise that you can't pr if the branching point is before the shallow clone depth - if git does not have the whole relevant history, how should it compute the changes? That would be magic, not the observed behaviour. Considering that the policy for PRS is (or at least was when I left) to branch off of current master, I don't see a big impact on typical use cases, though. In any case, you should be able to solve your problems by specifying the branch you want to target, when cloning the repo - by default git will clone with the master branch.