Rework checkout mechanism again, again

[capnfabs]

Ok, this time I think I've finally got something pretty performant with minimal hacks.

Here's a series of bash commands I used to do this on a repo with nested submodules (https://github.com/capnfabs/grouse/blob/master/test-fixtures/nested-submodules-missing-submodules.zip)

~# git clone /app/nested-submodules-missing-submodules clone
Cloning into 'clone'...
done.
~# cd clone/
~/clone# git submodule init themes/bare/
Submodule 'themes/bare' (https://github.com/orf/bare-hugo-theme.git) registered for path 'themes/bare'
~/clone# export REPO=/app/nested-submodules-missing-submodules
~/clone# git -c submodule."themes/bare".url=$REPO/themes/bare submodule update themes/bare
Cloning into 'themes/bare'...
done.
Submodule path 'themes/bare': checked out '6f2f95a1ce89a6e642491072e848eba0b23fac2d'
~/clone# cat .git/modules/themes/bare/config
[core]
    repositoryformatversion = 0
    filemode = true
    bare = false
    logallrefupdates = true
    worktree = ../../../../themes/bare
[remote "origin"]
    url = /app/nested-submodules-missing-submodules/themes/bare
    fetch = +refs/heads/*:refs/remotes/origin/*
[branch "master"]
    remote = origin
    merge = refs/heads/master
~/clone# git submodule update themes/bare
~/clone# # i feel like the remote is supposed to point at the original submodule URL, not this one
~/clone# cd themes/bare/
~/clone/themes/bare# git remote -v
origin  /app/nested-submodules-missing-submodules/themes/bare (fetch)
origin  /app/nested-submodules-missing-submodules/themes/bare (push)
~/clone/themes/bare# git remote remove origin
~/clone/themes/bare# git submodule init assets/css/
bulma/     site.sass
~/clone/themes/bare# git submodule init assets/css/bulma
Submodule 'assets/css/bulma' (https://github.com/jgthms/bulma.git) registered for path 'assets/css/bulma'
~/clone/themes/bare# git -c submodule."assets/css/bulma".url=$REPO/themes/bare/assets/css/bulma submodule update assets/css/bulma
fatal: repository '/app/nested-submodules-missing-submodules/themes/bare/assets/css/bulma' does not exist
fatal: clone of '/app/nested-submodules-missing-submodules/themes/bare/assets/css/bulma' into submodule path 'assets/css/bulma' failed
~/clone/themes/bare# echo $?
128
~/clone/themes/bare# git submodule update --depth 1 assets/css/bulma
Cloning into 'assets/css/bulma'...
remote: Enumerating objects: 898, done.
remote: Counting objects: 100% (898/898), done.
remote: Compressing objects: 100% (870/870), done.
remote: Total 898 (delta 58), reused 545 (delta 22), pack-reused 0
Receiving objects: 100% (898/898), 63.89 MiB | 269.00 KiB/s, done.
Resolving deltas: 100% (58/58), done.
Checking connectivity... done.
fatal: reference is not a tree: 617febbfbf5b95e827d20654a16748c66bb5d647
Unable to checkout '617febbfbf5b95e827d20654a16748c66bb5d647' in submodule path 'assets/css/bulma'
root@0778ade78b9c:~/clone/themes/bare# cd assets/css/bulma/
root@0778ade78b9c:~/clone/themes/bare/assets/css/bulma# git fetch --depth=1000000
remote: Enumerating objects: 14301, done.
remote: Counting objects: 100% (14301/14301), done.
remote: Compressing objects: 100% (3631/3631), done.
remote: Total 13879 (delta 10016), reused 13404 (delta 9542), pack-reused 0Receiving objects:  99% (1

Receiving objects: 100% (13879/13879), 16.74 MiB | 186.00 KiB/s, done.
Resolving deltas: 100% (10016/10016), completed with 273 local objects.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
From https://github.com/jgthms/bulma
 * [new tag]         0.0.1      -> 0.0.1
 * [new tag]         0.0.10     -> 0.0.10
 * [new tag]         0.0.11     -> 0.0.11
 * [new tag]         0.0.12     -> 0.0.12
 * [new tag]         0.0.13     -> 0.0.13
 * [new tag]         0.0.14     -> 0.0.14
 * [new tag]         0.0.15     -> 0.0.15
 * [new tag]         0.0.16     -> 0.0.16
 * [new tag]         0.0.17     -> 0.0.17
 * [new tag]         0.0.18     -> 0.0.18
 * [new tag]         0.0.19     -> 0.0.19
 * [new tag]         0.0.2      -> 0.0.2
 * [new tag]         0.0.20     -> 0.0.20
 * [new tag]         0.0.21     -> 0.0.21
 * [new tag]         0.0.22     -> 0.0.22
 * [new tag]         0.0.23     -> 0.0.23
 * [new tag]         0.0.24     -> 0.0.24
 * [new tag]         0.0.25     -> 0.0.25
 * [new tag]         0.0.26     -> 0.0.26
 * [new tag]         0.0.27     -> 0.0.27
 * [new tag]         0.0.28     -> 0.0.28
 * [new tag]         0.0.3      -> 0.0.3
 * [new tag]         0.0.4      -> 0.0.4
 * [new tag]         0.0.5      -> 0.0.5
 * [new tag]         0.0.7      -> 0.0.7
 * [new tag]         0.0.8      -> 0.0.8
 * [new tag]         0.1.0      -> 0.1.0
 * [new tag]         0.1.1      -> 0.1.1
 * [new tag]         0.1.2      -> 0.1.2
 * [new tag]         0.2.0      -> 0.2.0
 * [new tag]         0.2.1      -> 0.2.1
 * [new tag]         0.3.0      -> 0.3.0
 * [new tag]         0.3.1      -> 0.3.1
 * [new tag]         0.3.2      -> 0.3.2
 * [new tag]         0.4.0      -> 0.4.0
 * [new tag]         0.4.1      -> 0.4.1
 * [new tag]         0.4.2      -> 0.4.2
 * [new tag]         0.4.4      -> 0.4.4
 * [new tag]         0.5.0      -> 0.5.0
 * [new tag]         0.5.1      -> 0.5.1
 * [new tag]         0.5.2      -> 0.5.2
 * [new tag]         0.5.3      -> 0.5.3
 * [new tag]         0.6.0      -> 0.6.0
 * [new tag]         0.6.1      -> 0.6.1
 * [new tag]         0.6.2      -> 0.6.2
 * [new tag]         0.7.0      -> 0.7.0
 * [new tag]         0.7.1      -> 0.7.1
 * [new tag]         0.7.2      -> 0.7.2
 * [new tag]         0.7.3      -> 0.7.3
 * [new tag]         0.7.4      -> 0.7.4
 * [new tag]         0.7.5      -> 0.7.5
 * [new tag]         0.8.0      -> 0.8.0
root@0778ade78b9c:~/clone/themes/bare/assets/css/bulma# cd -
/root/clone/themes/bare
root@0778ade78b9c:~/clone/themes/bare# git submodule update assets/css/bulma
Submodule path 'assets/css/bulma': checked out '617febbfbf5b95e827d20654a16748c66bb5d647'

So basically, the idea is:

• Clone from the current working tree
• Init each submodule, attempt to clone from the filesystem
• Then, we should probably modify the remote such that 'origin' points at the real remote again
• If cloning from filesystem doesn't work, just use a regular git submodule update.

Something that I don't love about this is -- you can't use git submodule update --depth 1 for nested submodules, because under some circumstances, they point at commits (which you can't fetch directly from remotes). This is just a git problem in general though.

I tested this all on git 2.7.4, which is the current version running on Ubuntu LTS Xenial. I think it would be good to try on git 2.1.4 (Debian Jessie) as well -- but that's probably the oldest stable version we need to support by a long way.

Quick note to performance -- apparently if the cloned repo is on the same disk, then git uses hardlinks so this is mega fast (see Local Protocols in Git On the Server)

[capnfabs]

Ok, picking this up again. Rather than rely upon the hardlink behaviour, I'm going to do git clone --shared:

When the repository to clone is on the local machine, instead of using hard links, automatically setup .git/objects/info/alternates to share the objects with the source repository. The resulting repository starts out without any object of its own.

This works regardless of whether the temporary directory is on the same mount or not, which is a major advantage of relying upon hardlinks ✨

[capnfabs]

Wow, this is working really well and going much faster than I'd expected. I think having five months of noodling around in Rust code has taught me some pretty good habits, and a lot of the skills were transferable.

[capnfabs]

Still todo:

• Add the hack to swap back in the original remote on submodules after clone
• Fix the tests, make the test suite more comprehensive and also not reliant upon 180MB of remote repos 🙃

[capnfabs]

Just some logging improvements left!

[capnfabs]

Merged.