search

xmake-io / xmake

🔥 A cross-platform build utility based on Lua
https://xmake.io
Apache License 2.0
9.87k stars 776 forks source link

Improve git clone #5507

Closed star-hengxing closed 1 month ago

star-hengxing commented 1 month ago

Is your feature request related to a problem? Please describe.

Before:

git clone https://github.com/EsotericSoftware/spine-runtimes.git -c core.longpaths=true -c core.fsmonitor=false source.tmp\spine-runtimes
Cloning into 'source.tmp\spine-runtimes'...
remote: Enumerating objects: 120521, done.
remote: Counting objects: 100% (8354/8354), done.
remote: Compressing objects: 100% (3149/3149), done.
remote: Total 120521 (delta 5148), reused 7800 (delta 4715), pack-reused 112167 (from 1)
Receiving objects: 100% (120521/120521), 466.56 MiB | 16.85 MiB/s, done.
Resolving deltas: 100% (82459/82459), done.
Updating files: 100% (5541/5541), done.
git -c core.fsmonitor=false checkout d33c10f85634d01efbe4a3ab31dabaeaca41230c
Updating files: 100% (5871/5871), done.

Now:

Add --filter=tree:0 --no-checkout

git clone https://github.com/EsotericSoftware/spine-runtimes.git -c core.longpaths=true -c core.fsmonitor=false --filter=tree:0 --no-checkout source.tmp\spine-runtimes
Cloning into 'source.tmp\spine-runtimes'...
remote: Enumerating objects: 9495, done.
remote: Counting objects: 100% (580/580), done.
remote: Compressing objects: 100% (569/569), done.
remote: Total 9495 (delta 11), reused 571 (delta 11), pack-reused 8915 (from 1)
Receiving objects: 100% (9495/9495), 1.91 MiB | 7.91 MiB/s, done.
Resolving deltas: 100% (176/176), done.
git -c core.fsmonitor=false checkout d33c10f85634d01efbe4a3ab31dabaeaca41230c
remote: Enumerating objects: 703, done.
remote: Counting objects: 100% (507/507), done.
remote: Compressing objects: 100% (422/422), done.
remote: Total 703 (delta 26), reused 308 (delta 21), pack-reused 196 (from 1)
Receiving objects: 100% (703/703), 185.71 KiB | 2.00 MiB/s, done.
Resolving deltas: 100% (27/27), done.
remote: Enumerating objects: 496, done.
remote: Counting objects: 100% (183/183), done.
remote: Compressing objects: 100% (182/182), done.
remote: Total 496 (delta 3), reused 1 (delta 1), pack-reused 313 (from 1)
Receiving objects: 100% (496/496), 136.90 KiB | 1.63 MiB/s, done.
Resolving deltas: 100% (6/6), done.
remote: Enumerating objects: 3505, done.
remote: Counting objects: 100% (1488/1488), done.
remote: Compressing objects: 100% (1121/1121), done.
remote: Total 3505 (delta 908), reused 367 (delta 367), pack-reused 2017 (from 1)
Receiving objects: 100% (3505/3505), 51.29 MiB | 16.72 MiB/s, done.
Resolving deltas: 100% (1491/1491), done.
Updating files: 100% (3907/3907), done.

Describe the solution you'd like

https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone

Treeless clones are really only helpful for automated builds when you want to quickly clone, compile a project, then throw away the repository. In environments like GitHub Actions using public runners, you want to minimize your clone time so you can spend your machine time actually building your software! Treeless clones might be an excellent option for those environments.

Describe alternatives you've considered

No response

Additional context

No response

waruqi commented 1 month ago

这两参数的兼容性怎样，低版本 git 支持么

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

What is the compatibility of these two parameters? Is it supported by lower versions of git?

waruqi commented 1 month ago

另外，目前 clone 需要指定某个 branch，git clone --depth 1 -b branchname 这种支持么，depth 还需要指定么

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

In addition, currently clone needs to specify a certain branch. Is git clone --depth 1 -b branchname supported? Does depth still need to be specified?

star-hengxing commented 1 month ago

这两参数的兼容性怎样，低版本 git 支持么

暂时没找到哪个版本给的，但看 partial-clone 里面的时间，大概 2021 年就有了。

另外，目前 clone 需要指定某个 branch，git clone --depth 1 -b branchname 这种支持么，depth 还需要指定么

看 https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone ，这个就是 depth 1 加强版。

waruqi commented 1 month ago

快是快，但是 clone 下来没啥东西么。。

ruki-2:tmp ruki$ git clone --filter=tree:0 --no-checkout git@github.com:xmake-io/xmake.git 
Cloning into 'xmake'...
remote: Enumerating objects: 14502, done.
remote: Counting objects: 100% (370/370), done.
remote: Compressing objects: 100% (363/363), done.
remote: Total 14502 (delta 7), reused 330 (delta 7), pack-reused 14132 (from 1)
Receiving objects: 100% (14502/14502), 4.76 MiB | 704.00 KiB/s, done.
Resolving deltas: 100% (44/44), done.
ruki-2:tmp ruki$ cd xmake
ruki-2:xmake ruki$ ls
ruki-2:xmake ruki$ ls -a
.   ..  .git
SirLynix commented 1 month ago

that's because of the --no-checkout, not --filter=tree:0

star-hengxing commented 1 month ago

Useful for package that have git url(imgui docking/spine-runtimes/godotcpp4 etc...)

waruqi commented 1 month ago

https://github.blog/open-source/git/get-up-to-speed-with-partial-clone-and-shallow-clone

it works.

waruqi commented 1 month ago

On github.com and GitHub Enterprise Server 2.22+, there are two options available: Treeless clones: git clone --filter=tree:0

Does all git servers support it?

waruqi commented 1 month ago

https://github.com/xmake-io/xmake/pull/5509

waruqi commented 1 month ago

try it again.

waruqi commented 1 month ago

测试了下，似乎 depth 1 更快么。

depth 1

[ruki@5dd706d1f15e xmake-repo]$ time xmake update -s -v dev
checking for unzip ... /usr/sbin/unzip
checking for git ... /usr/sbin/git
checking for gzip ... /usr/sbin/gzip
checking for tar ... /usr/sbin/tar
checking for ping ... no
/usr/sbin/git ls-remote --refs https://github.com/xmake-io/xmake.git
update version dev from official source ..
prepared to download to temp dir /tmp/.xmake971/240826/xmakesrc/dev ..
  => downloading https://github.com/xmake-io/xmake.git .. /usr/sbin/git clone https://github.com/xmake-io/xmake.git -b dev --depth 1 -c core.fsmonitor=false /tmp/.xmake971/240826/xmakesrc/dev
Cloning into '/tmp/.xmake971/240826/xmakesrc/dev'...
remote: Enumerating objects: 4233, done.
remote: Counting objects: 100% (4233/4233), done.
remote: Compressing objects: 100% (3288/3288), done.
remote: Total 4233 (delta 1220), reused 2351 (delta 448), pack-reused 0 (from 0)
Receiving objects: 100% (4233/4233), 2.75 MiB | 2.62 MiB/s, done.
Resolving deltas: 100% (1220/1220), done.
  => download https://github.com/xmake-io/xmake.git .. ok
  => install script to /home/ruki/.local/share/xmake .. sh /home/ruki/.local/share/xmake/scripts/update-script.sh /home/ruki/.local/share/xmake /tmp/.xmake971/240826/xmakesrc/dev/xmake
ok

real    0m4.580s
user    0m0.587s
sys 0m0.806s

treeless

[ruki@5dd706d1f15e xmake-repo]$ time xmake update -s -v dev
checking for unzip ... /usr/sbin/unzip
checking for git ... /usr/sbin/git
checking for gzip ... /usr/sbin/gzip
checking for tar ... /usr/sbin/tar
checking for ping ... no
/usr/sbin/git ls-remote --refs https://github.com/xmake-io/xmake.git
update version dev from official source ..
prepared to download to temp dir /tmp/.xmake971/240826/xmakesrc/dev ..
  => downloading https://github.com/xmake-io/xmake.git .. /usr/sbin/git clone https://github.com/xmake-io/xmake.git -b dev --filter=tree:0 -c core.fsmonitor=false /tmp/.xmake971/240826/xmakesrc/dev
Cloning into '/tmp/.xmake971/240826/xmakesrc/dev'...
remote: Enumerating objects: 14506, done.
remote: Counting objects: 100% (374/374), done.
remote: Compressing objects: 100% (367/367), done.
remote: Total 14506 (delta 7), reused 333 (delta 7), pack-reused 14132 (from 1)
Receiving objects: 100% (14506/14506), 4.76 MiB | 3.02 MiB/s, done.
Resolving deltas: 100% (44/44), done.
remote: Enumerating objects: 1349, done.
remote: Counting objects: 100% (322/322), done.
remote: Compressing objects: 100% (288/288), done.
remote: Total 1349 (delta 7), reused 111 (delta 4), pack-reused 1027 (from 1)
Receiving objects: 100% (1349/1349), 160.39 KiB | 423.00 KiB/s, done.
Resolving deltas: 100% (15/15), done.
remote: Enumerating objects: 2883, done.
remote: Counting objects: 100% (2010/2010), done.
remote: Compressing objects: 100% (1720/1720), done.
remote: Total 2883 (delta 1018), reused 355 (delta 289), pack-reused 873 (from 1)
Receiving objects: 100% (2883/2883), 2.61 MiB | 2.20 MiB/s, done.
Resolving deltas: 100% (1168/1168), done.
Updating files: 100% (3494/3494), done.
  => download https://github.com/xmake-io/xmake.git .. ok
  => install script to /home/ruki/.local/share/xmake .. sh /home/ruki/.local/share/xmake/scripts/update-script.sh /home/ruki/.local/share/xmake /tmp/.xmake971/240826/xmakesrc/dev/xmake
ok

real    0m9.093s
user    0m0.932s
sys 0m1.501s
waruqi commented 1 month ago

depth 1

[ruki@5dd706d1f15e tmp]$ time git clone https://github.com/EsotericSoftware/spine-runtimes.git --depth 1
Cloning into 'spine-runtimes'...
remote: Enumerating objects: 5367, done.
remote: Counting objects: 100% (5367/5367), done.
remote: Compressing objects: 100% (3886/3886), done.
remote: Total 5367 (delta 1939), reused 3895 (delta 1351), pack-reused 0 (from 0)
Receiving objects: 100% (5367/5367), 81.73 MiB | 3.24 MiB/s, done.
Resolving deltas: 100% (1939/1939), done.
Updating files: 100% (5541/5541), done.

real    0m29.297s
user    0m3.782s
sys 0m7.461s

treeless

[ruki@5dd706d1f15e tmp]$ time git clone https://github.com/EsotericSoftware/spine-runtimes.git --filter=tree:0
Cloning into 'spine-runtimes'...
remote: Enumerating objects: 9496, done.
remote: Counting objects: 100% (581/581), done.
remote: Compressing objects: 100% (570/570), done.
remote: Total 9496 (delta 11), reused 571 (delta 11), pack-reused 8915 (from 1)
Receiving objects: 100% (9496/9496), 1.91 MiB | 1.93 MiB/s, done.
Resolving deltas: 100% (185/185), done.
remote: Enumerating objects: 703, done.
remote: Counting objects: 100% (507/507), done.
remote: Compressing objects: 100% (423/423), done.
remote: Total 703 (delta 25), reused 307 (delta 20), pack-reused 196 (from 1)
Receiving objects: 100% (703/703), 186.42 KiB | 763.00 KiB/s, done.
Resolving deltas: 100% (26/26), done.
remote: Enumerating objects: 4663, done.
remote: Counting objects: 100% (2417/2417), done.
remote: Compressing objects: 100% (1682/1682), done.
remote: Total 4663 (delta 1191), reused 1193 (delta 731), pack-reused 2246 (from 1)
Receiving objects: 100% (4663/4663), 81.45 MiB | 2.81 MiB/s, done.
Resolving deltas: 100% (1789/1789), done.
Updating files: 100% (5541/5541), done.

real    0m36.941s
user    0m4.327s
sys 0m8.773s
waruqi commented 1 month ago

I revert it first.

star-hengxing commented 1 month ago

Try xmake l scripts\test.lua --shallow -vD spine-runtimes

waruqi commented 1 month ago

Try xmake l scripts\test.lua --shallow -vD spine-runtimes

no package in xmake-repo

see

depth 1

[ruki@5dd706d1f15e tmp]$ time git clone https://github.com/EsotericSoftware/spine-runtimes.git --depth 1
Cloning into 'spine-runtimes'...
remote: Enumerating objects: 5367, done.
remote: Counting objects: 100% (5367/5367), done.
remote: Compressing objects: 100% (3886/3886), done.
remote: Total 5367 (delta 1939), reused 3895 (delta 1351), pack-reused 0 (from 0)
Receiving objects: 100% (5367/5367), 81.73 MiB | 3.24 MiB/s, done.
Resolving deltas: 100% (1939/1939), done.
Updating files: 100% (5541/5541), done.

real  0m29.297s
user  0m3.782s
sys   0m7.461s

treeless

[ruki@5dd706d1f15e tmp]$ time git clone https://github.com/EsotericSoftware/spine-runtimes.git --filter=tree:0
Cloning into 'spine-runtimes'...
remote: Enumerating objects: 9496, done.
remote: Counting objects: 100% (581/581), done.
remote: Compressing objects: 100% (570/570), done.
remote: Total 9496 (delta 11), reused 571 (delta 11), pack-reused 8915 (from 1)
Receiving objects: 100% (9496/9496), 1.91 MiB | 1.93 MiB/s, done.
Resolving deltas: 100% (185/185), done.
remote: Enumerating objects: 703, done.
remote: Counting objects: 100% (507/507), done.
remote: Compressing objects: 100% (423/423), done.
remote: Total 703 (delta 25), reused 307 (delta 20), pack-reused 196 (from 1)
Receiving objects: 100% (703/703), 186.42 KiB | 763.00 KiB/s, done.
Resolving deltas: 100% (26/26), done.
remote: Enumerating objects: 4663, done.
remote: Counting objects: 100% (2417/2417), done.
remote: Compressing objects: 100% (1682/1682), done.
remote: Total 4663 (delta 1191), reused 1193 (delta 731), pack-reused 2246 (from 1)
Receiving objects: 100% (4663/4663), 81.45 MiB | 2.81 MiB/s, done.
Resolving deltas: 100% (1789/1789), done.
Updating files: 100% (5541/5541), done.

real  0m36.941s
user  0m4.327s
sys   0m8.773s
star-hengxing commented 1 month ago

Update xrepo first https://github.com/xmake-io/xmake-repo/pull/5034

spine-runtimes need to checkout specify commit, so depth 1 not be useful on this case

waruqi commented 1 month ago

Update xrepo first xmake-io/xmake-repo#5034

spine-runtimes need to checkout specify commit, so depth 1 not be useful on this case

这个不能用来改进 git clone --depth 1，只能用来改进 git clone + git checkout 的 case。

https://github.com/xmake-io/xmake/pull/5510

star-hengxing commented 1 month ago

Useful for package that have git url(imgui docking/spine-runtimes/godotcpp4 etc...)

😥

waruqi commented 1 month ago

try it again.

Issues-translate-bot commented 1 month ago

Bot detected the issue body's language is not English, translate it automatically.

try it again.