Cargo downloads git dependencies very slowly - Githubissues

rust-lang / cargo

The Rust package manager

https://doc.rust-lang.org/cargo

Apache License 2.0

12.32k stars 2.33k forks source link

Cargo downloads git dependencies very slowly #13624

Open WaffleLapkin opened 3 months ago

WaffleLapkin commented 3 months ago

Problem

When compiling a project that uses a lot of git dependencies it takes a lot of time for cargo to download them.

Example that I just seen:

It took minutes to download 50 git dependencies
It then took 7.23 seconds to download 300 crates.io dependencies

Seems like there is a big potential for optimizations here.

Steps

Clone https://github.com/zed-industries/zed/tree/main
Run git submodule update --init --recursive
Run cargo build
Observe the slow downloading of git dependencies

Possible Solution(s)

Using a shallow clone, instead of clonning the full repo
Downloading multiple repositories in parallel

Notes

You may need to delete some caches to be able to repeat this test. I think you need to delete ~/.cargo/git/.

Version

cargo 1.76.0 (c84b36747 2024-01-18)
release: 1.76.0
commit-hash: c84b367471a2db61d2c2c6aab605b14130b8a31b
commit-date: 2024-01-18
host: x86_64-unknown-linux-gnu
libgit2: 1.7.1 (sys:0.18.1 vendored)
libcurl: 8.5.0-DEV (sys:0.4.70+curl-8.5.0 vendored ssl:OpenSSL/1.1.1w)
ssl: OpenSSL 1.1.1w  11 Sep 2023
os: Arch Linux Rolling Release [64-bit]

epage commented 3 months ago

For initial downloads, one big slow down compared to crates.io is that we download the entire history. We have unstable support for shallow clone and would recommend trying that to see how much it improves the problem. See #13285

For updates, there seem to be performance issues in libgit2 (#9167) and we are working on transitioning to gitoxide. Maybe give that a try as well. See #1181.3

Now, you may not want to rely on a nightly toolchain for development. Doing the above will help us narrow down root cause and determine if this is a duplicate. Once that is done, you could see if net.git-fetch-with-cli works around the problem for you.