search

frewsxcv / rust-crates-index

Rust library for retrieving and interacting with the crates.io index
https://docs.rs/crates-index/
Apache License 2.0
72 stars 37 forks source link

Slow tests and large checkouts #47

Closed Eh2406 closed 3 years ago

Eh2406 commented 3 years ago

I was noticing that the tests take a long time to run. 15min on travis. I was also noting that the temp dirs created by the test are much bigger then Cargos checkout. 800MB vs 75MB. I wonder if we are fetching all branches not just the HEAD branch.

I will happily make a PR, but did not see a libgit2 setting for this. I also did not figure out where Cargo does the fetch operation, so as to copy there code. Advice please?

kornelski commented 3 years ago

It is fetching just the master branch.

I think part of the problem is that crates index does a checkout by default. There's new BareIndex interface that avoids it.

Eh2406 commented 3 years ago

%temp%/opens_bare_index.*, %temp%/clones_bare_index.* are all 407 MB, and oddly so are all of the %temp%/test.* directories. Where as \.cargo\registry\index\github.com-1ecc6299db9ec823\.git is 82.7 MB. I will try to experiment with a BareIndex and a non %temp% directory to verifly.

Eh2406 commented 3 years ago

Can confirm:

fn main() {
    let index = crates_index::BareIndex::with_path("this_is_a_test".into(), "https://github.com/rust-lang/crates.io-index");
    index.open_or_clone();
}

takes ~5min and 449 MB. But git branch -l reports only master. So it is a problem even with BareIndex but it is not the other branches. No idea.