Closed arxanas closed 2 years ago
I noticed the same happens to a much smaller degree in mozilla/gecko-dev:
$ time git-stack
master no bug - Bumping Firefox l10n changesets r=release a=l10n-bump DONTBUILD
git-stack 2.82s user 0.14s system 99% cpu 2.969 total
$ git checkout --detach && git branch -D master
HEAD is now at 2dc538408a1f no bug - Bumping Firefox l10n changesets r=release a=l10n-bump DONTBUILD
branchless: processing checkout
Deleted branch master (was 2dc538408a1f).
$ time git-stack
2dc5384 no bug - Bumping Firefox l10n changesets r=release a=l10n-bump DONTBUILD
git-stack 0.01s user 0.01s system 89% cpu 0.020 total
Here's the output from git branch -a
for gecko-dev, if it's relevant:
I got a flamegraph with cargo flamegraph
, but I don't understand why it's doing a revwalk:
(For some reason, Github transformed the .svg into something less interactive, so you can't zoom in on it and read the function names. I took a screenshot of the .svg and uploaded that instead. Let me know if you want the original .svg and I can try to upload it somewhere else.)
Thanks for digging into that!
We are using revwalk for
It seems starting a revwalk is the most expensive step and most of that time comes from forcing it to be a topological sort.
So it seems like if we'd need to avoid a topological sort to make things faster though we'll need these walks to be topological. I assume doing a lot of this ourselves would speed it up.
What commit graph needs to be built/what distance needs to be calculated/etc. if you have only one branch? I would have figured that would only be necessary if you have multiple branches, or if the local branch tracks a remote branch which is far off (and even then, I wouldn't expect it to take 25s — that's around the time it takes to traverse every commit in the repository, in my memory).
if you have only one branch?
We could bypass some of the checks, for example, speeding up distance between HEAD
and each protected branch (to find the closest) can easily be optimized by protected_branches.contains(head)
. However, I decided to leave the code general for now so its easier to reproduce the performance issues so we can optimize it generally and then worry about it in the particular cases afterwards.
What commit graph needs to be built/what distance needs to be calculated/etc. if you have only one branch?
I was also listing every place we use revwalk. The "distance between HEAD and master" was the one taking the most time in my tests. In theory, that should be a trivial revwalk since it should terminate on the first item returned but all of the time is taken up before that first item is returned.
Ran the following in the gecko-dev repo
fn walk_noop(_repo: &git2::Repository, _base_id: git2::Oid, _head_id: git2::Oid) -> Vec<git2::Oid> {
Vec::new()
}
fn walk_custom(repo: &git2::Repository, base_id: git2::Oid, head_id: git2::Oid) -> Vec<git2::Oid> {
let mut revwalk = repo.revwalk().unwrap();
revwalk.push(head_id).unwrap();
revwalk
.filter_map(Result::ok)
.take_while(move |oid| *oid != base_id)
.collect::<Vec<_>>()
}
fn walk_custom_hide(
repo: &git2::Repository,
base_id: git2::Oid,
head_id: git2::Oid,
) -> Vec<git2::Oid> {
let mut revwalk = repo.revwalk().unwrap();
revwalk.push(head_id).unwrap();
revwalk.hide(base_id).unwrap();
revwalk.filter_map(Result::ok).collect::<Vec<_>>()
}
fn walk_custom_topo(
repo: &git2::Repository,
base_id: git2::Oid,
head_id: git2::Oid,
) -> Vec<git2::Oid> {
let mut revwalk = repo.revwalk().unwrap();
revwalk.push(head_id).unwrap();
revwalk.set_sorting(git2::Sort::TOPOLOGICAL).unwrap();
revwalk
.filter_map(Result::ok)
.take_while(move |oid| *oid != base_id)
.collect::<Vec<_>>()
}
fn walk_manual(repo: &git2::Repository, base_id: git2::Oid, head_id: git2::Oid) -> Vec<git2::Oid> {
let mut result = Vec::new();
let mut queue = std::collections::VecDeque::new();
queue.push_back(head_id);
while let Some(next_id) = queue.pop_front() {
if next_id == base_id {
continue;
}
result.push(next_id);
let commit = repo.find_commit(next_id).unwrap();
queue.extend(commit.parent_ids());
}
result
}
fn main() {
let base_id = git2::Oid::from_str("fd583822b7fb236b7dcdac5e7c4dad944b183944").unwrap();
let head_id = git2::Oid::from_str("327db8421f8bbe94ac26b4a5a68793cf817ad75c").unwrap();
let mut args = std::env::args().skip(1);
let name = args.next();
let algorithm = match name.as_deref() {
Some("custom") => walk_custom,
Some("custom_topo") => walk_custom_topo,
Some("custom_hide") => walk_custom_hide,
Some("manual") => walk_manual,
Some(name) => panic!("Unrecognized algorithm `{}`", name),
None => walk_noop,
};
{
let repo = git2::Repository::discover(".").unwrap();
let timer = std::time::Instant::now();
let commits = (algorithm)(&repo, base_id, head_id);
println!("elapsed: {}us", timer.elapsed().as_micros());
dbg!(commits);
}
}
$ cargo run --
Finished dev [unoptimized + debuginfo] target(s) in 0.01s
Running `target/debug/exp`
elapsed: 0us
[src/main.rs:74] commits = []
$ cargo run -- custom
Finished dev [unoptimized + debuginfo] target(s) in 0.01s
Running `target/debug/exp custom`
elapsed: 248us
[src/main.rs:74] commits = [
327db8421f8bbe94ac26b4a5a68793cf817ad75c,
]
$ cargo run -- custom_topo
Finished dev [unoptimized + debuginfo] target(s) in 0.01s
Running `target/debug/exp custom_topo`
elapsed: 17376063us
[src/main.rs:74] commits = [
327db8421f8bbe94ac26b4a5a68793cf817ad75c,
]
$ cargo run -- custom_hide
Finished dev [unoptimized + debuginfo] target(s) in 0.01s
Running `target/debug/exp custom_hide`
elapsed: 557us
[src/main.rs:74] commits = [
327db8421f8bbe94ac26b4a5a68793cf817ad75c,
]
$ cargo run -- manual
Finished dev [unoptimized + debuginfo] target(s) in 0.01s
Running `target/debug/exp manual`
elapsed: 114us
[src/main.rs:74] commits = [
327db8421f8bbe94ac26b4a5a68793cf817ad75c,
]
(multiple runs were done and a representative value was selected for inclusion, except in the topo case)
Granted, these all act slightly differently but one size does not fit all. You rarely need the most expensive option.
Also, each algorithm needed to be run in its own process, not just its own git2::Repository
, because of caching.
atm we don't support merge-commits and just topologically sort to resolve them. That is the main thing left to see the performance issue go away.
@epage Thanks for the quick 😉 fix! It works a lot faster now in my repository.
Thanks for the feedback as I don't normally get to work with large repos lately.
btw is this just for comparison purposes or is there a way it you are using it in your workflow along with git-branchless? Be curious to know how you might be using them together.
@epage I figured I'd try it out to manage my interactions with Github. Specifically, I don't have a way of pushing all branches in my current stack, which git stack --push
seems to do for me (well, it pushes all of the ready ones, which is fine for me).
For more discussion on --push
, see
I noticed in a large repository that
git-stack
invocations seem to take a while:But if I delete
master
, then it completes immediately:Is there any way to debug what's going on here?
This is on a large repository (500k+ commits) so it's understandable if you don't want to support this use-case.