search

src-d / go-git

Project has been moved to: https://github.com/go-git/go-git
https://github.com/go-git/go-git
Apache License 2.0
4.91k stars 542 forks source link

feat: improve clone performance and reduce memory usage. #1170

Closed orisano closed 5 years ago

orisano commented 5 years ago

I tried to clone github.com/knqyf263/vuln-list, but PlainClone took a too long time. I improve clone performance and reduce memory usage!

benchstat results

name          old time/op    new time/op    delta
PlainClone-4      605s ± 4%      173s ± 5%  -71.42%  (p=0.008 n=5+5)

name          old alloc/op   new alloc/op   delta
PlainClone-4    64.9GB ± 0%    29.8GB ± 0%  -54.12%  (p=0.008 n=5+5)

name          old allocs/op  new allocs/op  delta
PlainClone-4      117M ± 0%      114M ± 0%   -2.50%  (p=0.008 n=5+5)

thanks!

jfontan commented 5 years ago

It seems that the biggest improvement are the changes in worktree.go and storage/filesystem/index.go. Can yo split this PR in two? One with changes for those files and another with changes for packfile code.

I've been reviewing the changes for packfile and make sense but in some cases they make the code slower. I'm pretty sure this can be improved to get good gains in both memory and speed when it's generating the index.

@mcuadros what do you think?

orisano commented 5 years ago

Ok, I split PR.

orisano commented 5 years ago

Shall I close this PR?

jfontan commented 5 years ago

@orisano I would love to take a look to the changes for packfile you did. Even if they are not increasing speed now they look very promising. Can you open a new PR with only the changes from the files in plumbing/format/packfile/*. We can work from there to improve them.

Thanks!

orisano commented 5 years ago

ok

jfontan commented 5 years ago

The changes from this PR are now in #1180