Use libgit2/git2go instead of command line git

[jrwrigh]

Topic There's a library called lib2git that has a library made to integrate with git without having to run command line options in sub shell like (I believe) y'all are currently doing.

Said project has go bindings as well at git2go.

Your thoughts This is mostly just an "FYI, this thing exists which might be better/useful". I haven't looked into it much, but I feel like it might solve some of the issues I've been dealing with related to interacting with different versions of git, which seem to mostly be changing of flag names. (see #557, #553, #548 (already fixed, but possibly avoided doing it this way).

[dawidd6]

We are aware of this. In fact we are already using go-git for some operations, which is a pure Go git implementation.

Granted that using the library fully instead of mixing it with shell commands would be better, but one needs to actually take some time to make the transition and make sure everything works as expected.

PRs are always welcomed.

[jrwrigh]

Out of curiosity, any reason behind using go-git and not git2go? My (naive) point of view, seems like git2go is the more feature complete option (the go-git website even mentions that their goals is to "each the completeness of libgit2".

[dawidd6]

git2go is just a binding to libgit2, you still need libgit2 C library installed to be able to compile the project using it and go-git is a standalone pure Go implementation of git functionality

[jesseduffield]

also worth noting that I've spent a bit of time trying to use go-git and it didn't have as much functionality as I first expected, and proved slower in some ways (e.g. fetching commits)

[mjarkk]

These kinds of libraries that include c files also runes a bit the simplicity of go because if we would include this,
libgit2 also needs to also be installed on the computer separately and for programmers that have never used c this can be a bit scary and might stop some users from committing to lazygit.

I personally don't have a good experience with go libraries that include non go files because they never seem to be working out of the box and always require me to search for the right package in the package manager and for some linux distros this can be a bit tricky not even talking about windows.

[jesseduffield]

Is it possible to install lib2git along with lazygit? I.e. bundle it up into the binary? If so I'd be much keener on using it, especially for expensive operations like loading commits

[mjarkk]

As far as I know you only need libgit2 when you want to build the binary from source to use the build binary you don't need libgit2.

[jesseduffield]

That would be great, I do want to switch over to something which is reliably fast for loading lots of git objects

[mloskot]

Interestingly, it seems to me that the fact that lazygit uses git CLI makes it not prone to libgit2 bugs like this one I discovered while using gitui, https://github.com/extrawurst/gitui/issues/1696

[jesseduffield]

Funnily I was reading up on libgit2 yesterday: seems like Gitlab has deprecated it in their Gitaly project due to it lagging behind the git binary in features. I don't know how many advanced features they're trying to use but it's food for thought.

I'd still like to try libgit2 out and see how much it can speed things up in certain places.

[jesseduffield]

BTW if we introduce libgit2 I want it statically linked, not dynamically linked. We'd need to see how that affects the binary size but I really don't want users to have to install two separate things (the git binary is an exception because everybody already has it)

[stefanhaller]

I'm very much against using libgit2. It always lags behind git so significantly that I don't think the benefits are worth it.

The main reason for using it is probably performance: we'd avoid forking subprocesses, which is especially beneficial on Windows where forking is so expensive. But I think the price is too high.

Just as an example, I was looking whether its rebase API supports more advanced options such as --update-refs or --rebase-merges, just to find out that it doesn't even support interactive rebase at all.

For push, I was looking in the issue tracker to see if it has at least plans to support options like --force-if-includes, just to see that it doesn't even support --force-with-lease, which has been around forever.

Upstream git is working on many improvements under the hood all the time, some of which don't even affect the API; examples are the introduction of the merge-ort strategy, improvements to the diff algorithm, introduction of the commit-graph cache, and many, many others. I want to benefit from these as soon as they are available, instead of having to wait years until they are supported in libgit2.

The same goes for go-git (even more), support for even basic operations is extremely incomplete.

[mloskot]

I'm not lazygit contributor, but after evaluating Git TUI clients as a user I second @stefanhaller's opinion - as a user I can trade some run-time speed for Git feature-complete support.

[jesseduffield]

Thanks for that context @stefanhaller that's really informative. I should be clear: I would never replace the git CLI with libgit2 (unless it proved itself capable of keeping up with git). I still would like to experiment on how fast it can load commits, given that it's the slowest thing we do and I don't imagine the git CLI has advanced much on that front recently. At that point it's just a question of how much the binary size increases. Tell me if I'm wrong about that!

But I'm not expecting much: go-git has disappointed me in pretty much every place I've tried to use it.

[jesseduffield]

What I really want is for the git project to be split into a frontend and a backend where the frontend is just the CLI interface and the backend includes porcelain commands, so that I can then get some bindings to the backend and get all the benefits of up-to-date git without spawning new processes. I'm actually confused about why libgit2 needs to exist in the first place: it's written in C and git is written in C: seems like you should be able to get what you want directly from git. I'm definitely missing something.

[stefanhaller]

I still would like to experiment on how fast it can load commits, given that it's the slowest thing we do and I don't imagine the git CLI has advanced much on that front recently. At that point it's just a question of how much the binary size increases. Tell me if I'm wrong about that!

I'm very skeptical. I'd have to research this more, but git log is also a performance-critical command for command-line git, so I wouldn't be surprised if they are constantly tweaking its performance under the hood. Some of the Microsoft folks have done some amazing work in areas like this in recent years, especially for large repos. The only performance gain that we would get from libgit2 is avoiding one fork/exec; this is going to be totally insignificant in comparison.

What I really want is for the git project to be split into a frontend and a backend where the frontend is just the CLI interface and the backend includes porcelain commands, so that I can then get some bindings to the backend and get all the benefits of up-to-date git without spawning new processes. I'm actually confused about why libgit2 needs to exist in the first place: it's written in C and git is written in C: seems like you should be able to get what you want directly from git. I'm definitely missing something.

If you search for "libify" in the commit messages of the git repo, you'll see a lot of commits where things appear to be moving in that direction. It's just a slow process.

The two big problems that make this hard are 1. that a lot of the git code has traditionally done error handling by just calling exit, which is kind of bad in a library, and 2. that a lot of git code doesn't care about memory leaks, and just allocates stuff that it never frees, knowing that the process will terminate soon anyway. It seems that the libgit2 folks have decided that it's easier to do a rewrite from scratch rather than trying to fix these all over the place.

[jesseduffield]

The only performance gain that we would get from libgit2 is avoiding one fork/exec; this is going to be totally insignificant in comparison.

Great point.

It seems that the libgit2 folks have decided that it's easier to do a rewrite from scratch rather than trying to fix these all over the place.

That makes sense. Thanks again for the context!

[kytta]

My two cents on this: Although I use lazygit in 99% of my Git workflow tasks, I still value a lot that it prints a working Git command in the command log. I can look at it to grasp what lazygit does, I can learn from it, or I can send it to my colleague if they want to do the same operation as I do.

As far as I understand, lib2git (and its bindings) do not reproduce the CLI command — this would be a big setback for me!