Open Dich0tomy opened 2 years ago
Do you by any chance have the automatic fetch enabled? (that's the default) Maybe it hangs and causes said issues.
@mark2185 I didn't change any settings, so yes, that's probable that I have it enabled. I'll try to disable it and I'll come back with the results.
@mark2185 I tried. It doesn't help same issue.
I'm getting the same issue and I would love a fix or a workaround
@mattmiller-daxko could you specify your environment? E.g. OS, lazygit version?
If it's also on WSL2
, I found this (no answer) and this (long thread, but the link is to the answer).
In short, I cannot reproduce this, and if you're both on WSL
, I'd bet my two cents on WSL
itself being the root of the problems, not lazygit
.
I'm not using WSL. I'm on Windows 11 21H2 using Windows Terminal with cmder mini as my shell. Lazygit commit=367b0d331836c90c015bf0c45f88612f3d94d08a, build date=2022-07-20T09:27:56Z, build source=binaryRelease, version=0.35, os=windows, arch=amd64
Does the same happen with v0.34
?
Not that I ever noticed. It only started happening after upgrading to 0.35
Then it might be a regression. @mattmiller-daxko would you be willing to downgrade to v0.34
to see if it keeps happening? Same goes for @B4mbus
I seem to recall something about subprocesses being changed, but can't put my finger on it.
cc @jesseduffield for visibility and ideas
I have downgraded to v0.34
and I don't seem to have this problem anymore. I used v0.34
for several weeks before upgrading without any issues so I think it probably is a regression but I will leave another comment if I run into this issue again.
I also appreciate the quick response on this. I really love this git tool and would hate to have to switch to anything else.
@mark2185 I was running lazygit 0.34 all along, so the version is probably not a problem for me.
Uhhh, well then... try updating? 😅
@mark2185 Yeah I just upgraded it, nothing wrong so far, I'll report in a day or two.
I'm still having this issue with v0.35
. However, downgrading to v0.34
seems to solve it for me. I even tried disabling automatic refresh as suggested above but that did not solve the problem.
I'm getting this issue too on v0.35 (OSX). Quite annoying! Though for me I don't need to actually remove the lock file: I just retry the action and it works the second time.
Would be good if we were able to reproduce it reliably so I could write a failing test for it and fix it
I have 0.35 on both windows and linux and the issue doesn't occur... I don't know.
Welp, fun while it lasted
So it seems the issue is not version related and is somewhere else in the deepest corners of lazygit
.
Again - I'm using lazygit 0.35
on windows, and sometimes lazygit 0.35
on WSL2 and so far so good, no errors no more. For others, they appear.
It happened in my environment (macOS) even with v0.34.
It seemed to be caused by tmux's pane-border-format
(with git status) that was updating every second.
I had the same issue on the past versions and now have it again on v0.35
I've found that removing index file and reinstalling lazygit with scoop fixes the issue for me.
I'm working on migrating integration tests to the new format and I've been able to somewhat consistently reproduce the error on this PR: https://github.com/jesseduffield/lazygit/pull/2329
by running:
go run cmd/integration_test/main.go cli pkg/integration/tests/file/discard_changes.go
That test goes through a list of files and removes them one by one. Maybe one in ten times I get the index.lock error. Conveniently for me the test runs fast so it doesn't take long to get it to fail.
Some notes from my investigation:
go run cmd/integration_test/main.go cli --sandbox pkg/integration/tests/file/discard_changes.go
) and the index.lock error appears, I can wait a second and then try again and it works. This means that some process was running and then successfully completed (as opposed to dying and leaving the index.lock file there).git reset
commandstartBackgroundRoutines()
function in pkg/gui/background.go
)pkg/commands/oscommands/multi_mutex.go
). That is: assuming my code is right, no two lazygit commands should run simultaneously.deleted-them.txt
), right after doing a git reset
on the prior fileHas anybody had this issue since 0.39.3? We've added retry logic for when the index.lock file is temporarily in use
Closing until somebody has the issue again
Still meet this issue on 0.40.2.
My experience is that when this lock window appear, I exit lazygit, delete the index.lock
file, and enter lazygit, then:
a
, the lock window won't appear. Then if I hit a
again to unstage, and stage one file, the lock window won't appear.Still seeing this in 0.40.2 as well.
Still seeing this in 0.40.2 as well. Will this issue be re-open again?
Still seeing this in 0.40.2 as well.
commit=5e388e21c8ca6aa883dbcbe45c47f6fdd5116815, build date=2023-08-07T14:05:48Z, build source=binaryRelease, version=0.40.2, os=windows, arch=amd64, git version=2.34.1.windows.1
Can confirm it's still happening
commit=5e388e21c8ca6aa883dbcbe45c47f6fdd5116815, build date=2023-08-07T14:05:48Z, build source=binaryRelease, version=0.40.2, os=darwin, arch=arm64, git version=2.43.0
It seems that it's lazygit itself that creates the .git/index.lock
file. I thought that maybe it's related to the version of git running on my system (Ubuntu LTS) but it's not the case. It's happening with both the LTS version 2.37
as well as latest 2.43.2
.
I setup an audit for the file:
sudo auditctl -w /path/to/.git/index.lock -p w -k git-index-lock
Then when the bug is triggered i can see this :
ausearch git-index-lock
│ time->Thu Mar 7 08:55:02 2024
99 │ type=PROCTITLE msg=audit(1709798102.993:5255): proctitle=67697400616464002D2D006170702F66726F6E74656E642F7468656D65732F696D732F76696
│ 57773
100 │ type=PATH msg=audit(1709798102.993:5255): item=1 name="/path/to/repo/.git/index.lock" inode=11535492 dev=103:02 m
│ ode=0100664 ouid=1000 ogid=1000 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
101 │ type=PATH msg=audit(1709798102.993:5255): item=0 name="/path/to/repo/.git/" inode=11569583 dev=103:02 mode=040775
│ ouid=1000 ogid=1000 rdev=00:00 nametype=PARENT cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
102 │ type=CWD msg=audit(1709798102.993:5255): cwd="/path/to/repo"
103 │ type=SYSCALL msg=audit(1709798102.993:5255): arch=c000003e syscall=257 success=no exit=-17 a0=ffffff9c a1=5615a583ce60 a2=800c2 a3=1
│ b6 items=2 ppid=2598205 pid=2598291 auid=1000 uid=1000 gid=1000 euid=1000 suid=1000 fsuid=1000 egid=1000 sgid=1000 fsgid=1000 tty=pt
│ s0 ses=3 comm="git" exe="/usr/bin/git" subj=unconfined key="git-index-lock"
What may give a hint is that /usr/bin/git
exited with code -17 ?
@jorgheymans Thanks for that. It doesn't tell me much more, -17 seems to indicate that the git process died on a SIGSTOP signal, and I don't know what to make of that. Maybe someone with more Unix knowledge can figure out more?
I tried running lazygit with git tracing but then it doesn't start, probably output it does not expect.
GIT_TRACE=1 lazygit
2024/03/07 09:41:51 An error occurred! Please create an issue at: https://github.com/jesseduffield/lazygit/issues
*strconv.NumError strconv.ParseBool: parsing "09:41:51.648325 git.c:463 trace: built-in: git rev-parse --is-bare-repository\nfalse": invalid syntax
/home/runner/work/lazygit/lazygit/pkg/app/app.go:55 (0xb1b07c)
/home/runner/work/lazygit/lazygit/pkg/app/entry_point.go:150 (0xb1d286)
/home/runner/work/lazygit/lazygit/main.go:23 (0xb1eb1e)
/opt/hostedtoolcache/go/1.20.7/x64/src/runtime/internal/atomic/types.go:194 (0x437c87)
/opt/hostedtoolcache/go/1.20.7/x64/src/runtime/asm_amd64.s:1598 (0x467ec1)
Ah damn, that's a bug in lazygit. It tends to read both stdout and stderr most of the time, instead of just stdout. We fixed one specific occurrence of this problem in 438038a4f1, but there are many others, and fixing them all is probably a lot of work. Plus, if we do it in the way it was done in 438038a4f1 then it wouldn't help at all, because there we just swallow stderr instead of printing it to some log.
Next time it's occuring i'll try and setup an strace'd alias to git, perhaps that can reveal anything more useful.
I can confirm the bug with the same output:
GIT_TRACE=1 lazygit
2024/03/07 16:06:03 An error occurred! Please create an issue at: https://github.com/jesseduffield/lazygit/issues
*strconv.NumError strconv.ParseBool: parsing "16:06:03.040822 git.c:439 trace: built-in: git rev-parse --is-bare-repository\nfalse": invalid syntax
github.com/jesseduffield/lazygit/pkg/app/app.go:55 (0xb1dcbc)
github.com/jesseduffield/lazygit/pkg/app/entry_point.go:150 (0xb1fec6)
github.com/jesseduffield/lazygit/main.go:23 (0xb2175e)
runtime/internal/atomic/types.go:194 (0x439807)
runtime/asm_amd64.s:1598 (0x469ba1)
^[[I
but only in one specific project, can't make out the difference to the projects which are working
Is there some workaround for this?
A workaround for me was just to nuke the project and clone it again. the issue is gone.
Still meet this issue on 0.40.2.
My experience is that when this lock window appear, I exit lazygit, delete the
index.lock
file, and enter lazygit, then:
If I stage one file by hitting space on one file, the lock window appears again.- If I stage all files by hitting
a
, the lock window won't appear. Then if I hita
again to unstage, and stage one file, the lock window won't appear.
@owzim This workaround may save you from cloning your repo again.
What may give a hint is that /usr/bin/git exited with code -17 ?
Stating the obvious perhaps, but the -17 exit code actually relates to the openat syscall (syscall=257
). Which means git failed to open the lock file because it already exists, and the openat call probably specified O_CREAT or O_EXCL flag.
My bet is that it's a concurrency issue somewhere in lazygit or the git library that it's using. Or changes not being flushed to the filesystem right away allowing for other invocations of git to crash on the lockfile. https://github.com/jesseduffield/lazygit/issues/2050#issuecomment-1364375871 points in that direction, when all lazygit commands were forced to use the same mutex and the issue still occured even then.
@stefanhaller perhaps the issue could be opened again, i think there is sufficient evidence now.
EDIT perhaps related ? https://github.com/jesseduffield/lazygit/issues/652
Stating the obvious perhaps, but the -17 exit code...
Not obvious to me at all, thanks for explaining.
I have been running lazygit with debug flag enabled, and finally today had the error again. Don't think the output helps a great deal but in any case, here it is. (FWIW i think the logging output should have at least millisecond timestamps, also thread-ids perhaps?)
Mar 20 15:43:56 |DEBU| RunCommand command="git diff --submodule --no-ext-diff --unified=3 --color=always --cached -- var/myproject/pom.xml"
Mar 20 15:43:56 |INFO| postRefreshUpdate for files took 311.632µs
Mar 20 15:43:56 |DEBU| RunCommand command="git add -- var/myproject/pom.xml"
Mar 20 15:43:56 |ERRO| fatal: Unable to create '/home/user/src/backend/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Mar 20 15:43:56 |INFO| git add -- var/myproject/pom.xml (2.616191ms)
Mar 20 15:43:56 |WARN| index.lock prevented command from running. Retrying command after a small wait
Mar 20 15:43:56 |DEBU| RunCommand command="git add -- var/myproject/pom.xml"
Mar 20 15:43:56 |ERRO| fatal: Unable to create '/home/user/src/backend/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Mar 20 15:43:56 |INFO| git add -- var/myproject/pom.xml (5.264788ms)
Mar 20 15:43:56 |WARN| index.lock prevented command from running. Retrying command after a small wait
Mar 20 15:43:56 |DEBU| RunCommand command="git add -- var/myproject/pom.xml"
Mar 20 15:43:56 |ERRO| fatal: Unable to create '/home/user/src/backend/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Mar 20 15:43:56 |INFO| git add -- var/myproject/pom.xml (2.697067ms)
Mar 20 15:43:56 |WARN| index.lock prevented command from running. Retrying command after a small wait
Mar 20 15:43:56 |DEBU| RunCommand command="git add -- var/myproject/pom.xml"
Mar 20 15:43:56 |ERRO| fatal: Unable to create '/home/user/src/backend/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Mar 20 15:43:56 |INFO| git add -- var/myproject/pom.xml (2.640915ms)
Mar 20 15:43:56 |WARN| index.lock prevented command from running. Retrying command after a small wait
Mar 20 15:43:56 |DEBU| RunCommand command="git add -- var/myproject/pom.xml"
Mar 20 15:43:56 |ERRO| fatal: Unable to create '/home/user/src/backend/.git/index.lock': File exists.
Another git process seems to be running in this repository, e.g.
an editor opened by 'git commit'. Please make sure all processes
are terminated then try again. If it still fails, a git process
may have crashed in this repository earlier:
remove the file manually to continue.
Mar 20 15:43:56 |INFO| git add -- var/myproject/pom.xml (3.899458ms)
Mar 20 15:43:56 |WARN| index.lock prevented command from running. Retrying command after a small wait
Mar 20 15:43:56 |INFO| refreshing all scopes in async mode
Mar 20 15:43:56 |INFO| Refresh took 143.385µs
Mar 20 15:43:56 |DEBU| RunCommand command="git stash list -z --pretty=%gs"
Mar 20 15:43:56 |DEBU| RunCommand command="git worktree list --porcelain"
Mar 20 15:43:56 |DEBU| RunCommand command="git merge-base HEAD HEAD@{u}"
Mar 20 15:43:56 |DEBU| RunCommand command="git tag --list -n --sort=-creatordate"
Mar 20 15:43:56 |DEBU| RunCommand command="git merge-base HEAD refs/remotes/origin/main"
Mar 20 15:43:56 |INFO| refreshing the following scopes in async mode: files
Mar 20 15:43:56 |DEBU| using cache for key status.showUntrackedFiles
Mar 20 15:43:56 |INFO| Refresh took 805.133µs
Mar 20 15:43:56 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z"
Mar 20 15:43:56 |DEBU| RunCommand command="git diff --submodule --no-ext-diff --unified=3 --color=always --cached -- var/myproject/pom.xml"
Mar 20 15:43:56 |INFO| git -c log.showSignature=false log -g --abbrev=40 --format=%h%x00%ct%x00%gs%x00%p (4.886876ms)
Mar 20 15:43:56 |INFO| git branch -r (4.983607ms)
Mar 20 15:43:56 |INFO| postRefreshUpdate for remotes took 151.626µs
Mar 20 15:43:56 |INFO| postRefreshUpdate for remoteBranches took 12.851µs
Mar 20 15:43:56 |INFO| git merge-base HEAD HEAD@{u} (5.954334ms)
Mar 20 15:43:56 |INFO| git worktree list --porcelain (6.762329ms)
Mar 20 15:43:56 |INFO| postRefreshUpdate for localBranches took 311.912µs
Mar 20 15:43:56 |INFO| git stash list -z --pretty=%gs (7.956059ms)
Mar 20 15:43:56 |INFO| postRefreshUpdate for stash took 51.822µs
Mar 20 15:43:56 |INFO| postRefreshUpdate for worktrees took 30.032µs
Mar 20 15:43:56 |INFO| git merge-base HEAD refs/remotes/origin/main (8.564588ms)
Mar 20 15:43:56 |INFO| git log HEAD --topo-order --oneline --pretty=format:%H%x00%at%x00%aN%x00%ae%x00%D%x00%p%x00%s --abbrev=40 -300 --no-show-signature -- (13.720423ms)
Mar 20 15:43:56 |DEBU| RunCommand command="git rev-parse --abbrev-ref --verify HEAD"
Mar 20 15:43:56 |INFO| git tag --list -n --sort=-creatordate (13.805421ms)
Mar 20 15:43:56 |INFO| git rev-parse --abbrev-ref --verify HEAD (2.781575ms)
Mar 20 15:43:56 |DEBU| using cache for key rebase.updateRefs
Mar 20 15:43:56 |INFO| postRefreshUpdate for tags took 7.8235ms
Mar 20 15:43:56 |INFO| postRefreshUpdate for reflogCommits took 18.310613ms
Mar 20 15:43:56 |DEBU| RunCommand command="git for-each-ref --sort=-committerdate --format=%(HEAD)%00%(refname:short)%00%(upstream:short)%00%(upstream:track)%00%(subject)%00%(objectname) refs/heads"
Mar 20 15:43:56 |INFO| postRefreshUpdate for commits took 8.821605ms
Mar 20 15:43:56 |INFO| git for-each-ref --sort=-committerdate --format=%(HEAD)%00%(refname:short)%00%(upstream:short)%00%(upstream:track)%00%(subject)%00%(objectname) refs/heads (4.99527ms)
Mar 20 15:43:56 |INFO| postRefreshUpdate for localBranches took 329.302µs
Mar 20 15:43:56 |DEBU| using cache for key rebase.updateRefs
Mar 20 15:43:56 |INFO| git status --untracked-files=all --porcelain -z (28.833555ms)
Mar 20 15:43:56 |DEBU| using cache for key status.showUntrackedFiles
Mar 20 15:43:56 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z"
Mar 20 15:43:56 |INFO| postRefreshUpdate for submodules took 18.019µs
Mar 20 15:43:56 |INFO| postRefreshUpdate for files took 56.711µs
Mar 20 15:43:56 |INFO| git status --untracked-files=all --porcelain -z (20.726709ms)
Mar 20 15:43:56 |INFO| postRefreshUpdate for submodules took 19.486µs
Mar 20 15:43:56 |INFO| postRefreshUpdate for files took 58.737µs
Mar 20 15:43:56 |INFO| refreshing the following scopes in sync mode: files
Mar 20 15:43:56 |DEBU| using cache for key status.showUntrackedFiles
Mar 20 15:43:56 |INFO| refreshed merge conflicts in 22.489µs
Mar 20 15:43:56 |DEBU| RunCommand command="git status --untracked-files=all --porcelain -z"
Mar 20 15:43:56 |INFO| git status --untracked-files=all --porcelain -z (32.182313ms)
Mar 20 15:43:56 |INFO| refreshed files in 32.850556ms
Mar 20 15:43:56 |INFO| Refresh took 33.168754ms
Mar 20 15:43:56 |INFO| postRefreshUpdate for submodules took 32.965µs
Mar 20 15:43:56 |INFO| postRefreshUpdate for files took 115.867µs
I'm also experiencing the issue frequently, especially when working with very large repos (10000+ files). I assume lazygit runs a git status but it's not apparent anywhere that it's running, so when attempting any other task I get the error. However even after waiting a bit for the git status to complete (timed the command in my terminal, then waited that long inside lazygit), lazygit still complains that the index.lock exists. Manually removing it solves the issue for me, at least until two git processes spawn and conflict (or a process starts and fails), then we're back to square one.
I'm willing to try anything and provide any information needed to debug this - but I'm unsure where to start. If someone can provide some pointers it'd be much appreciated.
This is happening to me (regularly, in 0.41.0
) when I attempt to discard changes on a deleted file. I can work around it by leaving lazygit
, deleting the lock file (which is in a worktree directory, eg: ../.bare/worktrees/foo/index.lock
), and then invoking git
with checkout
to restore the file.
same issue on
$ lazygit -v
commit=3675570a391b1a49ddd198b4c7e71e17701d4404, build date=2024-03-23T09:09:11Z, build source=binaryRelease, version=0.41.0, os=darwin, arch=arm64, git version=2.39.3 (Apple Git-145)
Having the same issue to the point I've started using git add -p
more than the lazygit :\
$ lazygit -v
commit=, build date=, build source=homebrew, version=0.42.0, os=darwin, arch=arm64, git version=2.45.2
uname -a
Darwin mbp.fritz.box 23.5.0 Darwin Kernel Version 23.5.0: Wed May 1 20:12:58 PDT 2024; root:xnu-10063.121.3~5/RELEASE_ARM64_T6000 arm64
Facing this issue for a number of months now.
lazygit v0.43.1
macOS 14.6.1
git 2.46.0
The way I have been resolving it is to quit lazygit, rm .git/index.lock
and then start lazygit again.
I too am still getting this occasionally. This is the one true bug to rule them all.
I also see this, but very rarely.
I looked a bit for code that might cause it, and came across several places where we kill git processes using SIGKILL (which is pretty brutal and doesn't give it a chance to clean up). As far as I can see, we do this when quickly flicking through items in a list; now, for flicking through commits it shouldn't be a problem, because the git show
command that we use for this shouldn't need the index lock. But when flicking through local files, it is totally possible that we kill a git diff --cached
invocation which did take the lock.
So that would totally explain it, but if that's it, then I'm surprised that it doesn't happen much more often. Like, all the time when quickly moving the selection up and down in the files panel. I tried to reproduce it this way, but couldn't.
Still, I wonder if we should try killing processes a little more gracefully, like first sending them a SIGTERM, and only if they are still alive after so many milliseconds, use the bigger hammer. I'm not sure it's possible to do this asynchronously (I didn't look at the code to see if it expects the process to be gone immediately), and doing it synchronously could make things more laggy.
Curious about @jesseduffield's opinion on all this.
I'm certainly open to a SIGTERM approach as you describe, but I think the first step is to come up with a way to reliably reproduce the issue so that we can be sure we've fixed it (even if only in some narrow context). I'm imagining an integration test that forces the bug to occur.
I totally agree that we should try to make it better reproducible, and I mostly wrote the above in order to give those who run into it more often then me a clue in what direction to experiment with it. (Or those who know the code very well 😉)
I am, however, very sceptical that it will be possible to write a test for it. I bet it is timing related in some way, or maybe it only happens when the machine is under heavy load or something.
Agreed. I tried my hand at writing a test for it today but with no luck.
it happens occasionally to me too
lazygit 0.44.1
git 2.39.5
Debian 12.7
For some time now lazygit seems to bug all the time. I'm 100% sure that there are no other programs running in the background that could do some git operations.
I try to open lazygit from the terminal, but everytime I try to do any operation after that I get the error about the git lock. So I remove it by hand
rm -rf .git/index.lock
and get back to lazygit, same thing.It doesn't matter if I do this from bare console or from the console inside neovim. Sometimes after deleting the index.lock file and getting back it works fine, only to complain again a usage later.
If I remove the file manually and then use git commands instead of using lazygit, everything works fine.
Windows, but I'm on WSL2 kali-linux
Linux stacjonarny 5.4.72-microsoft-standard-WSL2 #1 SMP Wed Oct 28 23:40:43 UTC 2020 x86_64 GNU/Linux
Lazygit version 0.34