Lazygit crashes when pushing or pulling

[edhgoose]

Describe the bug When pulling or pushing a branch lazygit will occasionally crash.

To Reproduce It's really difficult to figure out what's occurring - not all pull's/push's fail, and the majority of the time lazygit works fantastic.

I haven't been able to create a reproduction step, but am happy to try out things if you've any suggestions.

Some things I've noticed that might be useful:

• It usually comes with a hang - lazygit will spend a long time with the pulling/pushing UI waiting
• Stopping the command (usually killing lazygit) and restarting it or running it manually via git usually unglitches things
• It's inconsistent - there doesn't seem to be a common pattern where it always occurs
• I've not seen anything obvious in the logs.

Expected behavior I'd expect lazygit doesn't hang and doesn't crash :)

Screenshots N/A

Version info: Run lazygit --version and paste the result here commit=5e388e21c8ca6aa883dbcbe45c47f6fdd5116815, build date=2023-08-07T14:05:48Z, build source=binaryRelease, version=0.40.2, os=darwin, arch=arm64, git version=2.39.2 (Apple Git-143)

Run git --version and paste the result here git version 2.39.2 (Apple Git-143)

Additional context Add any other context about the problem here.

I've started running in --debug mode, and I usually get a deadlock like this one:

POTENTIAL DEADLOCK:
Previous place where the lock was grabbed
goroutine 699846 lock 0x14000392150
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:51 oscommands.(*cmdObjRunner).RunWithOutput ??? <<<<<
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:50 oscommands.(*cmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:34 commands.(*gitCmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 commands.(*gitCmdObjRunner).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:189 oscommands.(*CmdObj).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:77 git_commands.(*SyncCommands).FetchBackground ???
/home/runner/work/lazygit/lazygit/pkg/gui/background.go:107 gui.(*BackgroundRoutineMgr).backgroundFetch ???
/home/runner/work/lazygit/lazygit/pkg/gui/background.go:65 gui.(*BackgroundRoutineMgr).startBackgroundFetch.func1 ???
/home/runner/work/lazygit/lazygit/pkg/gui/background.go:91 gui.(*BackgroundRoutineMgr).goEvery.func1.1 ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:679 gocui.(*Gui).onWorkerAux ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:665 gocui.(*Gui).OnWorker.func1 ???

Have been trying to lock it again for more than 30s
goroutine 719890 lock 0x14000392150
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:51 oscommands.(*cmdObjRunner).RunWithOutput ??? <<<<<
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:50 oscommands.(*cmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:34 commands.(*gitCmdObjRunner).RunWithOutput ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 commands.(*gitCmdObjRunner).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:189 oscommands.(*CmdObj).Run ???
/home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:48 git_commands.(*SyncCommands).Push ???
/home/runner/work/lazygit/lazygit/pkg/gui/controllers/sync_controller.go:180 controllers.(*SyncController).pushAux.func1 ???
/home/runner/work/lazygit/lazygit/pkg/gui/popup/popup_handler.go:155 popup.(*PopupHandler).WithLoaderPanel.func1 ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:679 gocui.(*Gui).onWorkerAux ???
/home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:665 gocui.(*Gui).OnWorker.func1 ???

Here is what goroutine 699846 doing now
goroutine 699846 [syscall, 10 minutes]:
syscall.syscall6(0x104038f18?, 0x102e20517?, 0x0?, 0x0?, 0x14001659848?, 0x90000101010108?, 0x12b7120f8?)
    /opt/hostedtoolcache/go/1.20.7/x64/src/runtime/sys_darwin.go:45 +0x68
syscall.wait4(0x14001659888?, 0x102efede8?, 0x90?, 0x103733480?)
    /opt/hostedtoolcache/go/1.20.7/x64/src/syscall/zsyscall_darwin_arm64.go:43 +0x4c
syscall.Wait4(0x140016598a8?, 0x140016598c4, 0x140016598e8?, 0x103304324?)
    /opt/hostedtoolcache/go/1.20.7/x64/src/syscall/syscall_bsd.go:144 +0x28
os.(*Process).wait(0x14002290120)
    /opt/hostedtoolcache/go/1.20.7/x64/src/os/exec_unix.go:43 +0x80
os.(*Process).Wait(...)
    /opt/hostedtoolcache/go/1.20.7/x64/src/os/exec.go:132
os/exec.(*Cmd).Wait(0x14003066160)
    /opt/hostedtoolcache/go/1.20.7/x64/src/os/exec/exec.go:890 +0x38
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runAndStreamAux(0x14000394560, {0x103770008, 0x14000046240}, 0x14001659c80)
    /home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:255 +0x550
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runAndDetectCredentialRequest(0x14000394560, {0x103770008, 0x14000046240}, 0x10375c9d0)
    /home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:328 +0xc0
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).runWithCredentialHandling(0x14000323d60?, {0x103770008, 0x14000046240})
    /home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:303 +0x54
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*cmdObjRunner).RunWithOutput(0x14000046040?, {0x103770008, 0x14000046240})
    /home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj_runner.go:56 +0xf0
github.com/jesseduffield/lazygit/pkg/commands.(*gitCmdObjRunner).RunWithOutput(0x140006f6240, {0x103770008, 0x14000046040})
    /home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:33 +0xe4
github.com/jesseduffield/lazygit/pkg/commands.(*gitCmdObjRunner).Run(0x5?, {0x103770008?, 0x14000046040?})
    /home/runner/work/lazygit/lazygit/pkg/commands/git_cmd_obj_runner.go:24 +0x24
github.com/jesseduffield/lazygit/pkg/commands/oscommands.(*CmdObj).Run(0x140006ec048?)
    /home/runner/work/lazygit/lazygit/pkg/commands/oscommands/cmd_obj.go:190 +0x38
github.com/jesseduffield/lazygit/pkg/commands/git_commands.(*SyncCommands).FetchBackground(0x1036add00?)
    /home/runner/work/lazygit/lazygit/pkg/commands/git_commands/sync.go:78 +0x28
github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).backgroundFetch(0x14000394570)
    /home/runner/work/lazygit/lazygit/pkg/gui/background.go:104 +0x30
github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).startBackgroundFetch.func1()
    /home/runner/work/lazygit/lazygit/pkg/gui/background.go:64 +0x24
github.com/jesseduffield/lazygit/pkg/gui.(*BackgroundRoutineMgr).goEvery.func1.1({0x1400177e798, 0x10311b3a4})
    /home/runner/work/lazygit/lazygit/pkg/gui/background.go:91 +0x2c
github.com/jesseduffield/gocui.(*Gui).onWorkerAux(0x0?, 0x102e7a624?, {0x1037682b0?, 0x140015a6040?})
    /home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:678 +0x6c
github.com/jesseduffield/gocui.(*Gui).OnWorker.func1()
    /home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:665 +0x34
created by github.com/jesseduffield/gocui.(*Gui).OnWorker
    /home/runner/work/lazygit/lazygit/vendor/github.com/jesseduffield/gocui/gui.go:664 +0x9c

Note: please try updating to the latest version or manually building the latest master to see if the issue still occurs.

[stefanhaller]

Have you ever seen it crash when running without --debug? (My guess is no.)

I've seen it hang too, and it's very annoying when it happens. My theory is that there are situations where a background fetch never returns, and so the pull/push waits forever to grab the "global" sync mutex.

For me, it seems to have something to do with sleep mode. I think I have seen it only when lazygit was running when I put the MacBook to sleep, and then after waking up the next fetch/pull/push would hang. Not always though. So one theory could be that if you put the machine to sleep while a background fetch is running, it will then never return after waking up again. I have no idea why it would behave this way though, or what to do about it if it's true.

[mark2185]

or what to do about it if it's true.

Add a kill after a timeout? So that a goroutine lives e.g. 30s or less?

[stefanhaller]

I don't think that would be a good idea. In very large repos, fetching can easily take much longer than 30s.

[edhgoose]

Good question @stefanhaller - I don't think I have seen a crash when running without --debug.

One thing I think is true (but I'll keep an eye out for) is that when it does hang, the command window doesn't populate with the git command. It'll just have Push in the window, but not the git command. It's like the command isn't even executing. Maybe that's useful?

[stefanhaller]

This only confirms that the Push command is still blocking on the global sync mutex (but we know this already from the logs); it will only log its command once it aquired the mutex, which it never does.

What we need to find out is why the other process (the background fetch) never finishes.

[edhgoose]

Ah, I'm with you. Is there anything I can do to help try and track that down?

[stefanhaller]

I don't know; do a bit of research, perhaps? I'm not very experienced with networking stuff. A very cursory google search turned up this, which sounds somewhat related. I suppose more reading might be necessary.

[edhgoose]

Am I right in saying that you think the issue is not to do with lazygit itself, but actually git? I was working to the former, but sounds like you think it might be the latter?

[stefanhaller]

Depends on how you look at it. It does have to do with lazygit insofar as its management of the global sync mutex goes, and how it assumes that no process will hold on to it forever. (Well, I suppose this is true of any mutex.)

But yes, I think the issue stems from the git fetch background process never returning, and that seems outside of lazygit's control.

[stefanhaller]

Of course, there's also the possibility that the git fetch process is actually fine, and it only hangs because of the way we poll for its completion in go. But that would point to a problem in go's exec module, which seems less likely to me.

[CassandraCat]

I also encountered the same problem. lazygit would always hang when pushing, and then it would take a long time to succeed.