Closed bensherman closed 2 years ago
More testing - I created a Procfile with the following contents, but the scripts didn't exist yet, and I got the same behavior:
sleepy: ./snoozy.sh
snoozy: ./snoozy.sh
And more strace data around the segfault:
[pid 1819623] nanosleep({tv_sec=0, tv_nsec=20000}, <unfinished ...>
[pid 1819637] write(5, "1629152057.071116 control_write_"..., 103) = 103
strace: Process 1819664 attached
[pid 1819663] <... clone resumed> child_stack=NULL, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7f4a3d224750) = 1819664
[pid 1819637] --- SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x120} ---
[pid 1819623] <... nanosleep resumed> NULL) = 0
[pid 1819663] wait4(-1, <unfinished ...>
[pid 1819664] close(10) = 0
[pid 1819623] futex(0x7f1178, FUTEX_WAIT_PRIVATE, 0, {tv_sec=0, tv_nsec=3615844} <unfinished ...>
[pid 1819664] execve("./snoozy.sh", ["./snoozy.sh"], [/* 42 vars */] <unfinished ...>
[pid 1819663] <... wait4 resumed> 0x7ffd72bfef4c, 0, NULL) = ? ERESTARTSYS (To be restarted if SA_RESTART is set)
[pid 1819664] <... execve resumed> ) = -1 ENOENT (No such file or directory)
[pid 1819637] +++ killed by SIGSEGV +++
I think I have isolated this to a tmux problem with the remain-on-exit flag: https://github.com/tmux/tmux/issues/2828
Thanks for digging into it! Awesome job!
Looks like there is a fix in the works for tmux!
Still waiting for the 3.3 release of tmux - https://github.com/tmux/tmux/issues/2880 any minute now!
tmux 3.3a is out and the issue is gone! People running 3.2 may still hit this - hopefully you are able to upgrade or downgrade tmux from 3.2
If I fire up a bunch of processes, and hit ^C, overmind generally stops the underlying processes and everything is fine.
Beginning with a recent update to tmux (v3.2a), it has started causing tmux to crash, and overmind exits with the error "overmind: Tmux client unexpectidly exited". (the mispelling here was recently fixed, thanks!)
When I strace the processes, I see tmux is segfaulting
This happens on both macOS and Linux machines with the latest versions of tmux. I am unable to reproduce it with versions prior to tmux 3.
I recompiled tmux to force logging, and it looks like the tmux client dies, a new one is spawned, but it can't connect, and then everything dies.
first client:
second client:
server:
I think there is a race condition somewhere, and I am having trouble nailing it down - any help would be appreciated. I'll open an issue over at tmux/tmux if I can reproduce this without overmind (which I have not yet been able to do).