Mosh won't quit on issuing exit command

[ericswpark]

Running into this really weird bug with one particular server. When I'm done with my SSH/mosh sessions I like to run exit to tell the server to "cleanly" break off the connection. On most of my servers this works well. However, on one shared server that I use with others, when I run exit from the shell, it is immediately followed by the output logout, but nothing happens.

The only way to quit mosh in this state is to issue the Ctrl-^ and then the period (.) command to quit mosh on my side (client), which is a bit annoying. Why isn't mosh terminating the connection?

Just to test, I spun up a fresh Ubuntu server and installed mosh, and mosh correctly kills both client and server when I issue the exit command. What could be going wrong?

[ericswpark]

Also, when I run the Ctrl-^ . command, this shows up after a couple of seconds:

mosh did not shut down cleanly. Please note that the
mosh-server process may still be running on the server.
[mosh is exiting.]
➜  ~

This is displayed on the client, and only for that shared server. Why?

[eminence]

On this particular server, what happens if you run "logout" instead of "exit"? Or what happens if you use kill to your current shell? What's the process tree look like between the mosh-server process and your shell?

[ericswpark]

I tried running logout, but the behavior is the same. mosh won't terminate the session until I tell it to with Ctrl-^ ., and when I do run that it shows this output:

[mosh is exiting.]

I don't know what you meant by killing the current shell, so I tried killing the mosh-server process. As soon as I did that (on a separate SSH session), the mosh client on my end exited cleanly ([mosh is exiting.]) and it returned the shell prompt to me as expected.

I'm not sure how to get the process tree? If you can explain how to obtain that I will post it here. Thank you!

[eminence]

When you mosh to a remote server, what generally happens is that mosh-server runs and spawns your interactive shell as a subprocess (bash or zsh or whatever you use). So the process tree looks like:

16420 ?        00:00:00 mosh-server
16421 pts/18   00:00:00  \_ bash

When you type "exit" or "logout" in your shell, the shell process exits. When mosh-server detects this, it will send a message to the mosh-client process asking it to exit, and then mosh-server terminates itself.

So in your scenario, I'm wondering what happens when you run "exit" or "logout":

• Is your shell not exiting?
• Is the child process of mosh-server something other than your shell?
• Is mosh-server not exiting?

In order to see the current process tree, try running ps -u $USER --forest.

If you kill your current shell with kill -9 $$ what happens? By "current shell" I mean the shell that was spawned by mosh-server. ($$ is a special variable in most shells that expands to the PID of the shell)

[ericswpark]

Thank you for the command to display the current process tree! This is what I get if I run the command ps -u $USER --forest (showing only processes related to mosh-server):

25198 ?        00:00:00 mosh-server
25213 pts/25   00:00:00  \_ bash
12048 pts/25   00:00:00      \_ ps

If I kill the current shell with kill -9 $$ the same thing occurs, the console hangs until I run Ctrl-^ .

I checked what the process tree is like after killing the current shell:

25198 ?        00:00:00 mosh-server
25213 ?        00:00:00  \_ bash <defunct>

So it seems like mosh-server is not exiting for some reason. Running kill 25198 cleanly exits mosh-server and correctly terminates the mosh client on my end. Really weird bug.

For reference, this is the mosh version on the remote side:

ideaman924@remote:~$ mosh --version
mosh 1.3.2 [build mosh 1.3.2]
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
ideaman924@remote:~$ mosh-server --version
mosh-server (mosh 1.3.2) [build mosh 1.3.2]
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[td-gonzales]

I'm seeing this as well on os x.

[ericswpark]

I no longer use the server that exhibits this behavior, but I'll keep this issue open as it seems like other people have the same issue.

[kmnlive]

Yes same problem in iPad Pro 12.9.

[achernya]

I'm actually not sure how this works in the general case. mosh-server only detects shutdown by way of receiving a signal (such as SIGTERM) or an error on one of the syscalls on the file descriptors. I see no calls to the wait(2) family of functions, which is why you can possibly get a defunct bash as you have observed. My guess is that there's something in your system that has resulted in the PTY being still alive, preventing the error on https://github.com/mobile-shell/mosh/blob/master/src/frontend/mosh-server.cc#L837-L844 being detected and triggering shutdown.

It seems to me like mosh-server's event loop needs to grow a

  int wstatus = 0;
  // Do we need to handle error returns from waitpid?
  waitpid(the_pid, &wstatus, WNOHANG);
  if (WIFEXITED(wstatus) || WIFSIGNALED(wstatus)) {
    network.start_shutdown();
  }

stanza.