Mosh server - Did not find mosh server startup message

[simonwjackson]

I'm having an issue connecting to mosh (Laptop -> nix-on-droid) but ssh works without issue. Here is what I've tried so far.

Investigation

SSH connects:

ssh nix-on-droid@usu -p 2222

-bash-5.2$ # connected to nix-on-droid

Mosh fails to connect:

mosh --ssh="ssh -p 2222" nix-on-droid@usu

bash: line 1: mosh-server: command not found
Connection to usu closed.
/nix/store/laiqnmhkmdvc8ia0bk1gf2j26d0xapx4-mosh-1.4.0/bin/.mosh-wrapped: Did not find mosh server startup message. (Have you installed mosh on your server?)

Mosh appears to be in my path:

-bash-5.2$ echo $PATH
/data/data/com.termux.nix/files/home/.nix-profile/bin:/usr/bin:/bin:/usr/sbin:/sbin:/nix/store/3fjf1amhwdnd0cxq9rd05g4g2w0z3nc1-openssh-9.3p2/bin

-bash-5.2$ mosh-server

MOSH CONNECT 60002 RJWOu3vO4H/Hdb5cScBlGw

mosh-server (mosh 1.4.0) [build mosh 1.4.0]
Copyright 2012 Keith Winstein <mosh-devel@mit.edu>
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

[mosh-server detached, pid = 421]

Where is the mosh binary?

-bash-5.2$ nix run 'nixpkgs#unixtools.whereis' -- mosh-server
mosh-server: /nix/store/cm83iwwvjnyqpdvk9d4w8mvc84i4hpxs-nix-on-droid-path/bin/mosh-server

Workaround

Mosh connects with full path to mosh-server:

mosh --server=/nix/store/icjmqd2nyrbxnzb7j7c3k3846k716x9l-nix-on-droid-path/bin/mosh-server --ssh="ssh -p 2222" nix-on-droid@usu

-bash-5.2$ # connected to nix-on-droid

Mosh connects with full path to symbolic link (in path) of mosh-server:

mosh --server=mosh --server=/data/data/com.termux.nix/files/home/.nix-profile/bin/mosh-server --ssh="ssh -p 2222" nix-on-droid@usu

-bash-5.2$ # connected to nix-on-droid

Client (laptop) configs

~/.ssh/config (client, laptop)

Host *
    SendEnv TZ LANG LC_*

Nix-on-droid configs

~/.ssh/sshd_config (nix-on-droid)

HostKey /data/data/com.termux.nix/files/home/.ssh/ssh_host_rsa_key
Port 2222
ListenAddress 100.108.246.149
AcceptEnv LANG LC_*

.bash_profile (generated)

-bash-5.2$ cat .bash_profile 
# include .profile if it exists
[[ -f ~/.profile ]] && . ~/.profile

# include .bashrc if it exists
[[ -f ~/.bashrc ]] && . ~/.bashrc

.profile (generated)

. "/etc/profiles/per-user/nix-on-droid/etc/profile.d/hm-session-vars.sh"

.bashrc (generated)

# Commands that should be applied only for interactive shells.
[[ $- == *i* ]] || return

HISTFILESIZE=100000
HISTSIZE=10000

shopt -s histappend
shopt -s checkwinsize
shopt -s extglob
shopt -s globstar
shopt -s checkjobs

if [[ ! -v BASH_COMPLETION_VERSINFO ]]; then
  . "/nix/store/pprx06iqv01sxqd4qikz3z5rb1qg7i66-bash-completion-2.11/etc/profile.d/bash_completion.sh"
fi

[t184256]

-bash-5.2$ echo $PATH
/data/data/com.termux.nix/files/home/.nix-profile/bin:/usr/bin:/bin:/usr/sbin:/sbin:/nix/store/3fjf1amhwdnd0cxq9rd05g4g2w0z3nc1-openssh-9.3p2/bin

Is that $PATH in the app or after ssh'ing in? Do they differ? How was SSH configured?