Mosh binary distribution needed to avoid dependency-hell

[xapple]

A very common use-case for ssh or mosh, is when you want to connect to a large server or cluster that is provided by a university or a company and you are not sudo on the remote server. You need to install everything in your home directory. Sadly, apt-get or yum are of strictly no use in such a case. One of the biggest glaring flaws of linux distributions IMO.

Mosh is notoriously difficult to install because of its multiple dependencies. For instance, it depends on protobuff, which itself depends on libtool and soon you find yourself battling with the autogen.sh step of protobuff to point it to the libtool install location with LD_LIBRARY_PATH variables to no avail.

Could we not have a simple static binary distribution of mosh, mosh-client and mosh-server for linux 64 bit systems? mosh is currently uninstallable.

[xtaran]

you are not sudo on the remote server

You can't "be sudo". sudo is a program, not a user.

Sadly, apt-get or yum are of strictly no use in such a case.

That's wrong. There are wrapper around apt-get, e.g. uapt-get which installs everything somewhere under your home directory.

One of the biggest glaring flaws of linux distributions IMO.

That depends a lot on the view. IMO proper dependencies and not having to install a library x times in different versions¹ is one of the best features of Linux distributions. The fact that Windows doesn't have such a dependency system is what causes DLL hell there.

¹) This also makes tracking security issues horribly complicated. A common issue of most container systems as commonly used with distribution-independent "packaging".

Could we not have a simple static binary distribution of mosh, mosh-client and mosh-server for linux 64 bit systems?

Do you really want to trust arbitrary binaries someone compiled somewhere and uploaded them somewhere and you can't check at all if they were compiled from the claimed source code or contain a backdoor? (Unless the binary is build in an environment with exactly declared dependencies—as common with Linux distributions—and a reproducibly building source code, you have nearly no chance to check that. Luckily mosh already builds reproducibly. So one step is already done.)

Sorry for the rant, but I can't let "One of the biggest glaring flaws of linux distributions" stand without a comment.

[xapple]

I like rants! And would enjoy continuing the conversation, but at the same time it's the GitHub issue tracker so I feel I should stick to the parts relevant to mosh. What's your policy on discussing linux design philosophy and shell etymology in a ticket?

Downloading a static binary distribution from https://mosh.org/ has the same trust implications as downloading the source code from https://mosh.org/mosh-1.3.2.tar.gz and compiling it. So no difference there as to security issues. There would be a difference in the UX and usability perspective though.

[xtaran]

I like rants!

:-)

I feel I should stick to the parts relevant to mosh.

Granted. Will try. ;-)

Downloading a static binary distribution from https://mosh.org/ has the same trust implications as downloading the source code from https://mosh.org/mosh-1.3.2.tar.gz and compiling it.

Good point. Basically the compiled binaries should be digitally signed as source code should be, too. (e.g. with GPG).

Just noticed that mosh only provides signed release-announcements which include SHA-256 checksums on the source-code and macOS binaries. While that's a good start, it does not make the source code machine-verifiable with standard tools like gpgv. Providing a detached signature file would be helpful here. I've opened #1008 for that.

Since the macOS binaries are already signed that way, I assume that any future binaries for other platforms will be signed at least that way, too.

\Nevertheless it's more tricky to compare compiled binaries against e.g. a checkout of the git repository than comparing a tar ball with a git checkout. (E.g. you might trust the signatures, but maybe not the build environment. You wouldn't notice a compromised compiler on the developer's machine just by digital signatures of the developer.)\

Then again, there seem to be efforts to make application container images reproducible, too. Can't find a reference on a first glance, though.

[javabrett]

Static compilation is tricky and fraught with traps and issues, to varying degrees depending on the target-platform, and the number of platforms you want to support.

This issue overlaps/duplicates a number of the discussion in #188.

For mine, I think simplifying the build-chain (and docs) so that anyone can create a reliable mosh-server build for any target platform, and ship that runnable to a remote-server without root access, is the way to go. Probably want to build in Docker to avoid cross-compilation complexities and dev-env set-up chores.

[deutrino]

I'm on a train with network handoffs happening on my mobile a few times an hour, and I don't care about the finer points of Linux theory, I just want to be able to drop a mosh binary on my shared web host so I can do my work.

[renatofrota]

@xapple

It is easier to deploy mosh-server to a remote server where you are not root/sudoer than you (and everybody else commenting on this issue) think.

1. Install mosh on your machine (or any other where you are root or sudoer - maybe a cheap hourly-charged VPS?)
2. Connect to remote server using ssh, create directories lib and bin, then exit: mkdir -p ~/lib; mkdir -p ~/bin; exit
3. Copy mosh-server binary to remote server scp /usr/bin/mosh-server remoteuser@remotehost:bin/
4. Create an alias: alias mosh2='mosh --server="export LD_LIBRARY_PATH=~/lib && ~/bin/mosh-server" '
5. Connect using mosh2 remoteuser@remotehost

If your system and the remote are the same O.S. (or from same family) the chances it works just by doing this are very high.

When the remote server does not have one of the dependencies (or the binary you uploaded is not compatible to the remote severs' O.S.) the connection will fail, displaying an error like this: /home/remoteuser/mosh-server: error while loading shared libraries: libprotobuf.so.10: cannot open shared object file: No such file or directory

In such cases, you can find the corresponding lib files on your own system and copy to the remote server respecting the exact version number the mosh-server binary is looking after:

scp $(find /usr -type f -name libprotobuf.so* | sort -n | tail -1) remotehost:lib/whatever-name-missing.so.specific.version

Of course, installing the exact version may be necessary depending on the library. But give it a try (you are copying the library to a /lib dir under your own homedir, there's no chance it will affect the server in any form, the worst case mosh won't connect).

The most problematic case I have seen so far is CentOS which is missing a lot of required libraries, so to save your time (and anybody else needing this - maybe even myself in the future) I have prepared 2 packages with mosh-server binary I compiled for CentOS 7 + the libprotobuf version it requires.

One package has mosh-server 1.3.2 (latest stable release) and the libprotobuf available at that time and the other has current github build (1.3.2-90-g03087e7) and latest libprotobuf release available as of today. A readme.txt with clear instructions on how to "install" && use is included on each.

Links:

mosh-server-1.3.2-libprotobuf-8.tar mosh-server-1.3.2-90-g03087e7-libprotobuf-23.tar

If you need mosh working on any server other than CentOS 7 you are not root/sudoer, you can follow this guide or install/compile on other server using the same O.S./version (if build tools are available you won't even need to be root/sudoer).

[xkortex]

I, too, would enjoy a static mosh binary. mosh-server seems like an ideal candidate to be delivered as a static bin. I'm trying to bootstrap on a system where I am not sudo, ahem, I'm not in the sudoers group / lack privileged access. Alright, I can compile mosh, oops I need libprotobuf. Ok, I can compile libproto...oh I don't even have the luxury of the libtool battle xapple ran into because I don't even have autoreconf.

IMHO, The "arbitrary binary" issue is kind of a red herring, not just because of signing, but also I can just compile on system A where I do have deps, and copy it wherever I want. Or, y'know, docker.

I could probably get renatofrota's workaround to work, looks like I at least have liboprotobuf.so, though it's the totally wrong version.

[Deathlok666]

I, too, would enjoy a static mosh binary. mosh-server seems like an ideal candidate to be delivered as a static bin. I'm trying to bootstrap on a system where I am not sudo, ahem, I'm not in the sudoers group / lack privileged access. Alright, I can compile mosh, oops I need libprotobuf. Ok, I can compile libproto...oh I don't even have the luxury of the libtool battle xapple ran into because I don't even have autoreconf.

IMHO, The "arbitrary binary" issue is kind of a red herring, not just because of signing, but also I can just compile on system A where I do have deps, and copy it wherever I want. Or, y'know, docker.

I could probably get renatofrota's workaround to work, looks like I at least have liboprotobuf.so, though it's the totally wrong version.

Dealing with static linking stuff these two days, I finally managed to fully static link (with Alpine Linux, which ships with musl-libc) 3 ssh related software referenced in ArchWiki: Secure Shell: dropbear,tinyssh,openssh-portable. The main reason why I did that is just for fun, and a little digging into static linking, which is quite charming for me these days because I am a Termux user and statically linked binaries (usually Go's build) usually work like a charm, with no flaws at all. However, if I extract a statically linked binary from a deb package (of course target architecture matches), for example, extract qemu-aarch64-static from the deb package qemu-user-static, then copy it to Termux, chances are that it won't work as expect (errors that I've got the most are Bad system call and invalid system call). I managed to bypass it by using proot -0 ....

The reason why I mentioned all these is that during these 2 days, I realized how tricky static linking is (the difficulty mainly lies in handling all kinds of dependencies, and all of the 3 above is wriiten in C, no shortcut like Go build ), although fully statically linked binaries are very portable and environment-independent. Given my experience in a non-standard Linux environment (which is native Termux, not its proot distros), I think if static linking is very difficult, then AppImage is probably the best option. Because I always want to use FUSE in Termux (without rooting)... and AppImages are always doing a great job in regard to simplicity and portability, although it is not static linked. And the most import one, /dev/fuse is not a luxuxy for a modern Linux distro. So I think it deserves a go.

But the downside is that, AppImage mainly supports x86 processors, which is a shame. If cross architecture is essential, then probably AppImage is not quite a good option.

For those of you that are interested, in order to port a package into Termux, it always involves some kind of patchwork for Termux package maintainers to do. See here. I've looked into build scripts and patches of dropbear, and changes mostly focus on path and authentication, where Termux is different from standard Linux environment.

[dtinth]

Thanks to @javabrett’s instructions back in 2017, I created a static x86-64 binary distribution, automated by GitHub Actions

→ dtinth/mosh-static

This GitHub Actions workflow clones mosh-1.3.2, compiles it inside an Alpine Docker image, and creates a GitHub release with the binary.

Here’s how to use it:

# On the server
wget https://github.com/dtinth/mosh-static/releases/latest/download/mosh-server
chmod +x mosh-server

# On the client
mosh --server=./mosh-server <username>@<hostname>

(If you don’t trust my binaries, you can fork the repo, then trigger a GitHub Actions build. The workflow will then release the binaries in your own repo, which you can then use.)

[xonq]

Thanks to @javabrett’s instructions back in 2017, I created a static x86-64 binary distribution, automated by GitHub Actions

→ dtinth/mosh-static

This GitHub Actions workflow clones mosh-1.3.2, compiles it inside an Alpine Docker image, and creates a GitHub release with the binary.

Here’s how to use it:

# On the server
wget https://github.com/dtinth/mosh-static/releases/latest/download/mosh-server
chmod +x mosh-server

# On the client
mosh --server=./mosh-server <username>@<hostname>

(If you don’t trust my binaries, you can fork the repo, then trigger a GitHub Actions build. The workflow will then release the binaries in your own repo, which you can then use.)

thanks for this, works great. I think this deserves wider mention (perhaps on mosh install guide) as it took quite a while and a lot of failure to find.

[kellpossible]

@dtinth you are a legend!

[carloscabanero]

We have taken the torch from the work on this thread. We forked @dtinth scripts and expanded it to support multiple architectures and platforms, including Linux with Armv7 and Arm64 (for Raspberry Pi) using QEMU in the GitHub workflow, and Darwin for both x86_64 and Arm64. You can find the workflows and binaries here: https://github.com/blinksh/mosh-static-multiarch/

Any ideas for how to improve it, testing in more platforms, etc... are greatly appreciated. I have even seen requests to make mosh installation more straightforward in weird routers over Reddit, so will try to jump on those as well.

In Blink we are now downloading the client and then uploading it to the remote, as we have deep control of SSH and SFTP as a library. But, talking to people some would prefer to just have a script they can run on the server themselves to trigger the flow on their own. We are working on that as well, and maybe something like that could be adapted to the regular mosh-client?

Hope this is useful!