search

matrix-org / rust-synapse-compress-state

A tool to compress some state in a Synapse instance's database
https://pypi.org/project/synapse-auto-compressor/
Apache License 2.0
142 stars 32 forks source link

Please provide static builds or static build instructions #82

Open cyphar opened 2 years ago

cyphar commented 2 years ago

As far as I can tell, it is not possible to build this program statically. This seems to mostly be the fault of rust, with the musl target no longer magically letting you compile rust statically, but a tool that is intended to run on a server really should be distributable as a static binary somehow.

It would be great if there were either static build instructions that folks could copy-paste, or the project itself provided static builds of its binaries. There are static builds for the 0.1.0 release but the main branch has the synapse_auto_compressor program which didn't exist ~2 years ago when the last release was made.

After reading lots of blog posts and rustc bug reports, I managed to reach the following command run inside an Alpine Linux container (good luck getting this to work on a glibc-based distribution installation because it will treat the musl target as a cross-compilation target which will then not work with building openssl):

% RUSTFLAGS="-C target-feature=+crt-static -C link-self-contained=yes -L native=/usr/lib -l static=ssl -l static=crypto -l static=c" cargo build --release --target x86_64-alpine-linux-musl

(The --target is necessary for the reasons mentioned in https://github.com/rust-lang/rust/issues/78210 -- basically it's because proc-macros don't get the RUSTFLAGS applied unless you explicitly set the --target -- even if the target is the same as the host.)

In addition, I had to patch out the jemalloc requirement because I couldn't figure out how to get it to build (I'm a little confused why the compressor binary needs to use jemalloc -- surely for a tool which runs once in a while you don't need a super optimised memory allocator?).

However the resulting binary is still dynamically linked, it's just that every library except the link loader are statically linked into the binary. This is a problem because the loader is /lib/ld-musl which won't work for my glibc-based server. In addition, the binary itself segfaults even if you try to run it on alpine -- it seems that one or all of the RUSTFLAGS are causing issues (it seems that even if you only use -C target-feature=+crt-static you get segfaulting binaries).

If you can offer some kind of script or anything that would help me solve this problem I would seriously appreciate it.

cyphar commented 2 years ago

Okay, I managed to figure out how to do it. My distribution (openSUSE) doesn't provide static OpenSSL libraries (they argue it's a security risk to allow OpenSSL to be embedded in this manner). So in order to fix this, you need to enable the vendor feature for openssl-sys:

diff --git a/synapse_auto_compressor/Cargo.toml b/synapse_auto_compressor/Cargo.toml
index dbcd175c26db..c02f766d6b0d 100644
--- a/synapse_auto_compressor/Cargo.toml
+++ b/synapse_auto_compressor/Cargo.toml
@@ -34,3 +34,6 @@ crate-type = ["cdylib", "rlib"]
 [dependencies.pyo3]
 version = "0.14.1"
 features = ["extension-module","abi3-py36"]
+
+[dependencies.openssl-sys]
+features = ["vendored"]

And then build like this:

% OPENSSL_STATIC=1 RUSTFLAGS='-C target-feature=+crt-static' cargo build --target x86_64-unknown-linux-gnu --release