search

rust-lang / rust

Empowering everyone to build reliable and efficient software.
https://www.rust-lang.org
Other
98.13k stars 12.69k forks source link

Tracking issue for musl host toolchain #59302

Open mati865 opened 5 years ago

mati865 commented 5 years ago

TODO:

For people looking how to use as it dynamic target:

RUSTFLAGS="-C target-feature=-crt-static" cargo build
wangbj commented 5 years ago

It seems x86_64-unknown-linux-musl target doesn't honor --enable-default-pie, a different behavior compare to x86_64-unknown-linux-gnu. Is this a known issue?

mati865 commented 5 years ago

@wangbj I don't think Rust supports --enable-default-pie for any target. Could you provide more details?

wangbj commented 5 years ago

hmm, the problem seems caused by musl-gcc wrapper instead of rustc:

details: https://gist.github.com/wangbj/ead5fa8c96418de7c9050bc354fbf353

--enable-default-pie is a gcc configure flag. sorry for the noise. (shortened the text to a gist).

Cogitri commented 5 years ago

FWIW, Void has been hacking around this for some time: https://github.com/void-linux/void-packages/blob/742a7d53eb0b373bf8dad3b51db3380c5cdf8dc1/srcpkgs/rust/patches/musl-dont-use-crt-static.patch and https://github.com/void-linux/void-packages/blob/742a7d53eb0b373bf8dad3b51db3380c5cdf8dc1/srcpkgs/rust/patches/link-musl-dynamically.patch

I'm currently working on this for Alpine too (BTW, maybe I can ping you on the PR for this @mati865 so you can take a look at it?).

mati865 commented 5 years ago

I knew about Alpine and Void. IMO Alpine approach is better because <arch>-unknown-linux-musl remain static by default (it behaves like upstream) and new target <arch>-alpine-linux-musl links dynamically by default.

I had looked at that PR and libc changes will be incompatible with upstream rust behaviour. Official Rust packages when statically linking to musl create executables that all fully static (run basically everywhere), that patch will create static executables that depend on musl runtime (use have to install musl to be able to run them).

Cogitri commented 5 years ago

Yup, but at least on Void we've come to terms that the Rust in the repos is only really meant for packages in the repos, while users interested in rust development should use upstream Rust via rustip.

smaeul commented 5 years ago

I had looked at that PR and libc changes will be incompatible with upstream rust behaviour. Official Rust packages when statically linking to musl create executables that all fully static (run basically everywhere), that patch will create static executables that depend on musl runtime (use have to install musl to be able to run them).

That doesn't make sense. Static executables by definition don't depend on having musl installed. Are you referring to rust-lang/libc#1327? That change would require having a musl-targeting toolchain installed for rustc to link (static) musl-targeting binaries. But any static binaries compiled by rustc would still be totally independent and require no musl installation.

mati865 commented 5 years ago

@smaeul I'm referring to https://github.com/alpinelinux/aports/blob/21c1983ce828ead1afc9e80aa77ede330c662182/community/rust/link-musl-dynamically.patch

smaeul commented 5 years ago

@mati865 Okay, so that patch is misnamed. It doesn't have anything to do with static versus dynamic linking. It makes the same change as rust-lang/libc#1327 -- using the C toolchain's copy of libc.a instead of bundling it inside liblibc.rlib. And my comment above applies to it as well.

Alpine also ships custom targets that use the C toolchain's crt*.o instead of rustc shipping a second copy. Since on Alpine targets rustc's copy of crt*.o are not used, this patch stops rustc from making the copy. (This change is unrelated to the liblibc change.)

mati865 commented 5 years ago

Thanks for the explanation.

thedrow commented 5 years ago

53968 also seems to be related but it's not included in the OP's checklist.

Am I correct?

mati865 commented 5 years ago

Updated

thedrow commented 5 years ago

61328 also seems important.

mati865 commented 5 years ago

I don't see how #61328 relates to this, there is no issue when using target-feature=-crt-static. It applies to musl toolchain in general but it has nothing to do with using musl toolchain natively.

gnzlbg commented 5 years ago

In https://github.com/rust-lang/libc/pull/1327, a #[cfg(target_env = "dyn_musl")] or similar was proposed to support both statically and dynamically linked musl environments. I don't know if that would be a good way to make progress here though.

How do other run-times deal with this ? Is it possible to statically link glibc and others into Rust binaries ? Maybe we need a #[cfg(target_env_linkage = "static/dynamic/...")] system instead of duplicating target_envs per target.

mati865 commented 5 years ago

AFAIK musl is the only target which you can link statically and dynamically at same time.

Duplicating target_env is easy but for me it sounds like a bad workaround because user would have to install 2 toolchains that have exactly the same libs and the only difference comes to target-feature=-crt-static. With static nobundle it would be third 99% identical toolchain.

Ideally they should use single shared sysroot because all that is required can be handled at runtime. I think Rust is not capable of it yet so this would have to go through RFC first.

smaeul commented 5 years ago

How do other run-times deal with this ? Is it possible to statically link glibc and others into Rust binaries ?

The two relevant target attributes are crt_static_respected and crt_static_allows_dylibs.

For glibc (and all other non-musl unix-like platforms), crt_static_respected is false, so the -C target-feature=+crt-static option is ignored by the backend and still dynamically links libc, and therefore dynamic native libraries can still be linked. So the only effect of the option is that it matches #[cfg(target_feature = "crt-static")].

For MSVC, crt_static_respected and crt_static_allows_dylibs are both true, so -C target-feature=+crt-static only affects the C runtime, and other (rust or native) libraries can still be loaded as DLLs. This is because on Windows, there's not really such a thing as a "static binary".

musl is in the unique situation of crt_static_respected being true and crt_static_allows_dylibs being false. Since musl uses ELF (and this would apply to glibc too if it supported static linking), linking libc statically requires all other libraries to also be linked statically. For native libraries, this means #[link(kind = "static")] or #[link(kind = "static-nobundle)]".

rustc currently handles this limitation correctly for rust libraries and direct native dependencies, but not for native libraries that are transitive dependencies of a dependent crate.

I had a patch (#55566) that "fixed" this by changing the semantics of #[link] with no kind in the +crt-static and not crt_static_allows_dylibs case. But the default kind is documented to be dylib, so that change was unacceptable. Instead, we need a patch to produce an error, similar to the non-transitive case here.

Maybe we need a #[cfg(target_env_linkage = "static/dynamic/...")] system instead of duplicating target_envs per target.

We already have that: #[cfg(target_feature = "crt-static")].

The problem in rust-lang/libc#1327 is that we try to support cross-compilation to musl without having the proper sysroot installed. The whole musl_root system is a giant hack to support that, and it breaks in the native case, where we want to use the libc.a that is actually present on the system (and updated by the package manager, etc.). Distributions that ship a native musl rust have to add a patch similar to the one in that PR (specifically e.g. this one for Adélie and Gentoo).

Cogitri commented 5 years ago

I had a patch (#55566) that "fixed" this

Fwiw Alpine Linux ships a similar patch (although we don't use crt-static for pur distro packages). Good to know what's the status on that, thanks.

https://github.com/alpinelinux/aports/blob/master/community/rust/musl-fix-static-linking.patch

Distributions that ship a native musl rust have to add a patch similar to the one in that PR (specifically e.g. this one for Adélie and Gentoo).

Void Linux and Alpine Linux carry a similiar patch: https://github.com/alpinelinux/aports/blob/master/community/rust/link-musl-dynamically.patch

gnzlbg commented 5 years ago

The problem in rust-lang/libc#1327 is that we try to support cross-compilation to musl without having the proper sysroot installed.

Many production users rely on this feature, so we are kind of constrained on not breaking this. If the only way to achieve that is have a second set of targets, then that's the way it is.

smaeul commented 5 years ago

Many production users rely on this feature, so we are kind of constrained on not breaking this. If the only way to achieve that is have a second set of targets, then that's the way it is.

What about shipping libc.a alongside rust, like we already do for crt*.o? Then we could link it static-nobundle from liblibc.rlib instead of bundling it inside. And then if we only add a -L for that directory to the linker args when cross-compiling, it would properly use the system libc.a for the native case as well. (And distro packagers could simply rm rust's copy of libc.a instead of patching).

gnzlbg commented 5 years ago

I think I would prefer something that also works great for distro packages. While they could tune things when packaging, a distro that uses musl as its stdlib, will have users that might install rust via rustup, and they will want their "default" rust target to dynamically link the system musl, instead of doing something else, right?

smaeul commented 5 years ago

Yes. That's why most (all except Void) musl distributions ship distro-specific targets (usually matching their $CHOST) that default to dynamic linking. On the other hand, the "static by default" semantics of the current official musl targets are well documented, and x86_64-unknown-linux-musl is already available in rustup.

The two feasible solutions I can think of are:

  1. Introduce new targets that are dynamic by default – by changing the vendor (e.g. x86_64-dynamic-linux-musl), not by changing the environment (e.g. x86_64-unknown-linux-musldyn), which would break all existing uses of #[cfg(target_env = "musl")].
  2. (At the risk of making the mess worse) override the crt_static_default = true in musl targets when the host environment is also musl, or alternatively change it to crt_static_default = false and override it when cross compiling.
smaeul commented 4 years ago

Linking these here for future reference, more cases where bundling libc.a into liblibc.rlib breaks things: https://github.com/rust-lang/rust/issues/71564 https://github.com/rust-lang/libc/pull/1704

msrd0 commented 3 years ago

Another problem I noticed trying to compile/run the rust compiler on a musl host like Alpine Linux is that using the jemallocator for the compiler itself does not seem to work. Given that rust is known to run slower with musl's default allocator than with jemalloc, I tried enabling jemalloc = true in rustc's config file (rustc version 1.48). However, probably due to the rust compiler only referencing jemalloc, and the jemalloc-sys crate forcing a prefix on musl targets, this seems to have no effect.

Cogitri commented 3 years ago

FWIW Rust recently got a new allocator with musl 1.2 (mallocng), so it might be worth to benchmark that again :) We've been avoiding jemalloc whereever possible on Alpine Linux and even patched it out of some packages, IIRC it doesn't work too well on musl, but maybe that has changed since we last tried using it.

msrd0 commented 3 years ago

I don't know much about jemalloc, I was just playing around to see if it improved stuff and had to realize that the compiler didn't seem to use it. So I don't know if the performance issues I'm seeing are caused by the allocator, but other rust projects like ripgrep have seen the allocator change to cause performance regressions on musl targets (and that one is also not patched on alpine).

Unfortunately, alpine:edge with musl 1.2.2_pre6-r0 and rust 1.47 from the alpine repos is consistently 5 to 10 seconds slower than alpine with musl 1.1.24-r10 and rust 1.47 compiled with a barely modified APKBUILD from aports (that is, enabling clippy and rustfmt tools). So it does not seem like the new allocator changed a great deal. Also, alpine is up to 30% slower compared to the official rust docker image when running cargo build for gotham. My docker host system is ArchLinux in case it matters (which performs the best but that's an unfair comparison).

haraldh commented 2 years ago

Opened a new issue against the musl version used in building https://github.com/rust-lang/rust/issues/91178

yshui commented 1 month ago

is it possible to have dylib/cdylib with +crt-static?

bjorn3 commented 1 month ago

Musl doesn't support that as the dynamic linker is part of libc and it doesn't support multiple copies of libc in a single process like on basically every Unix. The only mainstream system where that is possible is Windows as there the dynamic linker is part of the kernel and libc is not the system abi and multiple libc copies can co-exist in a single process.

yshui commented 1 month ago

i believe it would be possible to create "statically linked" shared library. i.e. all libc symbols are resolved at link time and the shared library is completely self-contained that it neither exposes nor references any libc symbols. we would also need the equivalent of -fno-semantic-interposition etc., and other things i might have missed, but i think it is doable at least in theory.

it might cause some confusion since two symbols ostensibly have the same name can actually be completely unrelated to each other, (which, btw, already happens in rust today if my project happens to use different versions of the same crate), but the shared library should otherwise work.

bjorn3 commented 1 month ago

There are various pieces of global state in libc as well as some process state like the tls register for which each libc instance assumes exclusive access.

yshui commented 1 month ago

Ah right, yeah I forgot about that. That definitely makes this unsupportable from Rust's perspective. There goes my dreams :'(

OTOH it's probably not that bad? I'd imagine process state and tls register setup generally happen before main and probably isn't going to be touched again when a shared library is loaded. I can probably still make it work as a hack.

bjorn3 commented 1 month ago

TLS registering also happens on every dlopen and pthread_create, so don't use either from within your statically linked dylib. And you better make sure that the host program uses musl as libc as errno is stored at a fixed offset from the TLS register, which other libc's may use for other purposes.