Is this far enough along yet that it can make a "Linux distro"?

[brandonros]

Bootloader -> kernel -> userspace where init is... Busybox shell?

So I guess this becomes, can the kernel and Busybox compile with this? I think you can build kernel with musl (I might have it wrong that you need libc to compile the kernel at all)

[sunfishcode]

It can run coreutils, but I'm not aware of anyone having tried a shell yet. I wouldn't be surprised if it's missing a few things needed for a shell yet, but I'd guess it's close.

(As you suspected, an OS kernel like Linux does not use a libc, so c-ward wouldn't be able to help there.)

[nrabulinski]

I have tried it (partially) and while I could get the very primitive https://github.com/nrabulinski/rush to compile, I couldn't get anything more sophisticated (but I also didn't try really hard and it was a while ago so take that with a grain of salt)

[sunfishcode]

I can also add here that c-ward has so far focused mostly on the parts of libc needed to run Rust programs, as well as a few important low-level C libraries like libgit2. This means while it's complete enough to run many whole applications, it's still missing some really basic C features like fopen and the whole FILE API, qsort, scanf, and others. A typical Linux distro has a lot of C code, so we're probably not ready for that yet.

[brandonros]

@nrabulinski when you build rush how do you tell it to link (statically?) against c-ward instead of system libc?

Working on setting up a little proof of concept for fun...

# start clean kubernetes pod
kubectl run -n default alpine --restart=Never --rm -i --tty --image=alpine:3.18 --overrides='{"spec": {"containers": [{"name": "alpine", "image": "alpine:3.18", "command": ["/bin/sh"], "stdin": true, "tty": true, "securityContext": {"privileged": true}}]}}' -- /bin/sh
# install dependencies to build kernel
apk update && apk add bash curl make gcc musl-dev linux-headers g++ build-base ncurses-dev openssl-dev elfutils-dev bc diffutils perl
# change shells
/bin/bash
# change directories
cd /root
# pull + extract sources
curl -O https://cdn.kernel.org/pub/linux/kernel/v6.x/linux-6.5.9.tar.xz &&
tar -xf linux-6.5.9.tar.xz &&
rm linux-6.5.9.tar.xz &&
cd linux-6.5.9
# build config
make defconfig ARCH=x86_64 V=1
# build bzImage
make -j$(nproc) bzImage ARCH=x86_64 V=1
# download kernel result locally
kubectl cp default/alpine:/root/linux-6.5.9/arch/x86/boot/bzImage ./bzImage
# install rust + dependencies
apk update && apk add curl git build-base
curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain nightly -y
source "$HOME/.cargo/env"
rustup component add rust-src --toolchain nightly
# clone repos
cd /root
git clone https://github.com/sunfishcode/mustang.git
git clone https://github.com/nrabulinski/rush.git
# build rush
export RUST_TARGET_PATH="$PWD/mustang/target-specs"
cd rush && cargo +nightly build -Z build-std --target=x86_64-mustang-linux-gnu && cd ..
cp rush/target/debug/rush /mnt
umount /mnt
# get qemu disk ready
apk update && apk add qemu-img e2fsprogs
qemu-img create -f qcow2 mydisk.img 10G
mknod /dev/loop0 b 7 0
losetup /dev/loop0 mydisk.img
mkfs.ext4 /dev/loop0
losetup -d /dev/loop0
mount -o loop mydisk.img /mnt
# boot
qemu-system-x86_64 -kernel bzImage -append "console=ttyS0 root=/dev/sda1 init=/rush" -hda mydisk.img -nographic

[sunfishcode]

@brandonros The easiest way to do that would be to use Eyra.

[brandonros]

Not sure how @nrabulinski got it to compile. I made a fork here and added Eyra: https://github.com/nrabulinski/rush/pull/1/files (attached build errors in PR description)

Not sure if these errors are expected:

error[E0425]: cannot find function `statx` in crate `libc`
error[E0425]: cannot find value `PTHREAD_KEYS_MAX` in this scope
error[E0425]: cannot find value `statx` in crate `libc`
error[E0425]: cannot find function `getentropy` in crate `libc`
error[E0308]: mismatched types
error[E0609]: no field `c_ispeed` on type `&libc::termios`
error[E0609]: no field `c_ospeed` on type `&libc::termios`
error[E0560]: struct `libc::termios` has no field named `c_ispeed`
error[E0560]: struct `libc::termios` has no field named `c_ospeed`
error[E0061]: this method takes 1 argument but 0 arguments were supplied
error[E0512]: cannot transmute between types of different sizes, or dependently-sized types

[sunfishcode]

Ah, this is using Alpine Linux, which uses musl as its libc, so this is the -unknown-linux-musl target, and it appears it doesn't have anything that c-scape/c-gull are expecting. Would it be feasible to build with -unknown-linux-gnu target? The resulting binaries should still work on Alpine, since they don't depend on the system libc.

[brandonros]

Would it be feasible to build with *-unknown-linux-gnu target?

Absolutely, I'll give it a shot. Thanks for your help so far.

The resulting binaries should still work on Alpine, since they don't depend on the system libc.

I switched to using mustang (which I realize is also your project... very nice...) and resulted in this:

# apk update && apk add curl git build-base
# curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain nightly -y
# source "$HOME/.cargo/env"
# rustup component add rust-src --toolchain nightly
# cd /root
# git clone https://github.com/sunfishcode/mustang.git
# git clone https://github.com/nrabulinski/rush.git
# export RUST_TARGET_PATH="$PWD/mustang/target-specs"
# cd rush && cargo +nightly build  -Z build-std --target=x86_64-mustang-linux-gnu && cd ..
~ # ldd rush/target/x86_64-mustang-linux-gnu/debug/rush
  /lib/ld-musl-x86_64.so.1 (0x7f93c1afe000)

Do you happen to know what I should be passing to cargo for the final binary to not link dynamically against /lib/ld-musl-x86_64.so.1 or is it not really an option?

[brandonros]

Ok, cool. Mustang + Rush (without Eyra) worked good and is statically linked by default. We can chalk up the non-static as more musl weirdness I guess (I understand if it was never coded to run against musl we shouldn't expect it to "just work" :))

root@alpine:~# ldd rush/target/x86_64-mustang-linux-gnu/debug/rush
    statically linked

Should I try Rush with Eyra and not Mustang? Not sure how different it would come out?

Edit: I was wrong.

root@debian2:~# file rush/target/x86_64-mustang-linux-gnu/debug/rush
rush/target/x86_64-mustang-linux-gnu/debug/rush: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=264b9fea717ae88b554fefeb2838953d27122720, with debug_info, not stripped
root@debian2:~# ldd rush/target/x86_64-mustang-linux-gnu/debug/rush
    statically linked

I need to figure out how to get it to not want /lib64/ld-linux-x86-64.so.2

[sunfishcode]

Eyra and Mustang are essentially the same thing at this point, just two different "user interfaces" for all the same underlying libraries. I find Eyra a little easier to use, myself, but you can use whichever works.

On that ld-musl-x86_64.so.1/ld-linux-x86-64.so.2 issue, another way to fix that might be to build with RUSTFLAGS=-C target-feature=+crt-static -C relocation-model=static. See here for more details.

(I should probably copy those docs about static linking into Eyra's and Mustang's README.md files to make them easier to find.)

[brandonros]

Pretty neat. Works good.

Documented it all here: https://gist.github.com/brandonros/b840f32a4e636b9679b3107f358777c0

Guess we can close this ticket. Hopefully somebody in the future benefits from it :)

Only actionable thing I can think of is... what would it take for Mustang/Eyra to support musl... and really... what's the benefit/reason I guess? I could make an issue over on one of those repos if you'd like.

Thanks again for the help.

[sunfishcode]

To get Mustang/Eyra to support the musl ABI would probably mostly just be adding cfg(target_env = "musl") etc. checks in various places. Oh, and figuring out some alternative approach to initializing the command-line args and environment variables, since that currently relies on the glibc extension of passing argc/argv/envp to .init_array functions.

I myself don't have an immediate benefit/reason for that, but it might be interesting to hear from anyone who does.