Busybox contaminated with i686 instructions from glibc

[papppmac]

I'm not sure what platforms you intend to target, but it looks like there are some more-modern-x86 instructions leaking in to your busybox build despite building the kernel and busybox itself for 486.

Just for fun I tried booting my Thinkpad 560E (Pentium MMX 166MHz, 80MB of RAM). It's usually my OpenStep box, but I used old tomsrtbt and NetBSD5.1 images while I was setting it up, so the idea of a more-modern *nix floppy that could boot it appealed. Booted from your disc images, it's trapping an illegal instruction in init, I tried both posted versions just for contrast.

Immediately following "Run /sbin/init as init process" I'm getting a

0.1.0: traps: init[1] trap invalid opcode ip:80c1a19 sp:bfb17128 error:0 in busybox[8049000+db000]
0.2.1: traps: init[1] trap invalid opcode ip:80c60a9 sp:bfdf3c88 error:0 in busybox[8049000+ec000]

I see that you have the usual /sbin/init is a link to busybox situation, and are inheriting busybox's default arch/i386/Makefile "-march=i486 -mtune=i386" setting, but my theory is that the busybox build is statically linking to your system glibc wherever you're doing the build, which must be built for something later (maybe -march=i686 or so).

I don't have a machine/VM with a recent version but built for earlier x86 variant glibc on hand to test the theory, and haven't decided if I want to burn an afternoon making one.

[w84death]

I was having troubles compiling on x64 machine that's why I moved to my older laptop with 32bit CPU. That problem was exactly what you described. I will look into this so we don't need to use VM or old hardware.

But any clues from you will be very helpful.

[papppmac]

It's been some years since I've done this, but thinking about it, glibc's build system knows how to differentiate ‘--build=build-system’ and ‘--host=host-system’ so it should be possible to just build an older x86 glibc by passing eg. --host=i486-pc-linux-gnu. It'll take a little bit of path munging to build busybox with the host compiler but a mutant library path, since modern gcc on an x86_64 host is perfectly capable of generating for -march=i486 or whatever, but I think that's just environment variable hackery.

Or, since now we're having to build a libc anyway, potentially shave some bytes with one of the lighter libcs like musl or uclibc.

When I was last dealing with provisioning weird clusters I was fairly regularly spinning miniroots for cross environments, it looks like the warewulf folks continue to do things that way, so we should be able to hack something like this to do what we want.

[papppmac]

I had a couple hours today and got obsessed.

I started screwing around with trying to do it with the system toolchain which totally should be able to do proper 32 bit binaries with adequate applications of -m32, and I did succeed in building a musl suitable for static linking that way, but the paths got messy trying to build BusyBox linked to said musl, so I took a lazier approach and used an existing cross toolchain.

The magic easy-out is the pre-built musl based cross toolchains provided by https://musl.cc/

Assume your working directory is $BASE, in your guide that would mean BASE=~/my-linux-distro/ wget https://musl.cc/i486-linux-musl-cross.tgz tar xvf i486-linux-musl-cross.tgz

Now into the busybox directory configured as your guide suggests and edit some make options in .config to use the i486-linux-musl-cross environment we just grabbed (the sysroot might be redundant, it also might be possible to do inside of menuconfig, I'm too lazy to convert this into sed scripts right now): CONFIG_CROSS_COMPILER_PREFIX="$BASE/i486-linux-musl-cross/bin/i486-linux-musl-" CONFIG_SYSROOT="$BASE/i486-linux-musl-cross/" CONFIG_EXTRA_CFLAGS="-I$BASE/i486-linux-musl-cross/include" CONFIG_EXTRA_LDFLAGS="-L$BASE/i486-linux-musl-cross/lib "

Also, make sure you turn on LFS (Large File Support) in your busybox configuration, modern versions apparently won't actually build without under some toolchains (throws a nasty "error: size of array ‘BUG_off_t_size_is_misdetected’ is negative" if not) - I assume you had it on and either I missed or it didn't make it into your docs. It builds successfully.

I also did two other things that are "unrelated issues" when building my disc Use XZ Only Since we're going for small, we only need to support the smallest compression scheme, on the Linux build General setup -> Initial RAM filesystem and RAM disk (initramfs/initrd) support select "Support initial ramdisk/ramfs compressed using xz" then when you build your initrd do a find . | cpio -H newc -o | xz --check=crc32 > ../rootfs.cpio.xz (The --check=crc32 IS required to make a valid initrd) instead of a gzip

and use that xz file for the rest, eg. qemu-system-i386 -kernel bzImage -initrd rootfs.cpio.xz

the change in compression is pretty negligible at this scale, though every byte counts and apparently some of my kernel config changes were a little more substantial, and not having the other compression algorithms in there might be a big part of it. Same directory, different compression: ls -lh rootfs.cpio. -rw-r--r-- 1 pappp pappp 155K 2021-05-27 18:40 rootfs.cpio.gz -rw-r--r-- 1 pappp pappp 140K 2021-05-27 19:04 rootfs.cpio.xz My kernel and yours: ls -lh bzImage -rw-r--r-- 1 pappp pappp 593K 2021-05-27 18:55 bzImage -rw-r--r-- 1 pappp pappp 632K 2021-05-27 11:21 bzImage.upstream

df on the loop mount

Filesystem 1K-blocks Used Available Use% Mounted on /dev/loop0 1424 909 515 64% /mnt

Your Syslinux appears from nowhere You assumed in your directions the host system has syslinux, which may or may not be true on any given host. I lazily pulled syslinux and mtools in with pacman on the host I was working on, but it's a hole in otherwise excellent instructions.

With those changes I've built a floppy that will in fact boot not just qemu-system-i386, but my test Pentium MMX machine.
I threw a copy of the image up at http://pappp.net/misc/floppinux_i486.img

[w84death]

Wow! Thanks a lot for those instructions.

I did your method and I got 32bit BusyBox! Linux boots but I'm struggling with:

I will look into this more in upcoming days.

BTW: can I use your photo in my blog about FLOPPINUX in the wild? I will credit you with link to you git profile or anything you want else. If yes can you add what thinkpad it is and CPU speed/RAM size?

[papppmac]

Huh, I got an init error while I was working it, but it was just because I typo'd a couple places in /etc and/or couldn't mount the initrd because I mismatched compression options, didn't take too much hunting to find.

You're absolutely welcome to use that picture just credit to pappp linked to my vanity domain. Speaking of images and vanity domain, I swapped out the the floppy image I posted with one that has more generous kernel and busybox options to make it a little more useful. I spent a few minutes trying to add tcc just for fun, but haven't got the cross compile and library munging right.

That specific machine is a ThinkPad 560E model 2642-40U ca. 1997. It contains a 166MHz Pentium MMX and 80MB of RAM. Described when I got it , installed OpenStep 4.2 on it and finally got that setup online. The only older x86 machines I have easy access to are 486s that have mutant firmware that basically require you chainload with loadlin from DOS.

[w84death]

Thanks!

New instruction for 64-bit systems at https://bits.p1x.in/how-to-build-32-bit-floppinux-on-a-64-bit-os/