building a rootfs without a kernel

[tianon]

I've been using make root BUILTIN=1 to build just a rootfs (for use in containers, which I maintain builds of at https://hub.docker.com/r/tianon/toybox; easier to browse at https://oci.dag.dev/?repo=tianon/toybox). Because containers run directly on the host kernel, there's no need for this to contain a kernel (very similar to a chroot use case, really). :smile:

As of the new 0.8.11 that is no longer working, with cryptic cpio errors. I think the refactoring in https://github.com/landley/toybox/commit/97b776b705ea26378f58ce3258a021c8bcf4a3ab is what made that stop working, but I'm not 100% sure. :thinking:

Perhaps what I'm trying to do is actually an unsupported use case? I'm very open to suggestions here. :smile: :bow: :heart:

Output from the `Dockerfile` as-is (with the 0.8.11 bump):

```console Step 6/9 : RUN make root BUILTIN=1 ---> Running in 62fc88de7ac7 mkroot/mkroot.sh -- BUILTIN=1 cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \ -DPROJECT_NAME=\"ToyBox\" scripts/genconfig.sh kconfig/conf -D /dev/null Config.in > /dev/null logpath:generated/{Config.in,newtoys.h,flags.h,tags.h,help.h} Compile /toybox/root/build/record-commands/logpath ................... find: invalid argument 'f,l' to '-type' find: invalid argument 'f,l' to '-type' find: invalid argument 'f,l' to '-type' find: invalid argument 'f,l' to '-type' find: invalid argument 'f,l' to '-type' find: invalid argument 'f,l' to '-type' Building for host mkroot/mkroot.sh: line 157: toybox: command not found mkroot/mkroot.sh: line 20: /dev/tty: No such device or address === toybox cleaned cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \ -DPROJECT_NAME=\"ToyBox\" scripts/genconfig.sh kconfig/conf -D /dev/fd/63 Config.in > /dev/null scripts/make.sh warning: using unfinished code from toys/pending generated/{Config.in,newtoys.h,flags.h,tags.h,help.h} Compile toybox ............................................................................................................................................................................................................... scripts/install.sh --long --symlink --force Compile instlist... Install commands... No $LINUX directory, kernel build skipped. mkroot/mkroot.sh: line 20: /dev/tty: No such device or address === initramfs mkroot/mkroot.sh: line 3[71](https://github.com/tianon/dockerfiles/actions/runs/8623651182/job/23637247602?pr=678#step:5:72): /toybox/root/host/docs/initramfs.cpio.gz: No such file or directory find: unrecognized: -printf cpio: unknown user/group +0:+0 BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary. Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS] Search for files and perform actions on them. First failed action stops processing of current file. Defaults: PATH is current directory, action is '-print' -L,-follow Follow symlinks -H ...on command line only -xdev Don't descend directories on other filesystems -maxdepth N Descend at most N levels. -maxdepth 0 applies actions to command line arguments only -mindepth N Don't act on first N levels -depth Act on directory *after* traversing it Actions: ( ACTIONS ) Group actions for -o / -a ! ACT Invert ACT's success/failure ACT1 [-a] ACT2 If ACT1 fails, stop, else do ACT2 ACT1 -o ACT2 If ACT1 succeeds, stop, else do ACT2 Note: -a has higher priority than -o -name PATTERN Match file name (w/o directory name) to PATTERN -iname PATTERN Case insensitive -name -path PATTERN Match path to PATTERN -ipath PATTERN Case insensitive -path -regex PATTERN Match path to regex PATTERN -type X File type is X (one of: f,d,l,b,c,s,p) -executable File is executable -perm MASK At least one mask bit (+MASK), all bits (-MASK), or exactly MASK bits are set in file's mode -mtime DAYS mtime is greater than (+N), less than (-N), or exactly N days in the past -atime DAYS atime +N/-N/N days in the past -ctime DAYS ctime +N/-N/N days in the past -mmin MINS mtime is greater than (+N), less than (-N), or exactly N minutes in the past -newer FILE mtime is more recent than FILE's -inum N File has inode number N -user NAME/ID File is owned by given user -group NAME/ID File is owned by given group -size N[bck] File size is N (c:bytes,k:kbytes,b:512 bytes(def.)) +/-N: file size is bigger/smaller than N -links N Number of links is greater than (+N), less than (-N), or exactly N -empty Match empty file/directory -prune If current file is directory, don't descend into it If none of the following actions is specified, -print is assumed -print Print file name -print0 Print file name, NUL terminated -exec CMD ARG ; Run CMD with all instances of {} replaced by file name. Fails if CMD exits with nonzero -exec CMD ARG + Run CMD with {} replaced by list of file names -delete Delete current file/directory. Turns on -depth option -quit Exit make: *** [Makefile:[81](https://github.com/tianon/dockerfiles/actions/runs/8623651182/job/23637247602?pr=678#step:5:82): root] Error 1 The command '/bin/sh -c make root BUILTIN=1' returned a non-zero code: 2 ```

The same, but this time with GNU `find` and `cpio` (via Alpine's `findutils` and `cpio` packages) instead of the BusyBox implementations Alpine gives us by default to resolve some of the errors seen in the above output (but still ultimately failing):

```console Step 6/9 : RUN make root BUILTIN=1 ---> Running in 2f0c8d3f2aa7 mkroot/mkroot.sh -- BUILTIN=1 cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \ -DPROJECT_NAME=\"ToyBox\" scripts/genconfig.sh kconfig/conf -D /dev/null Config.in > /dev/null logpath:generated/{Config.in,newtoys.h,flags.h,tags.h,help.h} Compile /toybox/root/build/record-commands/logpath ................... find: ‘/usr/local/sbin’: No such file or directory Building for host mkroot/mkroot.sh: line 157: toybox: command not found mkroot/mkroot.sh: line 20: /dev/tty: No such device or address === toybox cleaned cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \ -DPROJECT_NAME=\"ToyBox\" scripts/genconfig.sh kconfig/conf -D /dev/fd/63 Config.in > /dev/null scripts/make.sh warning: using unfinished code from toys/pending generated/{Config.in,newtoys.h,flags.h,tags.h,help.h} Compile toybox ............................................................................................................................................................................................................... scripts/install.sh --long --symlink --force Compile instlist... Install commands... No $LINUX directory, kernel build skipped. mkroot/mkroot.sh: line 20: /dev/tty: No such device or address === initramfs mkroot/mkroot.sh: line 371: /toybox/root/host/docs/initramfs.cpio.gz: No such file or directory cpio: blank line ignored make: *** [Makefile:81: root] Error 1 The command '/bin/sh -c make root BUILTIN=1' returned a non-zero code: 2 ```

[landley]

Huh, it works here. (You shouldn't need BUILTIN=1, that tells the kernel build to statically link the initramfs into the image rather than loading an external cpio.gz, shouldn't apply if you're not saying LINUX=/somewhere .)

The first error is indeed "busybox find -type f,l" not working, it doesn't know how to do fallbacks. Toybox added that in 2020 (commit 70a55cf954ec).

This is a build under Alpine?

[landley]

I'm flying between cities today and unlikely to have a chance to set up an alpine VM, but I note that I recently added scripts/prereq/build.sh to create a minimal toybox capable of running the rest of the build. The initial commit was d1acc6e88be5 and I got it working on the mac (without homebrew) in commit 3bbc31c78b41 .

I need to do a proper fix so this "just works" again on alpine, but in the meantime you might be able to use the mac instructions to work around it?

[tianon]

I don't remember why I added BUILTIN=1; I think there was something that failed without it previously, but I'm certainly not attached to it! :smile:

This is a build on Alpine, but I'm happy to make it not be! I'll swap it to Debian and/or try out your scripts/prereq/build.sh, and see if I can get a better result. :+1: :eyes:

Also, definitely not urgent -- I hope your travels are safe! :bow:

[tianon]

Just pre-running scripts/prereq/build.sh makes it succeed! :muscle:

I think I will switch to a Debian-based builder image though, especially since even with that it gives me weird output from BusyBox and I don't really want to be chasing that all the time. :smile:

Thanks so much for your help and for such a fun project. :+1:

(docker run -it --rm tianon/toybox:0.8.11 coming soon! :rocket:)

[tianon]

Ahhh, I remember why I was using Alpine now -- because otherwise glibc gets grouchy about the static build and warns about functions like getgrouplist and friends being used. :facepalm:

I guess I'll go back to installing GNU tools in Alpine for now so the builds stay safe from glibc-isms and avoid the issues with BusyBox. :+1:

[landley]

I tried to set up an alpine test environment (my last one was a chroot years ago), but it doesn't seem like they ship a livecd? Or at least the "extended" x86-64 image on their "downloads" page isn't one.

I downloaded their CD, kvm -m 2048 -cdrom blah.iso and got a login prompt instead of a desktop, the only account I could guess was "root", then I couldn't "git clone https://toybox" because it didn't have "git" installed. I googled and did an "apk add git" but it said it didn't know the package, "apk update" and "apk upgrade" didn't help...

This is not really a livecd.

[landley]

Used the setup program to install it to a virtual disk, booted that, logged in, installed git, logged in as the non-root user I'd created, cloned the repo, there was no make... and no sudo. And "apk add sudo" didn't work. Right... Ok, installed make, there was no gcc, installed that, and now it says ctype.h not found. I have to install an additional package to get standard posix headers supplied by musl, installing the compiler does not get me headers.

This is not the friendliest distro I've encountered. Also, what's the difference between the "extended" image and the "minimal" image?

Installed musl-dev. Installed bash. And now the build is complaining linux/rfkill.h isn't installed...

[landley]

I set up a busybox airlock directory (mv busybox dir; cd dir; for i in $(./busybox --list); do ln -s busybox $i; done) and stuck it at the start of the $PATH, and commit 7c6aecd477a9 build mkroot CROSS=i686 with that. (No LINUX= so it didn't build a kernel.)

Hopefully that means it works on alpine again. I replaced the -type x,y with ( -type x -o -type y ) because that's what busybox understands, and the airlock setup succeeding should handle the rest...

[tianon]

Oof, sorry you went through so much trouble on my behalf! Hopefully it felt productive for you? :see_no_evil:

If you need Alpine again (or want to set up some kind of CI / automated builds / testing) and are OK with a chroot, I'd suggest instead using the minirootfs tarballs from https://dl-cdn.alpinelinux.org/alpine/latest-stable/releases/x86_64/ (or the suitable alternatives from another directory if you're on a different architecture :heart:). :smile:

---

I tested using https://github.com/landley/toybox/archive/aedbaa5c4d451859b0e8b470e9520f722915bda0.tar.gz in place of the release tarballs (aedbaa5c4d451859b0e8b470e9520f722915bda0 being current HEAD), got rid of GNU's find and cpio so I was back to stock BusyBox tools, and got the following output and a successful end result:

Step 5/9 : RUN make root
 ---> Running in 15b164a9d71b
mkroot/mkroot.sh 
...................
find: /usr/local/sbin: No such file or directory
Building for host
mkroot/mkroot.sh: line 157: toybox: command not found
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== toybox
cleaned
cc -o kconfig/conf kconfig/conf.c kconfig/zconf.tab.c -DKBUILD_NO_NLS=1 \
    -DPROJECT_NAME=\"ToyBox\"
scripts/genconfig.sh
kconfig/conf -D /dev/fd/63 Config.in > /dev/null
scripts/make.sh

warning: using unfinished code from toys/pending
generated/{Config.in,newtoys.h,flags.h,tags.h,help.h}
Compile toybox
...............................................................................................................................................................................................................
scripts/install.sh --long --symlink --force
Compile instlist...
Install commands...
No $LINUX directory, kernel build skipped.
mkroot/mkroot.sh: line 20: /dev/tty: No such device or address

=== initramfs
find: unrecognized: -printf
cpio: unknown user/group +0:+0
BusyBox v1.36.1 (2023-11-07 18:53:09 UTC) multi-call binary.

Usage: find [-HL] [PATH]... [OPTIONS] [ACTIONS]

Search for files and perform actions on them.
First failed action stops processing of current file.
Defaults: PATH is current directory, action is '-print'

    -L,-follow  Follow symlinks
    -H      ...on command line only
    -xdev       Don't descend directories on other filesystems
    -maxdepth N Descend at most N levels. -maxdepth 0 applies
            actions to command line arguments only
    -mindepth N Don't act on first N levels
    -depth      Act on directory *after* traversing it

Actions:
    ( ACTIONS ) Group actions for -o / -a
    ! ACT       Invert ACT's success/failure
    ACT1 [-a] ACT2  If ACT1 fails, stop, else do ACT2
    ACT1 -o ACT2    If ACT1 succeeds, stop, else do ACT2
            Note: -a has higher priority than -o
    -name PATTERN   Match file name (w/o directory name) to PATTERN
    -iname PATTERN  Case insensitive -name
    -path PATTERN   Match path to PATTERN
    -ipath PATTERN  Case insensitive -path
    -regex PATTERN  Match path to regex PATTERN
    -type X     File type is X (one of: f,d,l,b,c,s,p)
    -executable File is executable
    -perm MASK  At least one mask bit (+MASK), all bits (-MASK),
            or exactly MASK bits are set in file's mode
    -mtime DAYS mtime is greater than (+N), less than (-N),
            or exactly N days in the past
    -atime DAYS atime +N/-N/N days in the past
    -ctime DAYS ctime +N/-N/N days in the past
    -mmin MINS  mtime is greater than (+N), less than (-N),
            or exactly N minutes in the past
    -newer FILE mtime is more recent than FILE's
    -inum N     File has inode number N
    -user NAME/ID   File is owned by given user
    -group NAME/ID  File is owned by given group
    -size N[bck]    File size is N (c:bytes,k:kbytes,b:512 bytes(def.))
            +/-N: file size is bigger/smaller than N
    -links N    Number of links is greater than (+N), less than (-N),
            or exactly N
    -empty      Match empty file/directory
    -prune      If current file is directory, don't descend into it
If none of the following actions is specified, -print is assumed
    -print      Print file name
    -print0     Print file name, NUL terminated
    -exec CMD ARG ; Run CMD with all instances of {} replaced by
            file name. Fails if CMD exits with nonzero
    -exec CMD ARG + Run CMD with {} replaced by list of file names
    -delete     Delete current file/directory. Turns on -depth option
    -quit       Exit
Output is in /toybox/root/host
Removing intermediate container 15b164a9d71b

(happy to share more details if you want them, or test more things, but this seems good :+1:)

---

Again, I really appreciate it, and enjoy playing with and using Toybox. :grin:

[landley]

Ah, the downloadable chroot is probably what I used last time.

I want it to work, you brought me an example of it not working, I'm gnawing at it.

Huh, it looks like it's not installing the airlock (which is why attempting to set VERSION via toybox --version is not finding it). I wonder why?

The /dev/tty thing is harmless (announce is trying to change your title bar), I suppose I could add a 2>/dev/null to suppress it, I've just never encountered it before...

The busybox find -printf thing at the end is again, because busybox doesn't support an option both toybox and gnu do, and it didn't build the airlock for some reason.