Simplify + Sustainability

[fosslinux]

This is still fairly rough around the edges; but all of the core functionality I wanted has been implemented.

This does not create any meaningful change to the bootstrap path used. Rather this is a fairly significant refactor of live-bootstrap to make it easier to reason about, more sustainable to make changes to in the future, and fix a number of long-standing issues.

• The most significant change made is the removal of the sys* system introduced very early in the project. That abstraction has become increasingly meaningless and ultimately created many issues jumping between syses, and sysa/sysc had so many different run scripts (run.kaem, run.sh, run-after-fiwix.sh, run2.sh), which didn't actually convey any meaning. There were so many varied cases depending on chroot/kernel bootstrap/etc. I believe that this is the biggest barrier to understanding the project. Technical details;
  • /init's are generated files now
  • The filesystem is persistent across the entire bootstrap; ie, packages are not reinstalled, rather everything is moved, after transitions. There are benefits and drawbacks to this, but for now, I think it is OK.
  • With the exception of builder-hex0 to fiwix, which needs a hardcoded list of files, that I would like to remove at some point, but I'm not sure if that will happen before 1.0.
• Now, we have a human AND machine readable manifest that details the progression of the bootstrap, stored in steps/manifest. The M2-Planet program seed/script-generator.c transforms the manifest dynamically into a series of kaem/shell scripts that invoke one another. The syntax of the manifest is detailed in the manifest. All associated scripts required to get to the point where the manifest can be used (which is almost immediately after stage0-posix), live in seed/.
  • I think this will prove to be much more sustainable and sensible than the sys* system.
  • Further documentation will be created regarding what script-generator creates, but it is expected that the average contributor will not need to understand the abstraction of the manifest.
• Another significant change is the clear separation of scripts and other sources external to live-bootstrap.
  • Everything that is NOT part of, or an artifact of, live-bootstrap, is stored in /external in the rootfs.
• When using kernel bootstrap/Linux kernel, there is a new propagation mechanism for data between kernels.
  • In builder-hex0, everything exists on its srcfs.
  • In fiwix, /external is mounted from a disk, everything else comes from an initrd. Once util-linux is built, everything transfers into that disk, moving off the initrd.
  • In Linux as a seed (replacing fiwix and builder-hex0), the system boots from a single disk.
  • In Linux built by live-bootstrap, the system boots from a single disk - either the same one if Linux was a seed, or the one created in fiwix.
  • The primary benefit of this is we are no longer restricted by size of initrd - this was a requirement of this PR for me, as that was the primary cause of the sysa/sysc split.
  • This was achieved by adding a very janky and hampered mknod and mount to mes libc, and creating early command line utils for them named efsu (Early FS Utils)
• The early kaem era has been modernized a bit, alongside environment variable handling.
  • The meaningless lowercase variables have been removed.
  • The /steps/env replaces .env. /steps/env propagates throughout the entire filesystem.
  • The system is musl-first in all cases, not mes-first. This means, that musl-era conforms largely to FHS, and mes-era largely does not conform to FHS. Practically, this means we have /usr/lib/mes and /usr/include/mes, and no /usr/include/musl.
  • I'm still uncertain about how to handle the mes environment variable spam. For now, I have tried to limit the scope of mes's environment variables as much as possible, and tcc & mes build scripts have been slightly decoupled - to be revisited at a later date.

TODOS

• [ ] Lint
• [ ] HELP NEEDED - kernel bootstrap mode is broken ?!. Consistently segfaults just into fiwix. statically compiled bash binary segfaults. memory mapping issue ?? not sure what's going on here
• [ ] finish doc

[eduardosm]

HELP NEEDED - kernel bootstrap mode is broken ?!. Consistently segfaults just into fiwix. statically compiled bash binary segfaults. memory mapping issue ?? not sure what's going on here

In order to make this a bit easier, I think some parts of this PR can be splitted to their own PRs, reducing the amount of changes introduced by this PR and hopefully making it easier to pinpoint the source of the problem.

However, that would require rebasing this PR and conflicts are expected, but I can help with that.

[fosslinux]

I'm in the process of cleaning up the commit history for this PR, and fixing little bits and pieces.

I think I've split out everything from this PR that can be logically split out.

We have as dependencies for this PR;

• [x] #340
• [x] #341
• [x] #337
• [x] #343
• [x] #344

[fosslinux]

Only large remaining error is fiwix -> linux not working. This is in the process of being fixed.

[fosslinux]

I'm fairly happy with this now. CI is obviously broken but I would still like to limit the scope of this PR.