Description

This project performs Automated Linux From Scratch jhalfs using docker. The approach is similar to the one taken by LFScm: lfs is built inside of a docker container on a loopback device, and the resulting image booted in qemu.

Warning: this project uses privileged containers, and may make your host (laptop) non-bootable if used improperly.

The jhalfs is a set of scripts which retrieves the Linux From Scratch lfs book, extracts the commands by parsing the chapter's XML, and generates scripts managed using a makefile. The systemd version of the lfs book is used, as systemd is the standard nowadays.

Note: the setup requires about 16GB of disk space (a bit less than twice the size of the loopback file). Intel processor architecture is supported. Apple silicon (unsuccessful yet) notes are collected in 1. To be clear, this project does not boot lfs on a physical drive attached to computer, but instead boots an lfs image in an x86_64/amd64 or aarch64/arm64 qemu.

Setup

Install docker and docker-compose. On MacOS install rancher-desktop, and during the installation select "moby/dockerd" as the engine. Docker is used to perform the jhalfs build inside of an Ubuntu container.
Install qemu. On Ubuntu use sudo apt-get install -y qemu-kvm and on MacOS use brew install qemu. Qemu is used as a virtual machine used to boot the created lfs system.
Install expect. On Ubuntu use sudo apt-get install -y expect and on MacOS use brew install expect. Expect is used to perform an automated test of lfs booting.

Usage

Building the system

The setup in this repository uses docker to start an ubuntu:latest container, inside of which lfs is built.
```
docker compose up -d
```
In preliminary steps a loopback device is created, formatted and mounted as the destination for the scripts generated byjhalfs. Due to the use of a loopback device the Ubuntu container needs to be started with --privileged option.

The 02-preparing-for-the-build.sh script in this repository performs these preliminary steps, up to and including chapter4 of the lfs book. After the preliminary steps are done, jhalfs run is invoked to generate the scripts and makefile. The configuration used by jhalfs run is stored in the configuration file in the root folder of this repository.

docker compose exec jhalfs bash -c "cd /vagrant && bash /vagrant/02-prepare-for-the-build.sh"

Fetch the commits of the jhalfs and lfs books with:

docker compose exec jhalfs bash -c "su - vagrant -c 'cd /home/vagrant/jhalfs && git show --oneline --no-abbrev --shortstat'"

docker compose exec jhalfs bash -c "su - vagrant -c 'source /vagrant/jhalfs/jhalfs.sh && cd \$LFS/jhalfs/book-source && git show --oneline --no-abbrev --shortstat'"

The references to /vagrant in this repository are leftovers from a vagrant based setup.

Due to the 6h limit on a github action runner, disable slow chapter8 tests.

docker compose exec jhalfs bash -c "su - vagrant -c 'source /vagrant/jhalfs/jhalfs.sh && cd \$LFS/jhalfs/lfs-commands/chapter08 && sed -i \"/make -k check/d\" 8*-glibc && sed -i \"/Timed out/d\" 8*-glibc && sed -i \"/test_summary/d\" 8*-glibc'"
docker compose exec jhalfs bash -c "su - vagrant -c 'source /vagrant/jhalfs/jhalfs.sh && cd \$LFS/jhalfs/lfs-commands/chapter08 && sed -i \"/make -k check/d\" 8*-gcc && sed -i \"/Timed out/d\" 8*-gcc && sed -i \"/test_summary/d\" 8*-gcc'"

Since the kernel .config is changing rapidly, use defconfig instead of a static configuration file. If any of the expected options are not included in the .config the kernel's make will hang waiting for user's y/n input, and defconfig is a way to prevent this.

docker compose exec jhalfs bash -c "su - vagrant -c 'source /vagrant/jhalfs/jhalfs.sh && cd \$LFS/jhalfs/lfs-commands/chapter10 && sed -i \"s|cp -v ../kernel-config.*|make defconfig|\" 10*-kernel'"

From this point the individual makefile targets can be executed, for example:
```
docker compose exec jhalfs bash -c "su - vagrant -c 'source /vagrant/jhalfs/jhalfs.sh && cd \$LFS/jhalfs && make ck_UID'"
```
See jhalfs/targets for the list of existing makefile targets. If you prefer to run all targets use make all. The result of the build is available under jhalfs/mnt/build_dir. If you want to access the contents of this directory from the host (laptop), mount it sudo mount $(cat jhalfs/build_dir.dev) $PWD/jhalfs/mnt/build_dir.

At the end of the build, print the SBUs report:
```
docker compose exec jhalfs bash -c "su - vagrant -c 'source /vagrant/jhalfs/jhalfs.sh && cat \$LFS/jhalfs/*SBU_DU*.report'"
```

Booting the system

The backups of the lfs filesystem are present in the root directory of this project. They can be used to boot a system. In order to boot the created system, https://www.qemu.org/ is used here.

Configure grub, see chapter10:

docker compose exec jhalfs bash -c 'source /vagrant/jhalfs/jhalfs.sh && cd /vagrant && bash 10.4-configure-grub.sh'

Note that when qemu is used the root device is /dev/vda1.

Test the system boot, from the host (laptop):
```
sudo chown $USER build_dir.img
time TERM=linux JHALFS_ARCH=$(uname -m) expect -f 10.4-test-boot.exp
```
The expect script will boot the system, login as root using the password configured in step 2. above, and print the pretty release of the system. It should contain "Linux From Scratch". See the expect script for a command used to launch the system using qemu-system.

If desired, backup the system, see chapter7:

docker compose exec jhalfs bash -c "source /vagrant/jhalfs/jhalfs.sh && cd \$LFS && bash /vagrant/10.4-save-backup.sh chapter10"

Copy the backup to the host (laptop):

docker compose exec jhalfs bash -c "source /vagrant/jhalfs/jhalfs.sh && cd \$LFS && cp -p jhalfs/chapter10*.tar* /vagrant"

Only after finishing the project exploration, stop the container, and cleanup the loopback device from the host (laptop):

docker compose stop jhalfs
for dev in $(losetup -n -l -O NAME -j /vagrant/build_dir.img); do echo "Detaching $dev" && sudo losetup -d $dev; done
for dev in $(losetup -n -l -O NAME -j /build_dir.img); do echo "Detaching $dev" && sudo losetup -d $dev; done

Do not neglect this step, otherwise you'll keep dangling loopback devices using virtually space on disk.

Saving intermediate images

In order to save the current work, commit the state of the container, for example after the make ck_SETUP target in II. Preparing for the Build, stop and remove the container:

docker commit -m "II. Preparing for the Build" jhalfs jhalfs:part2
docker save jhalfs:part2 > "jhalfs:part2.tar"
docker stop jhalfs
docker rm jhalfs

If the image was saved, and transfered to another host (laptop), it needs to be loaded, and used to start a container:

docker load < "jhalfs:part2.tar"
JHALFS_IMAGE=jhalfs:part2 docker compose up -d
# Attach and mount the loopback device, it is assumed to be free
docker compose exec jhalfs bash -c "losetup \$(cat /vagrant/jhalfs/build_dir.dev) /vagrant/build_dir.img && mount -a"

Other parts, after the make mk_CHROOT target in III. Building the LFS Cross Toolchain and Temporary Tools and after the make mk_BOOT target in IV. Building the LFS System are other convenient places to save work. This is useful to split work to deal with time limits of the continuous integration workers.

docker commit -m "III. Building the LFS Cross Toolchain and Temporary Tools" jhalfs jhalfs:part3
docker save jhalfs:part3 > "jhalfs:part3.tar"
docker stop jhalfs
docker rm jhalfs

docker commit -m "IV. Building the LFS System" jhalfs jhalfs:part4
docker save jhalfs:part4 > "jhalfs:part4.tar"
docker stop jhalfs
docker rm jhalfs

marcindulak / jhalfs-ci