Build system improvement and rework

[xblax]

The mod build systems how it was initially set up, currently lacks some flexibility and automation.

I intend to refactor that after the public beta, but before the public release milestone and before any touch control solution (https://github.com/xblax/flashforge_adm5_klipper_mod/issues/18) is pulled into the master branch. We can still have preview releases for Klipper-Screen in the mean time, built from the feature-branch.

Refactor happens after public beta, because the repo should not stay closed for too long.

Here's what I have in mind:

• Have an overall build process that includes:
  • building the wheels for all python components
  • building the buildroot project
  • assemble all release archives i.e. Adventurer5M-KlipperMod-v01.XX-flavor.tgz
• Automated creation of python venv on target from wheels (if not pre-compiled)
• Unified definition of build-environment (directories, toolchain) instead of copy-pasta
• Drop the buildroot fork repository:
  • Move buildroot configs in the the main repo
  • Use build root external tree feature to fix / add certain packages (see https://buildroot.org/downloads/manual/manual.html#outside-br-custom)
  • Point the buildroot submodule to upstream Git
• Support multiple build flavors for the mod: The mod needs to support multiple build-flavors, i.e. -lite and -KlipperSceen, that are all build from the same branch. To achieve that we need:
  • common base: buildroot base config and base overlay
  • per flavor: buildroot extended config and overlay (merged with base during build)
  • custom build steps, dependend on flavor (i.e. installing KlipperScreen)

[consp]

(Not) including klipperscreen as a build variant should be quite doable since it's X11 with GTK requirements (gobject-inspector specifically) which needs a clean build (due to buildroot not compiling the typelib files otherwise) and a few minor patches and files like xorg.conf and the svg loader for pixbuf since it breaks the buildroot. The venv build can be done the same way as the others are. I see no reason why it should be a problem. Maybe make two external trees or something similar like buildroot supports.

A testing version of ks should be (internally) tested at least since there are a few unknowns with respect to the touchscreen. My guess it will be fine but who knows.

[xblax]

Build system should produce checksum files that are uploaded together with the releases on GitHub. See https://github.com/xblax/flashforge_adm5_klipper_mod/issues/58

[xblax]

Started working on this. First step is to improve is the Buildroot integration and the overall build process.

Will use the Buildroot feature to generate an SDK that will be used as the external toolchain to build the actual variants. This speeds up building the actual variants, since no tool chain must be re-compiled for each of them. The SDK will also be provided as a build-artifact for each release and can be used for third-party development.

Buildroot config files get split up into sdk, base, variant-lite, variant-klipperscreen files that are automatically merged into a single .config. Also the overlay will be split into a base overlay and a variant specific overlay.

It would also be nice if we can also use a GitHub hosted runner to do a complete build for us, but maybe the resources provided by GitHub for the build VM are not enough: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources

[consp]

Nice! I'll have a look once you worked out the rough edges, not that familiar with is so I'll see how it goes and read up on it when it's a bit more clear.

It would also be nice if we can also use a GitHub hosted runner to do a complete build for us, but maybe the resources provided by GitHub for the build VM are not enough

14 GB storage

Is the only one I would be worried about and I'm not worried about it now. Less cores == more time but that should not be an issue. Memory is plentiful. It's always possible to reduce the base image by clearing caches and dropping unused packages.

[xblax]

I pushed the first "milestone" of the build system rework to branch feature/buildsystem.

As always initial documentation is spare, this commit has some of the key points in the commit description:

* Move buildroot configs into this repo, split into fragments
* Separate targets for sdk, and mod variants "lite" and "klipperscreen"
* Build targets are orchestrated by top level Makefile
* build.sh in build_scripts implements the individual build steps (called from Makefile)
* Add skelton for Buildroot external tree. ADM5 customizations still need to be 
  moved from Buildroot submodule into integrated external tree.

This rework does essentially integrates the klipperscreen feature branches into master, therefore I want to merge these changes rather sooner than later. Running make in the git root should build everything, but individual build targets are available of course.

Adventurer5M-KlipperMod-uninstall.tgz
Adventurer5M-KlipperMod-v00.03-beta-59-gc6c3f90-dirty-klipperscreen.tgz
Adventurer5M-KlipperMod-v00.03-beta-59-gc6c3f90-dirty-lite.tgz
KlipperMod-SDK-v00.03-beta-59-gc6c3f90-dirty.tar.xz
md5sums

The configs for sdk, lite, and klipperscreen are automatically merged from build_scripts/buildroot/configs and the buildroot build environments are set in build_output/buildroot/ It's still possible to a local menuconfig and do a manual build of the chroot if needed.

ls -1 build_output/buildroot/
sdk
variant-klipperscreen
variant-lite

@consp Can you give this a try? If it works for you, I will merge it tonight. Then we can focus a bit on fixing some of the reported issues and doe v00.04-beta release.

[consp]

Sure, should be able to try it tonight.

ADM5 customizations still need to be moved from Buildroot submodule into integrated external tree.

Will give an attempt for that tonight as well. Since there are a bit more changes in the klipperscreen variant (wrt buildroot) might need a bit more time since there are some patches for gobject-introspection and xorg packages which need to be installed (i.e. to get rid of eudev requirements), might need to rework those, don't know yet.

It's still possible to a local menuconfig and do a manual build of the chroot if needed

Considering how often I needed that for testing with Xorg that is highly valuable.

[xblax]

Will give an attempt for that tonight as well. Since there are a bit more changes in the klipperscreen variant (wrt buildroot) might need a bit more time since there are some patches for gobject-introspection and xorg packages which need to be installed (i.e. to get rid of eudev requirements), might need to rework those, don't know yet.

Those changes are still in the buildroot submodule. The feaure/buildsystem in buildroot has all the klipperscreen changes. I think everything should be functional already.

These changes are supposed to be moved into the external tree (if possible), but that is not critical to merge the branch.

[consp]

Also testing a WSL build for the windows users. This might take a while to build.

[xblax]

Also testing a WSL build for the windows users. This might take a while to build.

Full build of SDK, lite and klipperscreen variants took me about ~1h with initially empty ccache (Ryzen 5900X).

I think there are some optimizations left, i.e. only the SDK should have host python, not the variants. Build process can also be optimized to take less space for running on Github. But that's something to look at later.

[consp]

~1h with initially empty ccache (Ryzen 5900X).

Doing it now, same cpu, -j20, don;t know if that's propagated and realy only matters with the xorg parts. Been busy for about 27 minutes now.

SDK should have host python, not the variants

Do you mean for the target device (I always mix up up which is which)? I remember numpy being an issue.

[xblax]

Doing it now, same cpu, -j20, don;t know if that's propagated and realy only matters with the xorg parts. Been busy for about 27 minutes now.

No it's not propaged to buildroot. Buildroot automatically compiles in parallel, but all the individual buildroot packages are build sequential. However setting -j20 on the top level Makefile, makes the lite and klipperscreen variant build in parallel at least.

There is a small bug, that the checksum file is generated before all packages are build. I fixed it now: https://github.com/xblax/flashforge_adm5_klipper_mod/commit/c3c3fb95255f94230fff66827a20a1911a8f9914

We could also try out https://buildroot.org/downloads/manual/manual.html#top-level-parallel-build to speed up the build process.

[xblax]

SDK should have host python, not the variants

Do you mean for the target device (I always mix up up which is which)? I remember numpy being an issue.

No the one for the host. Target must be build by each of them. The idea is that the wheels for the venv are built via the SDK.

[consp]

Apperently only built the klipperscreen variant, on WSL it takes 1:45, now building the rest.

[consp]

* created checksums:
0bbf447e6f290ed50a7a1d8711258d5a  Adventurer5M-KlipperMod-uninstall.tgz
7332425e0e793b48d87ea56e92124814  Adventurer5M-KlipperMod-v00.03-beta-59-gc6c3f90-dirty-klipperscreen.tgz
6b65dfabd9b4d90ce27bf4c6ff71daae  Adventurer5M-KlipperMod-v00.03-beta-60-gc3c3fb9-dirty-klipperscreen.tgz
a0cb83c3c43c39efe688d215e25e67ef  Adventurer5M-KlipperMod-v00.03-beta-60-gc3c3fb9-dirty-lite.tgz
865c3ef781379f895399f63b503526d9  KlipperMod-SDK-v00.03-beta-59-gc6c3f90-dirty.tar.xz
865c3ef781379f895399f63b503526d9  KlipperMod-SDK-v00.03-beta-60-gc3c3fb9-dirty.tar.xz

in total took 2 hours in WSL, double klipperscreen is due to pull for checksum fix.

Maybe it's nice to also create the "pro" named files, I've done it wrong several times now...

[xblax]

in total took 2 hours in WSL, double klipperscreen is due to pull for checksum fix.

At least it works out of the box! I will remove the "dirty" tag, as we cannot make builds where the repo stays "clean" yet.

Maybe it's nice to also create the "pro" named files, I've done it wrong several times now...

Thought about that as well, but not sure if there are any GitHub limitations about the release sizes. Feels a bit like wasting disk space, but they might use de-duplication internally.

[consp]

At least it works out of the box!

After installing all the required components 😉

---

one minor issue, see 626e01b

Stray "+' 's caused tz_info to not be included.

everything else works as expected in the short window i've now tested it.

GitHub limitations about the release sizes

I've seen public non-commercial repo's with gigabites of sizes, I do not think it will be an issue (yet)

might use de-duplication internally

With these kinds of thing's that's 1000% worth the effort.

---

rebuilding now to see if that fixes it but i'm 100% sure it will.

[pedantic mode] Might also need to fix the "Klipper mod is installed ..." "Please wait" screen during installation, technically it would be "is installing" and after finishing "has been installed"

It might also be nice to have an earlier screen dump to the FB since it goes black for a while. But that's "final product" things. [/pedantic mode]

---

I've discovered a bug (which represents on a clean install) with the timezones in klipperscreen. Going to fix that now. Timezone info works as expected though!

Fixed in 4e68105 and https://github.com/consp/KlipperScreen/commit/e151558ee4bbccbce4ef12203c41e2e7c87d2559

[xblax]

Ok seems to be "good enough" for now, merged in into master. Ticket stays open because more improvements to the build system are planned, but less critical.

On the list:

• Move changes to buildroot git into the external tree in build_scripts/buildroot
• GitHub automatic build
• Python venv / wheel build process
• Klipper/Moonraker installation, applying patch series (maybe via Buildroot package)

[consp]

• Klipper/Moonraker installation, applying patch series (maybe via Buildroot package)

Include KlipperScreen into that, should be doable with just patch files. At least for the changes done so far.

Python venv / wheel build process

My guess automating that is the biggest challenge.

[consp]

https://github.com/xblax/flashforge_adm5_klipper_mod/tree/feature/copy-artifact-as-pro

I copied the normal file and started into stock too many times now so added a copy to "pro" name in the build script.

[xblax]

https://github.com/xblax/flashforge_adm5_klipper_mod/tree/feature/copy-artifact-as-pro

I copied the normal file and started into stock too many times now so added a copy to "pro" name in the build script.

Added to master and fixed the wording of the pending install message: https://github.com/xblax/flashforge_adm5_klipper_mod/commit/9b220c1f937635d65797bda171d2d6dda28736b2

[xblax]

On branch feature/buildsystem I have integrated all custom buildroot packages into the main repo.

While doing that I also cleaned up the buildroot submodule an rebased the remaining changes on top of upstream 2023.02.11. Now we only have 4 commits left in the submodule, these will probably have to stay there. Currently the changes are on the 'cleanup' branch: https://github.com/xblax/flashforge_adm5_buildroot_chroot/commits/cleanup/

Seems to build and run fine. If no one notices any issues I will push that to master soon and make the cleaned-up buildroot branch the new master branch in the submodule.

I have also integrated the Guppyscreen branch while doing that.

[ballaswag]

This is very cool. I have limited context for buildroot, but the K1 does the same on a mipsel SoC. I have been building custom executables using their buildroot variant to enhance the functionality, but looks like how you guys are overlaying a custom buildroot distribution on top and I think that's the best way to go.

Would love some context if you guys get some time. My understanding from a quick read through of this repo is as follows:

1. A custom buildroot config that builds desire packages.
2. Building this config will give a buildroot image output. This is essentially /.
3. Kinda fuzzy here, create an overlayfs with custom buildroot image on /?
4. Intercept init by replacing existing init rc? With your custom init.
5. Custom init bootstraps overlayfs (the goodies) and create a chroot jail, runs custom init scripts to spin up services.
6. Now the mod is running.

Questions:

1. Does the system still depend on the existing buildroot /libs? If that's the case does this mean the custom buildroot version has to match, that's abi, glib versions and etc has the compatible.
2. How does init intercepting and chroot work? Does chroot require specific kernel modules (I see some prebuilt kernel modules).

Would appreciate if you guys give some insight before I start digging.

[xblax]

@ballaswag Sure I can explain some details and background. It's not an overlayfs to the orginal system, but a complete replacement. The mod system environment is completely separate from the original Flashforge Linux (but it's contained within the original system).

Why I did choose this approach: In the very beginning when I tested the printer I was using the same approach to have a matching buildroot and install/copy/modify individual components in the original system.

But the original system is based on very outdated Buildroot 2019.02, missing essential components. It's also very tedious to do such a piecewise modification. But most importantly the AD5M has no real factory reset / full stock system image. All changes are permanent without rollback option. So for a public mod I needed a "safe" method to install a mod without heavily modifying the original system, with very low (or close to zero) chance to brick the printer.

How the mod is booted: Basically, the mod installation in the original system only consists of two modifications:

1. /etc/init.d/S00klipper_mod (see https://github.com/xblax/flashforge_ad5m_klipper_mod/blob/master/build_overlay/common/etc/init/S00klipper_mod - it's copied to the host system root during installation)

This script runs before any other original services. It setups the chroot jail environoment and hijacks the init process to prevent any of the original services from beeing started (the script kills it's parent process in the end).

It also checks various runtime conditions, so we are able to "dual boot" into the original system and do some other stuff.

start_klipper_mod() {
    echo "Start Klipper Mod Init" | tee /dev/kmsg
    check_usb_marker
    mount_data
    check_mod_marker
    prepare_chroot
    chroot $chroot_dir /etc/init/init_chroot.sh
    # now we have hijacked the init process and let it go :-)
    kill -9 $PPID
    sync # make sure our logs are saved
} > $log_file 2> $log_file_debug

2. The entire mod system is self contained in /data/.klipper_mod and /data/.klipper_mod/chroot is our / We have some custom init scripts to continue the startup process in the chroot, very similar to normal initialization. See https://github.com/xblax/flashforge_ad5m_klipper_mod/blob/master/build_overlay/common/etc/init/init_chroot.sh

The mod is a complete Linux environment. We have our own syslog, ssh server, udev, python installation, klipper, moonraker ... basically everything. So we are very flexible what we install and how we tune the system. Everything is one integrated build-process (mostly) and to update the mod we just replace the entire chroot (some configuration data is retained).

What are the dependencies?

Basically only the original Linux Kernel and Bootloader and the Klipper firmware running on the MCUs.

We don't really depend on any ABIs. The mod is based on buildroot 2023.02 (LTS), we use recent glibc and toolchain. The mod is build with hard float toolchain and all CPU optimizations turned on, while the original system uses soft float ABIs.

The buildroot system must be compiled against the correct kernel headers. That is important so that our glibc is compatible with provided kernel, but that's it. We have included some prebuilt kernel modules to add some additional features that are missing in the original kernel.

[ballaswag]

Thanks for the detail walkthrough, makes a lot of sense. I really didn't want to pick up another project but I might take a look at doing the same for the K1/nebula pad.

[consp]

Basically only the original Linux Kernel

Isn't that by choice? I't not as easy, but since the entire device tree is present there is nothing stopping anyone from running a modified kernel, though that would be another level of change and maybe a bit out of scope of the mod.

[xblax]

Basically only the original Linux Kernel

Isn't that by choice? I't not as easy, but since the entire device tree is present there is nothing stopping anyone from running a modified kernel, though that would be another level of change and maybe a bit out of scope of the mod.

Yes sure, that's by choice. It would technically be possible to swap out the kernel or even the bootloader. But it's not necessary for the mod to work as-is and now it's a dependency by design. If they would decide to update the kernel to another major release (which I doubt will ever happen) the mod would probably still boot, but the additional modules would fail to load.

[ballaswag]

Your on buildroot feature/buildsystem and the work @consp put in to resolve klipperscreen deps made it very simple for me to build the mipsel toolchain and build the klipperscreen variant.

Mipsel doesn't support some float defined in numpy, so only need to patch buildroot to use newer version of numpy to by pass some compilation error.

However, since the K1 SoC is quite custom, I'm having a hard time trying to bootstrap and init all the devices (kernel module and all). Also they run on Linux 4.4.x and 2023.02 buildroot has 4.14.x as the minimum. Not sure how big of a problem this is yet.

My goal was to create a minimal chroot (no klipper and fiends just yet) as a poc, I removed most of the init.d and only left the system related one, removed mdev too because its modprobe was causing some error.

On reboot I see the mounts and chroot setup calling init_chroot.sh. Had to do this over serial because wifi is not configured properly yet. On the serial console, the hostname changed, but looks like it broke out of chroot. Login is same as whatever creality had it originally. The / is not actually what's given to chroot but instead the original fs. Not sure why the jails escape. Is this expected?

If I run chroot mod_path init_chroot.sh manually, I'm dropped to a shell that mounts / on the mod fs. Things like python and other custom overlay works.

Any tip would be appreciated! Especially around kernel version mismatch and the chroot jail escaping on boot.