Separate development environment

[MedicineYeh]

As we discussed here. I think it is better to have a simple build system and the building environment in a separate repo. This helps to manage and isolate the images, guest libraries, etc. My proposal is as described here: https://github.com/apertus-open-source-cinema/axiom-beta-qemu/issues/6

The alpha version of the implementation is available here (all functions completed): https://github.com/MedicineYeh/axiom-beta-env

It should work on any Debian dist. (including Ubuntu) and ArchLinux (my development system). As for CentOS and Fedora, a test is still needed.

[anuejn]

All in all, axiom-beta-env looks very good, and i think that its a quite handy tool for development :) I would be happy if we could adopt this into the ci build path instead of calling qemu manually. Moreover I think, that it is correct to have the qemu scripts and the build tools in a seperate repo (as you said) and call them during the build.

[MedicineYeh]

So, I guess the plan will be something like removing the folder build_tools. And suggest user download the sources of build environment in README. Then, this repo is completely free from depending on QEMU, Linux, etc. It will be a pure SW/HW(FPGA) library repo of AXIOM beta. Is that okay to the maintainer?

Roughly, there will be three repos when someone tries to build the whole thing from scratch. It will be just one after one has built the environment.

1. axiom-beta-env contains all the build environments, binaries. This is supposed to be the main and root repo.
2. microzed-image is the repo containing build scripts for building a guest image from scratch. This part can vary a lot. Read more in the following paragraph.
3. beta-software which contains all the SW/HW developed by apertus team specifically for AXIOM beta.

The isolation of guest images gives the freedom on the versions of microzed-image as well as adding other repos in the whole build systems. The overall hierarchy will be:

`beta-software` (could be placed anywhere)
`axiom-beta-env`
├── externals (other 3rd party tools)
├── Xilinx QEMU
├── patches (patches for Xilinx QEMU)
└── guest_images (Use directory to isolate different versions of kernels/AXIOM libs/AXIOM binaries
    ├── dev
    │   ├── `microzed-image`-v1.2
    │   ├── `microzed-image`-v1.3
    │   ├── `some-other-repo`
    │   │    └── `beta-software` (might be downloaded by the build script)
    │   └── ......
    └── AXIOM-beta (official prebuilt images with all necessary files)
        ├── devicetree.dtb
        ├── rootfs.ext4
        ├── zImage
        └── runQEMU.sh (stand-alone runQEMU script)

The last question would be: should be put beta-software as an out-of-guest build or in-guest build? Building beta-software without depending on guest libraries and binaries brings the best compatibility, but it will require one to build all the libraries this repo depends on. Build the beta-software in-guest takes time, but it utilizes the guest libraries and packages. The advantages are too many, including faster binary load time(in execution), reach of latest (fancy) libraries like tensorflow, and etc.... On the other hand, the only advantage of building beta-software out-of-guest is that it requires nothing to build, which will be easier to build it and test it on x86 platform.

My suggestions: If we want to build beta-software on x86 platform, the best bet is simply downloading the root file system of ArchLinux and using it as the build environment (including gcc). (like this). This does not require to build the whole image.

P.S. In this proposal, beta-software will be stand-alone test-able (every function other than kernel modules). It just needs some fake environments. For example, physical memory accesses can be replaced by a shared memory through a file in testing mode. And we can launch another process to take and respond to the tested program through that file.

However, custom kernel modules are still required to be built inside guest OS. In other words, the beta-software will generate a package (whether in ArchLinux package format or not). The package will be installed inside the guest OS by the user themselves. In the installation step, the scripts will automatically build the kernel modules in the guest OS. After the finalization, everything is fine. This package style installation is also important if you want to allow users to upgrade their firmware on their cameras.

[MedicineYeh]

Update: A repo building up the whole guest image including the beta-software is always necessary. I think that could be something like a full test. We can have another repo, let's say beta-software-image which builds up axiom-beta-env as well as microzed-image. Then, it runs the QEMU and clone beta-software in the guest system. Configure it and build it in guest system. Finally, save all the changes in the image. In short, this new repo generates a prebuilt image for deployment.

Or we can simply use the current solution in beta-software/builld_tools with simply some refactors of the build scripts.

Both of them will work.

[anuejn]

A repo building up the whole guest image including the beta-software is always necessary.

I completely agree. But I think, that the repository where the code lives is a good place for the build tools as well. This lowers the hurdles of building the code...

Or we can simply use the current solution in beta-software/builld_tools with simply some refactors of the build scripts.

This is what i think is most useful.

[anuejn]

All in all, i think, that you are mostly thinking about providing a development environment for the image and the axiom-tools, I am mostly thinking about creating deployable images via CI. There is definitly some things that are shared between these two use cases but the main code should be kept seperated from each other. I think you are more on a route to create a solution for https://lab.apertus.org/T738 and quite like it :).

However, from a CI point of view, I think, that the current aproach for building the image is quite ok. Wehn we wanted to use the build_tools/ in a development setup, they would definitly need some refactoring and, for example, checks, wether the previeous stages already ran.

[MedicineYeh]

Yeah, I agree. I looked through your script again and feel it's better to leave it like this since it's more like your current flow of CI and it's also easy to read. Also, it's more like instructions can be shown on a wiki page.

I made some minor changes to make some parts of the code cleaner. Also, I found one problem with inside/update.sh. You used git pull to ensure the build is the latest one while used rsync to copy the whole directory to /opt/beta-software in outside/build_rootfs.sh. The git pull will likely fail if the directory is dirty. I feel it's enough to not git pull again since local changes of files have higher priority, normally.

[anuejn]

You are completely right... My intention was, that update.sh can also be run by the users of the camea to update the apertus tools... But you are right, that local changes should have higher priority. Mabybe we could add a check, wether we do a fresh install or add a flag to the update tool, to skip the pull?

[MedicineYeh]

I guess the simplest solution is making this directory read-only in the guest image which is a common assumption to installed packages. With this assumption, we can safely remove and overwrite any changes made by users if they run update.sh in the guest env.

We can also add an extra detection in update.sh to test if the git source files are dirty. If so, stop doing anything and inform the user to git stash or clean up the directory. This solution works fine with CI build since there are no modifications to the sources. (should be)

As for detecting dirty condition, this command might help $(git diff --shortstat 2> /dev/null | tail -n1). It prints out the changes to stdout. (empty string when the repo is clean)

[anuejn]

We can also add an extra detection in update.sh to test if the git source files are dirty. If so, stop doing anything and inform the user to git stash or clean up the directory. This solution works fine with CI build since there are no modifications to the sources. (should be)

👍 I would vote for this totally

[MedicineYeh]

Let's forget about all the complex stuffs and begin a new repo hierarchy design. I just finished up the design (comments and descriptions are coming on the way.) You can give it a try!! Let's now assume you are a newbie contributor and wants to experiment the work....

NOTE: Please run this command before doing the following commands. (If you reboot the computer, the binfmt will not be there. Don't worry about messing around your settings on your computer.)

sudo docker run --rm --privileged multiarch/qemu-user-static:register

Day 1

The first step you do is to run (without pull is fine):

docker run --rm -it medicineyeh/axiom-sim

Woala!! I'm running AXIOM beta environment in bash!!

Day 1.5

I just edit a hello.c. Let's try compile and run it in our AXIOM-beta env.

alias dr="docker run --rm -it -v $(pwd):/workdir -w /workdir medicineyeh/axiom-dev"
dr gcc ./hello.c -o ./hello
dr ./hello

Cool!!

Day 2

Cool!! I can upgrade the system with Apertus's new releases!!!

docker run --rm -it medicineyeh/axiom-sim -Ssy axiom-beta

NOTE: This command update the package database(information) and then list the packages in the package group - axiom-beta.

Day 3

I'm now trying to run the full simulation including the hardwares (FPGA IPs). (I assume you have the axiom.img in your computer.) I'm now in the directory with ./axiom.img.

docker run --rm -it -v $(pwd)/axiom.img:/root/axiom.img medicineyeh/axiom-qemu:axiom-beta

Fantastic!!!

NOTE: This is not done completely. It only has the QEMU without FPGA simulation in the current version.

---

Day N

I'm now trying the contribute in some part of the flow. Go to https://github.com/axiom-PR-playground, it's organized in a very simple manner!! Just use the previous techniques, I can develop these repos separately without having to manage them all on my computer! It's super easy to build and run!!! (Day 2 and Day 3)

P.S. AXIOM-beta and accelerators are still not touched yet. I will find some time to make them fit into the proposed architecture. P.S.2 AXIOM-beta-docker can utilize the CI with cron jobs to build image weekly/monthly and push to Dokcerhub for package upgrades.

[MedicineYeh]

Oops, I forgot to mention that I'm still working on the image assembler. All the changes made in the docker container can be exported to a tar file and used it to build the image with partitions. i.e. The root filesystem of a docker container can be exported easily. That means all the developments can be done inside the container and we can used it to build the real image easily. In addition, building custom kernel modules is now more robust. (you build on the real env, not the host system with separate linux directory and headers.)

[anuejn]

looks good. you made quite some improvements over the current setup. I really like that some things now are proper arch linux packages, which is better for distribution and more robust overall. Also the release system with github releases is nice and it is a big step forward to be able to deliver binary artifacts. We also have one (single) docker container for building (https://github.com/anuejn/axiom-build-container ). I don't know, wether multiple docker containers are much better. It is more complex to maintain, easier for small tests but more difficult for big changes compared to the current chroot solution.

What is also missing is multi device support: The current build system can create images for the micro (https://github.com/axiom-micro/ ) and the beta from the same source tree, which is quite handy.

I am still not a huge fan of seperating out the development environment, since it makes the "application development" easier at the cost of making the integration more complex. I think (at least at this stage), that for the Beta this is not the right thing to do, because there is not much "application development" without integration (ie. fpga or linux kernel work) going on.

[MedicineYeh]

I think the proposal here can fit your needs as well. It's very flexible after decoupling the components in the original centralized environment.

First

multi device support is super easy. It's just a matter of how.

1. Multi-repo for multi-arch supports. (This is suitable if we may have many architectures.)
2. We can rename the configs folder to be the arch name like you did. It doesn't harm anything.
3. Docker container can have different tags for different architecture. ex: axiom-sim:beta and axiom-sim:micro. It's the same thing, the simulation env but with different arch-dependent prebuilt files.

Second

For system integration, it's much faster/easier to test since not all the softwares are rebuilt again and again for no updates. In addition, each part of the engineer can participate in their role without worrying about other components.

For example, the QEMU simulation developer only needs to focus on the Dockerfile of axiom-qemu. Other images, software stacks, etc. doesn't really matter to him. The only thing he cares is whether all that softwares runs on the simulation environment; not to develop/build/compile them.

Second example, the hardware driver developers can run the QEMU docker directly and easily with docker-compose. All the developer needs to do is to modify the config file for docker-compose. With a single and simple command, the TFTP root file system, QEMU and FPGA simulation containers can be run easily. All the complex stuffs are now being layered with docker.

Please refer to here on how to set up co-simulation. And now it can be done by making them into containers and integrate them with docker-compose. No more complex scripts, it's all in standard config files; easy to read/modify.

Third

For kernel driver developer, they finally does not need to cross-compile which is hard to write and test. They can run make inside the simulation, which is the exactly same environment the kernel modules will be being executed. It's much easier for them since they do not need to worry about the compiler version and build their own kernel headers on their computer.

Forth

To answer you the question whether to have a single container or multiple small containers. Docker is now entering the era of application driven. It's actually pretty nice. In this modern usage, each docker container only takes part of a specific application. It's not a good suggestion to have a docker container runs the environment. It should be the application.

User does not need to wait for a long download time. CI can also build projects much faster without long waiting. I still remember how long it takes to download the prebuilt image (4GB!!!!).

Also, there are a lot of benefits. We can have all presets in the container with tag. For example axiom-qemu:axiom-beta contains the prebuilt dtb, u-boot. Their source codes and settings are not needed for developers at all. We never change the settings anyway!!!!!!!!! REALLY!!! For example, we never have a situation that we need to modify and hot-fix u-boot. Even if that happens, it's never the job of the people who are writing FPGAs anyway!!! Second, the DTB file provided to QEMU is only used for setting up the QEMU. It's never passed to linux!!! The one that really does the job is in /boot/devicetree.dtb. That's why there's no reason to expose these settings and files to the developers. We never change them anyway. And that's also why we should have multiple containers for various purposes. We only need one or two more container types and that's all.

Fifth

axiom-sim container is not just a simulation environment to the user. It's not set up completely yet. It should be the final root filesystem!! In other words, it can also help system integration easier by having a Dockerfile. No more host and __in_chroot scripts. They are now in either file patches (additional__ folder) or being written in Dockerfile. It's much more elegant.

In addition, any one can modify their downloaded container. After that, I'm planing (hoping) we can have a docker-compose config file that utilize the existing root filesystem to be the TFTP root so that users can run the QEMU simulation with their own axiom-sim version. No more image packing and dd in development. Superb!

You can run this command to export the root file system into a tar file.

docker run --name output --rm -d medicineyeh/axiom-sim /bin/sleep 1000 && docker export output -o ./output.tar && docker kill output

[MedicineYeh]

After a second review, I'm starting to know what you say about the big changes. I guess what you mean is something like someone develop the FPGA and kernel driver and some user applications at the same time. It's not easy in my proposal indeed. The development flow will be.

1. Make changes to VHDL/Verilog.
2. Run axiom-qemu with some other helps (containers or configs) for full HW simulation.

Since all the other source codes are not changed. We can evaluate HW changes without rebuilding other things.

3. Make changes to the driver.
4. Run axiom-qemu with some other helps (containers or configs) to test the correctness of the new driver.
5. Make changes to the user application
6. Run axiom-qemu with some other helps (containers or configs) to test the correctness of the new software.

The second flow is when we change VHDL/Verilog, driver, applications at the same time.

1. Make changes to all the HWs/SWs
2. Run some script to compile HWs/SWs (Use axiom-dev to build SWs in the target env. Use Xilinx tools to build HWs. Finally use a script to package all of them.)
3. Run axiom-qemu

It looks similar to what we have now!! The magic here is "some other helps" and "scripts". With the proposal, we can connect various development flow with a clear boundary. This is the best part of the proposal. There is no learning curve on the WHOLE building system. You can focus on what you develop from the beginning to the end. I have to say there is nothing simpler than modify codes and run a magic script. I believe this is what we have now in this repo. The main point here is how hard it is to write that magic script to do all the integrations. I believe what I proposed previously few months ago, the build system/environment, is not an easy thing to understand. To be honest, there's no one could maintain that env except me. That's not good as an open source framework. If you look carefully on the current proposal. You will find that there is no complex scripts and chaining of scripts. Everything script is shallow in terms of codes. There is no complex detection on edge cases since things are containerized, packed in a standard form. Finally, there is no tough uses of linux commands that only a portion of developers can understand what it does. All these benefits lead to the possibility for the new developer, who downloaded the codes, to write their own scripts for their needs.

By the way, if using the build system is the favor, why don't we choose peta-linux? It's the Xilinx provided development environment for their platform. For me, two reasons I don't like that. 1. Too big. It takes a lot of time to download. 2. Too complicated between the connections of each part of the system, I need to dig into the codes (no enough documents) in order to make some variable changes and achieve what I want. It's painful and time-consuming on meaningless learning. (Tuning the knobs can't apply to other future works.)

Take home question: Which one will be your choice?

• Use your script to mount several directories and chroot
• Simply run docker run

They both achieve the goal of running guest env on you computer. (This is the point of standardization.)

[anuejn]

For system integration, it's much faster/easier to test since not all the softwares are rebuilt again and again for no updates.

The current setup accomplishes this using gnu makes partial rebuilds.

For kernel driver developer, they finally does not need to cross-compile which is hard to write and test. They can run make inside the simulation, which is the exactly same environment the kernel modules will be being executed.

The same happens if you use the current emulation in this repo. Just type 'makefiles/docker_make.sh qemu-shell'

how long it takes to download the prebuilt image

Hm... actually, they should be compressed. I think they are way less than one GB.

They both achieve the goal of running guest env on you computer.

You are right. We already opted for creatig a containerized build environment. The main difference is, that you popose to use dockerfiles, where we use makefiles. Personally I dont really see the advantages of making heavier use of docker yet.