Building QEMU on ArchLinux

[BAndiT1983]

Not really an issue, but placed here for reference. Should be moved to more fitting place later. Also scripts have to be created, of course.

Required packages: pkg-config, python2, flex, bison

Configure: ./configure --target-list="aarch64-softmmu,microblazeel-softmmu" --enable-fdt --disable-kvm --disable-xen --python=python2

Make: sudo make -j4 CFLAGS=-Wno-error

Note: CFLAGS have to be set, as new GCC version is very picky

[MedicineYeh]

Hi, I just wonder if this repo alive for your project? I saw some discussions in your forum. If so, I can help you to set up (rewrite) all the scripts so that it can run on most OSs (Ubuntu, CentOS, Fedora, ArchLinux). In addition, I can enhance the features of this project for developer convenience, such as auto-completion (bash and zsh) and image management system of QEMU, etc. If this repo is still alive, I might be able to help you fulfill some functions.

[se6astian-pi]

Sounds good, please do :)

[MedicineYeh]

Cool!! I will make a PR at the end of the week.

[se6astian-pi]

Please note the progress being made on this front right now here: https://github.com/apertus-open-source-cinema/beta-software/pull/7

[MedicineYeh]

It looks nice! Much better than before. I have some questions and want to confirm with you while I'm doing the coding. In my personal experience, this repo is a bit redundant when beta-software is available. Also, it's good that I don't have to add CI myself. Thanks! I'm thinking about redefining what this repo meat to be or even deprecating this repo. I have built similar deployment before. My arrangement style would be like:

├── externals
│   ├── linaro-gcc
│   ├── ...(other 3rd party tools)
├── Xilinx QEMU
│   ├── ... (many files)
│   └── build (A separate build directory of QEMU)
├── install.sh (Install **ALL** required packages, including those required to build Linux kernel, AXIOM libraries)
├── prepare_all.sh (A handy script to do all the preparations and builds, except installing system packages and downloading the guest images.)
├── download.sh (Download all (prebuilt or not) guest images.)
├── image_manager.sh (The main script to manage the guest images and execution of QEMU)
├── run_image.sh (The main script to start a QEMU emulation)
└── guest_images (Use directory to isolate different versions of kernels/AXIOM libs/AXIOM binaries
    ├── dev
    │   └── AXIOM-beta-linux-4.6
    │       ├── devicetree.dtb (downloaded/generated file)
    │       ├── rootfs.ext4 (generated file)
    │       ├── zImage (generated file)
    │       ├── linux-xlnx.git (downloaded file)
    │       ├── ArchLinuxARM-zedboard-latest (generated folder)
    │       ├── ArchLinuxARM-zedboard-latest.tar.gz (downloaded file)
    │       ├── build.sh (The build script to generate all the files/images of guests)
    │       └── runQEMU.sh (The run script of QEMU)
    └── AXIOM-beta
        ├── devicetree.dtb
        ├── rootfs.ext4
        ├── zImage
        └── runQEMU.sh (stand-alone runQEMU script)

runQEMU.sh would be something like this. It can be run alone without image_manager.sh. However, when you run with image_manager, the $PATH would be set properly to utilize the QEMU built in this project. This is much better than forcing users to build and install QEMU in their system.

#!/bin/bash
SCRIPT_PATH="$(readlink -f "$(dirname "${BASH_SOURCE[0]}")")"

# These are preset of this image
QEMU_ARGS=()
QEMU_ARGS+=(-M arm-generic-fdt-7series)
QEMU_ARGS+=(-machine linux=on)
QEMU_ARGS+=(-kernel "$SCRIPT_PATH/zImage")
QEMU_ARGS+=(-dtb "$SCRIPT_PATH/devicetree.dtb")
QEMU_ARGS+=(-append "root=/dev/mmcblk0p2 ro rootwait rootfstype=ext4")
QEMU_ARGS+=(-drive "if=sd,format=raw,index=0,file=$SCRIPT_PATH/rootfs.ext4")
QEMU_ARGS+=(-boot mode=5)

QEMU_ARGS+=(-m 1024)
QEMU_ARGS+=(--nographic)
QEMU_ARGS+=(-serial /dev/null)
QEMU_ARGS+=(-serial mon:stdio)
QEMU_ARGS+=(-chardev socket,server,nowait,path=qemu.monitor,id=monsock)
QEMU_ARGS+=(-mon chardev=monsock,mode=readline)

QEMU=qemu-system-aarch64
# Running QEMU with custom arguments and arguments passed to this script
# echo "Running command: $(which ${QEMU}) ${QEMU_ARGS[*]} $@"
${QEMU} "${QEMU_ARGS[@]}" "$@"

build.sh would be any form you write as long as it generates all the stuff required by runQEMU.sh. The template of mine is like:

# Contributor: ABC <abc@email>

sources=('http://vserver.13thfloor.at/Stuff/AXIOM/BETA/kernel-4.6.0-xilinx-00016-gb49271f.config')
sha256sums=('2e97f9f66eaaf4833e9961111195798480ccb46f8f0df39ebd651cabc06b3c87')

function pre_install() {
# Download all the required files with sha256sum checking
}

function prepare() {

}

function build() {

}

function post_install() {
# Generate configuration files or copy generated files to the correct locations/names
}

pre_install
prepare
build
post_install

[MedicineYeh]

The instruction to normal users is simple:

1. sudo ./install.sh
2. ./prepare_all.sh
3. ./download.sh
4. ./run_image.sh AXIOM-beta or ./run_image.sh run AXIOM-beta/runQEMU.sh

The instruction to developers is also not complicated at all:

1. sudo ./install.sh
2. ./prepare_all.sh
3. ./download.sh
4. cd guest_images/dev/AXIOM-beta-linux-4.6 and then ./build.sh
5. ./run_image.sh dev/AXIOM-beta-linux-4.6

I guess you can feel that any developer can mess around their different versions of guest images easily with this isolation. Since the whole system relies on runQEMU.sh only. It is much more easily and fast to develop guest applications and having version controls over it.

In addition, the image_manager.sh script comes with auto-completion.

$ ./image_manager.sh <TAB>
list   -- List all existing images
push   -- Push a file/folder into image
pull   -- Pull a file/folder from image
ls     -- List files in image folder
rm     -- Remove file/folder from image
mkdir  -- Make a folder in image
--help  -h  -- Display help message and information of usage.

You can also tab all the way through the file system in image

$ ./image_manager.sh push arch_init.c arm/busybox/rootfs_x86.ext2@/etc/environment<TAB>
environment    hostname       inittab        mtab@          os-release     profile        resolv.conf@   shadow                      
fstab          hosts          inputrc        network/       pam.d/         profile.d/     security/                                
group          init.d/        issue          nsswitch.conf  passwd         protocols      services

Example of ./image_manager.sh list. This would list all image files in guest_image directory.

IMAGE NAME                                TYPE                  SIZE                
rootfs.cpio                               CPIO                  21M                 
arm/busybox/rootfs.cpio                   CPIO                  21M                 
x86/debian/rootfs_x86.ext2                EXT3                  60M                 
x86/busybox/rootfs_x86.ext2               EXT3                  256M                
rootfs_x86.ext2                           EXT3                  60M                 
arm/busybox/rootfs_x86.ext2               EXT3                  60M                 
x86_busybox/rootfs_x86.ext2               EXT3                  257M                
x86/arch/arch_x86.img                     MBR                   14G                 
disk.img                                  MBR                   4.1G

I personally prefer to create another command to honor all the scripts including running QEMU, so things would be something like. (All come with auto-completion system.

1. axiom run AXIOM-beta
2. axiom image list
3. axiom image pull dev/AXIOM-beta-linux-4.6@/etc/resolv.conf ./resolv.conf
4. etc.

Finally, I want to mention that the auto-completion system works much much better on zsh. Though I made it work on bash as well, it doesn't look and behave like zsh. It's.... well...

Oh, I forgot to mentioned that when using run_image.sh (in the root directory) to run QEMU. It runs runQEMU.sh with extra QEMU arguments, things like setting up network, gdb debug, mount an extra disk, etc. These commands are too common and sometimes complicated to tell users. So I wrapped it up in the script.

[MedicineYeh]

So that's the concept of my proposal. I guess it might contain too many changes to be accepted. :P

I didn't mention where to place AXIOM guest libraries and binaries since I think it's gonna be a separate repo focusing on libraries and binaries only. And the placement is not restricted by this architecture. One can simply create symbolic links or copy all the files in the guest_image/dev/AXIOM-beta-linux-4.6 if they want it to be like this. The isolation of image folders brings the flexibility of developing guest anywhere. One can simply leave the hook (runQEMU.sh) in the image folder.

So, any idea on how many things I can do for this repo? I can also follow your style but I found it not easy to extend for future uses.

[MedicineYeh]

By the way, to have your current repo architecture work on this system is as simple as

1. cd guest_image
2. git clone ... (or we can do the first two steps in my functional download.sh)
3. Do what ever steps in the new folder.
4. ./run_image.sh beta-software/.... (Where you place the runQEMU.sh)

Other versions of your past and future projects can utilize the proposed solution as long as it at least contains runQEMU.sh. My scripts will automatically capture all image files without having to specify them.

[se6astian-pi]

In my personal experience, this repo is a bit redundant when beta-software is available. right, indeed.

Please get in touch with https://github.com/jatha to coordinate these efforts. He made a lot of progress over the last couple of days with automated firmware building and qemu https://github.com/apertus-open-source-cinema/beta-software but is more busy now after the weekend. The tech details are over my head here as I have no experience with this but it would be wonderful if you could split the work or complete what is still missing.

[MedicineYeh]

I've followed his latest commits. It looks great and that's also one the reason I think this repo is a bit redundant. I've made the whole thing complete (for now) and you might have a look and give it a try. The install.sh will install the packages depending on OSs. https://github.com/MedicineYeh/axiom-beta-env And by the way, this is my version of building guest image. There is no inside which configures guest image content, e.g. username. This repo is (should be) very robust, and all the steps are done under user privilege. There is no root nor sudo required to run this script. You might take a look. https://github.com/MedicineYeh/microzed-image

Update: I've made them work on Travis CI. So, they should work on any Ubuntu (or some minor package deps).

[anuejn]

As I said in https://github.com/apertus-open-source-cinema/beta-software/issues/30, I think taht this is a great tool and a big step forwards compared to the current qemu scripts.

I think this repo is a bit redundant.

Yeah currently, this repo (axiom-beta-qemu) contains much redundant functionality (ie. building the kernel), but that coud be removed. The core is still important and the ci build process should use the code in this repo to avoid duplication.

Personally, I would vote for replacing the code in this directory with you code, but i am not the maintainer :)

[anuejn]

And by the way, this is my version of building guest image. There is no inside which configures guest image content, e.g. username. This repo is (should be) very robust, and all the steps are done under user privilege. There is no root nor sudo required to run this script. You might take a look. https://github.com/MedicineYeh/microzed-image

I looked quickly at your build path and I think your shell scripting skills are way better than mine ;). I would be very happy, if you try to upstream cahnges to the scripts in the beta-software repo? I would really appreciate you efforts, as I dont have that much time for it currently. Feel free to edit & improve :)

[MedicineYeh]

@jatha Thanks for your feedback. I actually learned many command usages from your builds and your CI scripts are way better than mine, too. I will try to finish up the inside part first before having a PR. And by the way, most importantly, the partition table changed (starting from 2048, default) because I'm not quite sure why using 8 sectors. The image downloaded from pulsar8-avnet-microzed-7020 uses 2048 as well. Also, the uImage and uBoot are still missing parts to have something really work on microzed. That will be kind of hard to me. I don't have a real hardware to test and debug.

As for upstreaming the changes, since your script works, I assume mine will also work without having the build env. Is that okay to you? The problematic binaries are sfdisk and mke2fs, i.e. mkfs.ext4. In your and mine builds, the scripts use features that are not supported in older versions. That is also why I put these two in my build environment repo and detect whether the host system contains valid binaries. In my Travis CI script of building guest images, I actually downloaded the build env repo first and build up the binaries before building the image.

[se6astian-pi]

Personally, I would vote for replacing the code in this directory with you code, but i am not the maintainer :)

I guess I am the maintainer (doh!) but since you are way more into this I trust your judgement and will be happy to merge any pull requests that go into this direction.