automate building flists

[Omarabdul3ziz]

Introduction

TFGrid is used to run ZMachine (Virtual Machine) instances. that contains all the necessary packages and configurations for booting and running specific services or tasks on a virtualized environment.

flist is short for "File List," which is a cloud image containing a pre-configured solution or application. It allows users to deploy applications on the grid quickly and easily.

There is two types of ZMachine

1. VM

   A VM flist includes its own kernel, This means that the ZMachine can run directly on the hardware, utilizing its own kernel for better isolation and performance.

2. Container

   A container flist does not include its kernel just a minimal rootfs includes the necessary source code to run the application

During boot If ZOS finds the /boot/vmlinuz file, it will use this kernel along with the initrd.img if available and run the ZMachine as VM. If the /boot/vmlinuz file is not found, ZOS will provide minimal kernel and run the ZMachine in container mode. for more details check zos docs

Problem

1. Inconsistency
   • The process of creating an flist involved several manual steps, including writing the Dockerfile, building the Docker image, and converting it to an flist.
   • This manual approach could lead to issues where the Dockerfiles in the tf_images repo might not be up-to-date, or updated Docker images may not have an equivalent flist conversion.
2. Storage inefficiency
   • The 0Hub is designed to optimize storage and network usage by identifying and storing duplicated files in multiple flists only once.
   • However, the previous approach to VM flist creation involved providing the entire flist as an image.raw file. This caused each flist (image.raw) to be unique, even if multiple flists shared common files.

Solution

• A workflow that fully automate the process of creating the flists and storing it on the 0hub on a centralized repos (one for containers and the other for vms)
• Create the VM flist from scratch, with a complete rootfs and with the kernel installed, instead of just providing an image.raw file.

Needed flists

• Container flists (without a kernel)
  • Base (ubuntu22.04) Only contain the base package and configurations, it can be used as a base for other flists Needed packages
  • openssh-server, ufw, wget, curl, vim, ntp
  • config admin password (needed for cloud-console)
  • zinit ( with the needed configurations for ssh, ufw, … )
  • DevEnv (development environment) Base image + some extra packages for development
  • docker, caddy, nginx, node
  • App Solutions that can be deployed from the clients. the base should be Base or DevEnv.
  • peertube, funkwhale, discourse, presearch, …
• VMs flist (with a kernel)
  • Minimal Cloud image + kernal
  • Standard with apt-repos, updated/upgraded + container base packages
  • Full extra drivers ( gpu ) For ubuntu/arch/centos

Approaches

1. Converting from Docker image (for container flists)
   • Generating the Dockerfile form vbuilders This will make it easier to reuse snippets of code (recipe) that is common across multiple Dockerfiles
   • Pushing the dockerfile to tf-images repo which will trigger the workflow to
     • build the docker image and push it to ghcr
     • covert to flist and store on 0hub
2. Building from scratch (for vm flists)
   • an executable script that do the steps illustrated here
     • preparing rootfs from any cloud image + including the needed packages
     • install a kernel
   • tar and push directly to the hub

[Omarabdul3ziz]

@xmonader @AhmedHanafy725 @AbdelrahmanElawady if you have any comments