[feat] Recipe compiler

[tulilirockz]

I just made this issue to document the idea of a recipe compiler for this project, similar to Vanilla OS'es Vib. It could be an amazing idea considering it could be supported by Zerolayer for self-sufficient updates without any kind of hosting!

[gerblesh]

I agree, this would need a lot of work to get working, and we would most likely have to move away from shell, to something like go, python, or even rust. I'd say we should probably just stick to python, as it's pretty accessible and is architecture agnostic/interpreted

[xynydev]

I already have plans for this idea, great that it's an issue now so I can report here. I want to use a compiled & strongly typed language so no python. Probably prototype in nimlang but can change later if wanted. My initial idea is to leverage the bling container to house the module's as shell scripts which we can just call with RUN in the Containerfile. This would vastly simplify development.

[mirkobrombin]

Why not join forces and collaborate together in Vib? ☺️

[xynydev]

Why not join forces and collaborate together in Vib? ☺️

They have a different config format and are under a different project, I think some healthy competition is cool and all. Besides, correct me if I'm wrong, but I doubt they'd be willing to support building OS' not built on top of Vanilla. If you present some good counterarguments, I will reconsider though.

[mirkobrombin]

Tbh, even if I named it Vib, it can be used to build images based on top of every Linux distribution, that's why the base key is exposed:

base: whatever/image:branch # <--
name: Whatever OS
id: whatever-id
labels:
  maintainer: Whatever OS Contributors
args:
  DEBIAN_FRONTEND: noninteractive
runs:
  - echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/01norecommends
modules:
  - name: init-setup
    type: shell
    commands:
...

[xynydev]

@mirkobrombin you're the dev for Vanilla? That's cool, didn't get that at first. Our config formats are similar, but vibs looks more clunky, but I'm biased on that ofc. What would be the required additions to vib to make it build images like ours? An rpm-ostree module? I'm sure your work is very fine, but the docs don't really shine.

Can you shed some light on what it would mean to you to join forces?

[mirkobrombin]

I made it for my distribution but it can indeed be used for any other distribution, it's just a compiler which generates Containerfile(s).

New module types can be implemented by adding it here then map it in BuildModule At the end those are just simple parsers. You can add an rpm-ostree parser plus any other format you need.

Vib generates a Containerfile with all the final commands in the format you want. Here you can see a generated one.

[xynydev]

With the approach used by vib, every module would have to be rewritten. It's great to know that vib can be used to build OS' other than Vanilla, but I was kind of thinking of a drop-in tool that leverages work already done for the modules in bash, for the first version at least. This is interesting, and I'm kind of inclined to both contribute docs & modules for vib and do this separately. I quite like the easy extensibility we've got going via bash-based modules.

[mirkobrombin]

I am not here to convince you, I saw the issue and what I got was "let's do the same", that's why I was asking for joining forces.

For my personal knowledge, can you point me to an example of your existing modules?

[xynydev]

For my personal knowledge, can you point me to an example of your existing modules?

https://github.com/ublue-os/bling/tree/main/modules

I am not here to convince you, I saw the issue and what I got was "let's do the same", that's why I was asking for joining forces.

I'm delighted, thank you. I'll definetly look into joining my force with yall's with some PRs and docs. I joined the Vanilla discord. I can't seem to find contributor docs and such, but we'll figure it out.

[xynydev]

Ok,,, I tried this out in a few languages that are usually nice to work with, but with hit roadblocks with cli complexity and inflexible yaml reading. So I thought that why not just make a test in bash, just changing some stuff around build.sh. And I did it! See the compile.sh script in the compiler branch. It's definetly not pretty, and it's definetly not functional (somehow only the yafti module runs, I suspects that's got something to do with how the config is passsed). But I think it kind of proves that this is possible to do this way.

I'll for sure be looking forward to contributing to vib, but not today.

[gerblesh]

completely forgot about vib, thanks for reminding us! I'll have to learn some go as i've never used it before. But it seems like vib would work pretty well for our usecase here

[gerblesh]

mm, so maybe we could make a sort of "intermediary" compiler, which takes the recipe.yml configuration, and turns it into the vib configuration, and then builds that into a Containerfile?

[mirkobrombin]

Your existing shell scripts could be used in vib using the shell module. Take a look here https://github.com/Vanilla-OS/core-image/blob/main/modules/00-vanilla-abroot.yml

PS. the one I linked above is an external module which is loaded using the includes module.

[xynydev]

Your existing shell scripts could be used in vib using the shell module.

Of course. They just take in a json string, which would be somewhat inconvenient.

[tulilirockz]

Even if it were inconvenient, it would be a nice idea to use Vib just for the centralization aspect of it. Like, two relatively big communities working on a project simultaneously would be amazing!

Im sure usability issues could be resolved eventually

[gmpinder]

Hey, was just reading through this and thought I'd tell you I've been working on my own ublue cli tool in Rust. It's in somewhat early stages and I've been using it for my own CI in Gitlab. Currently it's only setup for gitlab CI, but with some additions and some help understanding the GitHub Actions stuff it could work over here too.

https://gitlab.com/wunker-bunker/ublue-cli

[gmpinder]

Also here's my own image repo which follows the current starting point layout with the modules.

https://gitlab.com/wunker-bunker/wunker-os

[xynydev]

@gmpinder Hey, what you're doing is really cool IMO! Do you have a Discord or smth, I'd love a real-time chat to share some thoughts and get on the same page?

[gmpinder]

Yes I do, I'll reach out to you. I'm in the Ublue Discord as of yesterday.

[gmpinder]

I've renamed the tool per a convo with @xynydev . It's now called "BlueBuild". You can find it on crates.io and gitlab.com.

[xynydev]

Thanks to #223, we got this as a side effect lol... https://github.com/blue-build/cli

[xynydev]

I'm deleting the compiler branch now, but here's the script that existed in there:

#!/usr/bin/env bash

set -oue pipefail

RECIPE_FILE="${1-config/recipe.yml}"
CONTAINERFILE="${2-out.Containerfile}"
CONFIG_DIRECTORY="${3-config/}"

IMAGE_NAME="$(yq '.name' "$RECIPE_FILE")"
BASE_IMAGE="$(yq '.base-image' "$RECIPE_FILE")"
IMAGE_VERSION="$(yq '.image-version' "$RECIPE_FILE")"

echo "Building $CONTAINERFILE for $IMAGE_NAME from $RECIPE_FILE $BASE_IMAGE:$IMAGE_VERSION."

MODULE_DIRECTORY="/tmp/modules"
echo "FROM $BASE_IMAGE:$IMAGE_VERSION" > "$CONTAINERFILE"
echo "
COPY --from=docker.io/mikefarah/yq /usr/bin/yq /usr/bin/yq
COPY cosign.pub /usr/share/ublue-os/cosign.pub
COPY --from=ghcr.io/ublue-os/bling:latest /rpms /tmp/bling/rpms/
COPY --from=ghcr.io/ublue-os/bling:latest /files /tmp/bling/files/
COPY --from=ghcr.io/ublue-os/bling:latest /modules /tmp/modules/
COPY modules /tmp/modules/
COPY config /tmp/config/
" >> "$CONTAINERFILE"
echo "
ENV IMAGE_NAME=$IMAGE_NAME
ENV BASE_IMAGE=$BASE_IMAGE
ENV CONFIG_DIRECTORY=\"/tmp/config\"
ENV OS_VERSION=\"\$(grep -Po '(?<=VERSION_ID=)\d+' /usr/lib/os-release)\"
" >> "$CONTAINERFILE"

add_module() {
    MODULE="$1"
    TYPE=$(echo "$MODULE" | yq '.type')
    if [[ "$TYPE" != "null" ]]; then
        # If type is found, that means that the module config
        # has been declared inline, and thus is safe to pass to the module
        # shellcheck disable=SC2028
        echo "echo \"=== Launching module of type: $TYPE ===\" \n\\
        bash \"$MODULE_DIRECTORY/$TYPE/$TYPE.sh\" '$(echo "$MODULE" | tr -d '\n')' \n\\
        echo \"======\" \n\\"
    else
        # If the type is not found, that means that the module config
        # is in a separate file, and has to be read from it
        FILE=$(echo "$MODULE" | yq '.from-file')
        add_modules "$CONFIG_DIRECTORY/$FILE"
    fi
}

add_modules() {
    MODULES_FILE="$1"
    readarray MODULES < <(yq -o=j -I=0 '.modules[]' "$MODULES_FILE" )
    if [[ ${#MODULES[@]} -gt 0 ]]; then
        for MODULE in "${MODULES[@]}"; do
            add_module "$MODULE"
        done
    else
        MODULE=$(yq -o=j -I=0 '.' "$MODULES_FILE")
        add_module "$MODULE"
    fi
}

# shellcheck disable=SC2028
echo "RUN printf '\
#!/usr/bin/env bash \n\
set -oue pipefail \n\
get_yaml_array() { \n\
    readarray "\$1" < <(echo "\$3" | yq -I=0 "\$2") \n\
} \n\
export -f get_yaml_array \n\
$(add_modules "$RECIPE_FILE") 
echo "Done!"' \
> /tmp/build.sh" >> "$CONTAINERFILE"

echo "RUN chmod +x /tmp/build.sh && /tmp/build.sh && \
    rm -rf /tmp/* /var/* && ostree container commit" >> "$CONTAINERFILE"
``