Feature/build incus image in container

[theCalcaholic]

So far, the incus container image is built on the host with the executing (potentially unprivileged) users' permissions. During the build, the following steps are performed (among others)

1. Extract the rootfs.tar to a temporary directory
2. Generate a metadata.yaml for the incus image
3. Package the extracted rootfs directory and the metadata.yaml into a new tarball that can be imported into incus as container image

This, however, causes an issue if the build process is not performed as root. When extracting the rootfs.tar as a non-root user, the ownership is reassigned to the extracting user which has serious security implications for the final image and will cause services to fail.

One solution, is to perform most of the build process inside of a container (where we can use root and arbitrary permissions). That's what's implemented in this PR, however, there are a few drawbacks:

1. The buildimage command/script now requires incus to use the build container. This is just a minor drawback in my opinion and should be fine.
2. In order to have full reproducibility, it would be necessary to build all binaries that are used in the container using buildroot. This means at least bash and tar should be built during the build process and pushed to the container. But maybe that's not even enough and we should build the whole container image inside skiffOS? I'm not entirely sure about the implication of using an image from the linuxcontainers image repository instead (as in my current implementation)

@paralin Please let me know what you think about those drawbacks and also if you see a better solution than using containers

[paralin]

Could you include metadata.yaml in either a post build hook or a root_overlay? Then it will end up in the rootfs.tar.

Or you can edit the .tar without extracting it.

[theCalcaholic]

I'm not sure if I understand you correctly, but the structure of the resulting archive needs to be like this:

incus.tar.gz
|
|--- rootfs (unpacked rootfs.tar)
|
|--- metadata.yaml

I don't see how I could possibly inject the metadat.yaml in the rootfs.tar and get the above structure...

[paralin]

@theCalcaholic two approaches to try:

1. Edit the tar with standard tools.

#!/bin/bash

# Check if input file is provided
if [ "$#" -ne 1 ]; then
    echo "Usage: $0 <input.tar>"
    exit 1
fi

INPUT_TAR="$1"
OUTPUT_TAR="modified_${INPUT_TAR}"
TEMP_DIR=$(mktemp -d)

# Create metadata.yaml
cat << EOF > "${TEMP_DIR}/metadata.yaml"
architecture: "amd64"
creation_date: $(date +%s)
properties:
  description: "Ubuntu 20.04 LTS"
  os: "ubuntu"
  release: "focal"
EOF

# Add metadata.yaml to the new tar
tar cf "${OUTPUT_TAR}" -C "${TEMP_DIR}" metadata.yaml

# Append contents of original tar with modified paths
tar tf "${INPUT_TAR}" | sed 's|^|rootfs/|' | tar --append --file="${OUTPUT_TAR}" --transform='s|^|rootfs/|' -C "$(dirname "${INPUT_TAR}")" -T -

# Clean up
rm -rf "${TEMP_DIR}"

echo "Modified tar created: ${OUTPUT_TAR}

2. Edit with python

import tarfile
import io

def modify_tar(input_tar, output_tar, prefix='rootfs/', metadata_content=''):
    with tarfile.open(input_tar, 'r') as original_tar:
        with tarfile.open(output_tar, 'w') as new_tar:
            # Add metadata.yaml to the root
            if metadata_content:
                metadata_info = tarfile.TarInfo('metadata.yaml')
                metadata_file = io.BytesIO(metadata_content.encode('utf-8'))
                metadata_info.size = len(metadata_content)
                new_tar.addfile(metadata_info, metadata_file)

            # Add all other files with modified paths
            for member in original_tar.getmembers():
                member_data = original_tar.extractfile(member)
                if member_data is not None:
                    member.name = prefix + member.name
                    new_tar.addfile(member, member_data)

# Example usage
input_tar = 'input.tar'
output_tar = 'output.tar'
metadata_content = '''
architecture: "amd64"
creation_date: 1620000000
properties:
  description: "Ubuntu 20.04 LTS"
  os: "ubuntu"
  release: "focal"
'''

modify_tar(input_tar, output_tar, metadata_content=metadata_content)
print(f"Modified tar created: {output_tar}")

Try these and let me know how it goes!

[theCalcaholic]

Just a small update (I didn't find the time to continue working on this until yesterday):

• the tar route doesn't work, because I would have to use --concatenate (not append) and that doesn't seem to support --transform
• the python route doesn't work due to this bug which breaks extracting system images like the rootfs.tar in this case

This I will try next:

• bsdtar or tar-stream (both mentioned in this SO answer). bsdtar is already supported by buildroot as a guest package, so I could probably adapt that to build the host binary
• Find out whatever buildroot uses to build the tar archive in the first place - in theory it should have to solve the same issues that I am facing

[theCalcaholic]

I finally decided to use fakeroot, see #319 :)