Reproducible Conversion For Overlaybd Format is not Working on Userspace Convertor

[estebanreyl]

What happened in your environment?

I tried running the user space convertor from a container and I am noticing that the outputted layers for the same image are different every time. I have yet to verify if this happens for every image, but below is my repro using python:3.8 at least.

What did you expect to happen?

I expected converted layers for the same image to output identically between runs.

How can we reproduce it?

I made a simple script to aide with reproducibility which includes an image I am seeing this issue for and converts it four times then outputs the resulting descriptor. I am unclear yet if this happens for other images/why it occurs. The output will show the descriptor of the different images. I also output the manifests in the results folder for easier comparisons. (The script can be run as is with no extra parameters or changes)

#!/bin/bash
# This script outputs the results of converting the same image multiple times given a specific overlaybd file.
# Prerequisites:
# - Docker
# - ORAS CLI

os="mariner"
version="4eee2a3b590a6a25b19f331cf17c3b5a1051befc"
outputDir="./results/$os"
manifestDir="manifests"

# Setup local registry
docker pull registry
docker run --entrypoint htpasswd  httpd:2 -Bbn testuser testpassword > auth/htpasswd
docker run -d -p 5000:5000 --name registry registry:latest
docker build -f ./$os.Dockerfile --build-arg OVERLAYBD_VERSION=$version -t userspace-convertor:$os .

# Do multiple conversions of the same image
registry="docker.io/library"
repository="python"
tag="3.8"

rm -rf "$outputDir" # Cleanup old instances
oras cp "$registry/$repository:$tag" localhost:5000/$repository:$tag --to-plain-http # Add Image to local registry

$repository:$tag > $outputDir/descriptors-obd.log
for i in $(seq 1 4);
do
    mkdir -p "$outputDir/$manifestDir/$i"
    echo "docker run userspace-convertor:$os ./bin/convertor --repository $registry/$repository -u testuser:testpassword --input-tag $tag --oci --overlaybd $tag-obd"
    docker run --network=host userspace-convertor:$os ./bin/convertor --plain --repository "localhost:5000/$repository" -u testuser:testpassword --input-tag "$tag" --oci --overlaybd "$tag-obd"
    oras manifest fetch --pretty localhost:5000/$repository:$tag-obd > $outputDir/$manifestDir/$i/$repository-$tag-obd.json
    oras manifest fetch --descriptor --pretty localhost:5000/$repository:$tag-obd >> $outputDir/descriptors-obd.log
done
cat $outputDir/descriptors-obd.log

The above script also requires the mariner.Dockerfile:

FROM mcr.microsoft.com/cbl-mariner/base/core:2.0 AS base
# Required Build/Run Tools Dependencies for Overlaybd tools
RUN yum install e2fsprogs-devel -y && \
    yum install libaio-devel -y && \
    yum install ca-certificates -y

# --- OVERLAYBD TOOLS ---
FROM base As overlaybd-build
RUN yum install -y libaio-devel libcurl-devel openssl-devel libnl3-devel e2fsprogs-devel glibc-devel libzstd-devel binutils ca-certificates-microsoft build-essential && \
    yum install -y rpm-build make git wget sudo tar gcc gcc-c++ cmake && \
    yum install golang -y

ARG OVERLAYBD_COMMIT
RUN git clone https://github.com/containerd/overlaybd.git && \
    cd overlaybd && \
    git submodule update --init && \
    git checkout $OVERLAYBD_COMMIT && \
    mkdir build && \
    cd build && \
    cmake .. && \
    make -j && \
    make install

# --- BUILD LOCAL CONVERTER ---
FROM overlaybd-build AS convert-build
WORKDIR /home/limiteduser/
RUN git clone https://github.com/containerd/accelerated-container-image.git
WORKDIR /home/limiteduser/accelerated-container-image
RUN make

# --- FINAL ---
FROM base
WORKDIR /home/limiteduser/

# Copy Conversion Tools
COPY --from=overlaybd-build /opt/overlaybd/bin /opt/overlaybd/bin
COPY --from=overlaybd-build /opt/overlaybd/baselayers /opt/overlaybd/baselayers

# This is necessary for overlaybd_apply to work
COPY --from=overlaybd-build /etc/overlaybd/overlaybd.json /etc/overlaybd/overlaybd.json

COPY --from=convert-build /home/limiteduser/accelerated-container-image/bin/convertor ./bin/convertor
CMD ["./bin/convertor"]

What is the version of your Accelerated Container Image?

accelerated-containers: 5a4b6635df9cf8aa5eb986b718224edf1f598cf7 (latest as of 7/19/2023) overlaybd: 4eee2a3b590a6a25b19f331cf17c3b5a1051befc (latest as of 7/19/2023)

What is your OS environment?

Mariner (Docker Container)

Are you willing to submit PRs to fix it?

• [X] Yes, I am willing to fix it.

[WaberZhuang]

Have you tried converting the same image using a userspace converter without going through the container? Does it have the same problem? I think something may have gone wrong causing overlaybd-tools to not properly link our customized libe2fs when using it through a container.

[WaberZhuang]

I think it's the addition of the following sentence to # Copy Conversion Tools that solves the problem.

COPY --from=overlaybd-build /opt/overlaybd/lib /opt/overlaybd/lib

[estebanreyl]

I added the line as mentioned and things now work! I think that makes sense, if that is not included there is a fallback to the installed libe2fs which I suppose breaks reproducibility in this instance. I don't remember this being an issue before, but I realize it might be one moving forward. Ill open a little PR to note this detail on the user space convertor docs. https://github.com/containerd/accelerated-container-image/blob/main/docs/USERSPACE_CONVERTOR.md#libext2fs

[estebanreyl]

Resolved