containerd / accelerated-container-image

A production-ready remote container image format (overlaybd) and snapshotter based on block-device.
Apache License 2.0
408 stars 75 forks source link

User space Convertor hangs during Apply step when run in a container #212

Closed estebanreyl closed 1 year ago

estebanreyl commented 1 year ago

What happened in your environment?

I am trying to run the user space converter within a Mariner container but as of late have been encountering an issue were the userspace convertor will hang on the first apply step and it doesn't seem to ever get past this. I am not entirely sure what changed but at the moment.

What did you expect to happen?

I expected the full conversion process to complete.

How can we reproduce it?

This can be reproduced by building a mariner image with the latest bits. To do so you can use the following dockerfile:

FROM mcr.microsoft.com/cbl-mariner/base/core:2.0 AS base
# Required Build/Run Tools Dependencies for Overlaybd tools
RUN yum install e2fsprogs-devel -y && \
    yum install libaio-devel -y && \
    yum install ca-certificates -y && \
    yum install shadow-utils -y

# --- OVERLAYBD TOOLS ---
FROM base As overlaybd-build
RUN yum install -y libaio-devel libcurl-devel openssl-devel libnl3-devel e2fsprogs-devel glibc-devel libzstd-devel binutils ca-certificates-microsoft build-essential && \
    yum install -y rpm-build make git wget sudo tar gcc gcc-c++ cmake && \
    yum install golang -y

RUN git clone https://github.com/containerd/overlaybd.git && \
    cd overlaybd && \
    git submodule update --init && \
    git checkout b432a80504a34d0a8601029de4dd200fc7ee4bf3 && \
    mkdir build && \
    cd build && \
    cmake .. && \
    make -j && \
    make install

# --- BUILD LOCAL CONVERTER ---
FROM overlaybd-build AS convert-build
WORKDIR /home/limiteduser/
RUN git clone https://github.com/containerd/accelerated-container-image.git
WORKDIR /home/limiteduser/accelerated-container-image
RUN make

# --- FINAL ---
FROM base
WORKDIR /home/limiteduser/

# Copy Conversion Tools
COPY --from=overlaybd-build /opt/overlaybd/bin /opt/overlaybd/bin
COPY --from=overlaybd-build /opt/overlaybd/baselayers /opt/overlaybd/baselayers

# This is necessary for overlaybd_apply to work
COPY --from=overlaybd-build /etc/overlaybd/overlaybd.json /etc/overlaybd/overlaybd.json

COPY --from=convert-build /home/limiteduser/accelerated-container-image/bin/convertor ./bin/convertor
CMD ["./bin/convertor"]

This file can then be run as:

docker build -f run-userspace-convert.Dockerfile . -t userspace-convertor-latest
docker run userspace-converter-latest ./bin/convertor ./bin/convertor --repository <repository> --username <user>:<password> --input-tag <tag> --oci --overlaybd <output-tag>

I've been able to reproduce this with several images, for the run below I used python:3.8 stored in my own registry. (Digest: sha256:b904ee5365f53d19de5ff11c1ce96a616224a55306d55e3cf29a11e969a3a7e0). The result just shows the conversion hanging (it wont advance even after hours) image

Investigation


At this point any help would be appreciated :)

### What is the version of your Accelerated Container Image?

I am using the latest commit for this:
**accelerated-container-image** 5e13470827896d8f7c11264415359c2cada4d5aa
**overlaybd** b432a80504a34d0a8601029de4dd200fc7ee4bf3 
### What is your OS environment?

Mariner (Docker Container)

### Are you willing to submit PRs to fix it?

- [X] Yes, I am willing to fix it.
estebanreyl commented 1 year ago

As an update, I have verified that I can reproduce the above while using an ubuntu base image as well:

FROM ubuntu:latest AS base
# Required Build/Run Tools Dependencies for Overlaybd tools
RUN apt-get update && \
    apt-get install -y ca-certificates && \
    update-ca-certificates

RUN apt update && \
    apt install -y libcurl4-openssl-dev libext2fs-dev libaio-dev

# --- OVERLAYBD TOOLS ---
FROM base As overlaybd-build
RUN apt update && \
    apt install -y libssl-dev libnl-3-dev libnl-genl-3-dev libgflags-dev libzstd-dev && \
    apt install -y zlib1g-dev binutils && \
    apt install -y make git wget sudo tar gcc cmake && \
    apt install -y golang

RUN git clone https://github.com/containerd/overlaybd.git && \
    cd overlaybd && \
    git submodule update --init && \
    git checkout b432a80504a34d0a8601029de4dd200fc7ee4bf3 && \
    mkdir build && \
    cd build && \
    cmake .. && \
    make -j && \
    make install

# --- BUILD LOCAL CONVERTER ---
FROM overlaybd-build AS convert-build
WORKDIR /home/limiteduser/
RUN git clone https://github.com/containerd/accelerated-container-image.git
WORKDIR /home/limiteduser/accelerated-container-image
RUN make

# --- FINAL ---
FROM base
WORKDIR /home/limiteduser/

# Copy Conversion Tools
COPY --from=overlaybd-build /opt/overlaybd/bin /opt/overlaybd/bin
COPY --from=overlaybd-build /opt/overlaybd/baselayers /opt/overlaybd/baselayers

# # This is necessary for overlaybd_apply to work
COPY --from=overlaybd-build /etc/overlaybd/overlaybd.json /etc/overlaybd/overlaybd.json

COPY --from=convert-build /home/limiteduser/accelerated-container-image/bin/convertor ./bin/convertor
CMD ["./bin/convertor"]
estebanreyl commented 1 year ago

As a further investigation note, the issue appears to begin from overlaybd commit fc255f39800ba01e80ae414514ac953ddced4842 onwards.

liulanzheng commented 1 year ago

overlaybd commit fc255f enhanced the destruction process that ignore in previous versions. This enhancement waits some related content to destruct. I think the hang might be caused by some content not being destructed as expected. I will see and find it. you may temporarily use the previous version.