search

containerd / accelerated-container-image

A production-ready remote container image format (overlaybd) and snapshotter based on block-device.
Apache License 2.0
405 stars 73 forks source link

unable to build images `failed to prepare: parent snapshot does not exist: not found` #248

Closed goller closed 9 months ago

goller commented 10 months ago

What happened in your environment?

Hello! I'm trying to use overlaybd and I'm having problems with parent snapshots not existing.

After following the quickstart guide I am not able to build images via buildkit. In the section below I've outlined all versions and ran to get the following error:

error: failed to solve: failed to prepare sha256:ec9d7d32d86b2d290c7c462c974ad1bb83e2912c8d8963de31b4531a86e4e38e as u464ilm6ong9k3equvlkgb4ng: parent snapshot sha256:ec9d7d32d86b2d290c7c462c974ad1bb83e2912c8d8963de31b4531a86e4e38e does not exist: not found

Any ideas about what this might be?

Here are the overlaybd logs:

2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/image_service.cpp:219|read_global_config_and_set:log config: [log_level=0][log_path=/var/log/overlaybd.log][log_size=10485760][log_num=3]
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/image_service.cpp:331|init:cache config: [cache_type=file][cache_dir=/opt/overlaybd/registry_cache][cache_size_GB=4][refill_size=262144]
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/image_service.cpp:58|create_dir:dir /opt/overlaybd/registry_cache doesn't exist. create succ.
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/image_service.cpp:345|init:create registryfs with cafile:/etc/ssl/certs/ca-bundle.crt, version:v2
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/build/_deps/photon-src/io/epoll.cpp:289|new_epoll_engine:Init event engine: epoll
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/build/_deps/photon-src/io/epoll.cpp:289|new_epoll_engine:Init event engine: epoll
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/image_service.cpp:424|init:use gzip file cache
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/image_service.cpp:58|create_dir:dir /opt/overlaybd/gzip_cache doesn't exist. create succ.
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/main.cpp:441|main:blocking netlink
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/main.cpp:444|main:blocking netlink done
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/main.cpp:452|main:resetting netlink
2023/12/06 15:43:13|INFO |th=00000000013053E0|/src/src/main.cpp:454|main:reset netlink done

Here are the overlaybd-snapshotter logs:

Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=info msg="Mounts (key: buildkit/1/q6gznysvl3dywo3vsen6vzr39)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=debug msg="Mounts (key: buildkit/1/q6gznysvl3dywo3vsen6vzr39, id: 9, parentID: [], kind: 2)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=info msg="Mounts (key: buildkit/2/zc0iclr71ye08u8cl8k1nln07)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=debug msg="Mounts (key: buildkit/2/zc0iclr71ye08u8cl8k1nln07, id: 10, parentID: [], kind: 2)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=info msg="Mounts (key: buildkit/1/q6gznysvl3dywo3vsen6vzr39)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=debug msg="Mounts (key: buildkit/1/q6gznysvl3dywo3vsen6vzr39, id: 9, parentID: [], kind: 2)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=info msg="Mounts (key: buildkit/2/zc0iclr71ye08u8cl8k1nln07)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=debug msg="Mounts (key: buildkit/2/zc0iclr71ye08u8cl8k1nln07, id: 10, parentID: [], kind: 2)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=info msg="Mounts (key: buildkit/3/k9hqd8042swl4fdsayn89zv1d)"
Dec 06 16:52:34 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:34Z" level=debug msg="Mounts (key: buildkit/3/k9hqd8042swl4fdsayn89zv1d, id: 11, parentID: [], kind: 2)"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="Prepare (key: buildkit/14/tmp-9ktmesjik025846wwpsb8lqcz sha256:a790f937a6aea2600982de54a5fb995c681dd74f26968d6b7>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="Prepare (targetRefLabel: sha256:a790f937a6aea2600982de54a5fb995c681dd74f26968d6b74286e06839e4fb3, accelLayerLab>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snapshots/22/fs/overlaybd.comm>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by writable_data(sn: 22), try to identify by config.v1.json"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="Prepare (storageType: 1)"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="empty snID get. It should be an initial layer."
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="snapshot R/W label: overlayfs"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="return mount point(R/W mode: overlayfs): [{bind /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snap>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="Mounts (key: buildkit/3/k9hqd8042swl4fdsayn89zv1d)"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="Mounts (key: buildkit/3/k9hqd8042swl4fdsayn89zv1d, id: 11, parentID: [], kind: 2)"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="Prepare (key: buildkit/15/tmp-ft59jy045borvnn0tm0kwets2 sha256:a790f937a6aea2600982de54a5fb995c681dd74f26968d6b7>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="Prepare (targetRefLabel: sha256:a790f937a6aea2600982de54a5fb995c681dd74f26968d6b74286e06839e4fb3, accelLayerLab>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snapshots/23/fs/overlaybd.comm>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by writable_data(sn: 23), try to identify by config.v1.json"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="Prepare (storageType: 1)"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="empty snID get. It should be an initial layer."
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="snapshot R/W label: overlayfs"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="return mount point(R/W mode: overlayfs): [{bind /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snap>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="Walk (fs: [])"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="Remove (key: buildkit/14/tmp-9ktmesjik025846wwpsb8lqcz sha256:a790f937a6aea2600982de54a5fb995c681dd74f26968d6b74>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snapshots/22/fs/overlaybd.comm>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by writable_data(sn: 22), try to identify by config.v1.json"
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=info msg="Remove (key: buildkit/15/tmp-ft59jy045borvnn0tm0kwets2 sha256:a790f937a6aea2600982de54a5fb995c681dd74f26968d6b74>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by /var/lib/containerd/io.containerd.snapshotter.v1.overlaybd/snapshots/23/fs/overlaybd.comm>
Dec 06 16:52:35 ip-172-31-36-217.us-east-2.compute.internal overlaybd-snapshotter[4834]: time="2023-12-06T16:52:35Z" level=debug msg="failed to identify by writable_data(sn: 23), try to identify by config.v1.json"

What did you expect to happen?

No response

How can we reproduce it?

Repro

Started machine in aws us-east-2 running RHEL 9.3 and load target_core_user.

sudo modprobe target_core_user

install requirements for overlaybd

sudo dnf install libaio
sudo dnf install compat-openssl11

install requirements for buildkit

sudo dnf install git
sudo dnf install golang

install overlaybd version 1.0.7

curl -L --output overlaybd-1.0.7-1.el8.x86_64.rpm https://github.com/containerd/overlaybd/releases/download/v1.0.7/overlaybd-1.0.7-1.el8.x86_64.rpm
sudo rpm -ivh overlaybd-1.0.7-1.el8.x86_64.rpm

set logs to debug

edit /etc/overlaybd/overlaybd.json and set logLevel to 0.

    "logConfig": {
        "logLevel": 0,

Start overlaybd

sudo systemctl enable /opt/overlaybd/overlaybd-tcmu.service
sudo systemctl start overlaybd-tcmu

install overlaybd-snapshotter 1.0.2

curl -L --output overlaybd-snapshotter-1.0.2-1.x86_64.rpm https://github.com/containerd/accelerated-container-image/releases/download/v1.0.2/overlaybd-snapshotter-1.0.2-1.x86_64.rpm

sudo rpm -ivh  overlaybd-snapshotter-1.0.2-1.x86_64.rpm

set logs to debug

edit /etc/overlaybd-snapshotter/config.json and set verbose to "debug"

   "verbose": "debug",

start overlaybd-snapshotter

sudo systemctl enable /opt/overlaybd/snapshotter/overlaybd-snapshotter.service
sudo systemctl start overlaybd-snapshotter

install containerd 1.7.10

 curl -L --output cri.tar.gz https://github.com/containerd/containerd/releases/download/v1.7.10/cri-containerd-1.7.10-linux-amd64.tar.gz
cd /
sudo tar zxvf ~/cri.tar.gz
sudo mkdir /etc/containerd

Configure config.toml

# make a default containerd config
sudo /usr/local/bin/containerd config default > config.toml

Edit and add this section.

[proxy_plugins]
  [proxy_plugins.overlaybd]
    type = "snapshot"
    address = "/run/overlaybd-snapshotter/overlaybd.sock"

Edit and update this section:

    [plugins."io.containerd.grpc.v1.cri".containerd]
      default_runtime_name = "runc"
      disable_snapshot_annotations = false
      discard_unpacked_layers = false
      ignore_blockio_not_enabled_errors = false
      ignore_rdt_not_enabled_errors = false
      no_pivot = false
      snapshotter = "overlaybd"
sudo mv config.toml /etc/containerd/

Start containerd

sudo /usr/local/bin/containerd &

Buildkit

Build buildkit

git clone https://github.com/data-accelerator/buildkit.git
cd buildkit
git checkout 202210
go build -o ./bin/buildkitd ./cmd/buildkitd/ 
go build -o ./bin/buildctl ./cmd/buildctl/

Start buildkit

sudo ./bin/buildkitd --containerd-worker-snapshotter=overlaybd --oci-worker=false --containerd-worker=true  --containerd-worker-snapshotter=overlaybd --oci-worker=false --containerd-worker=true --addr="tcp://0.0.0.0:9999" &

create an example dockerfile

mkdir ~/example
cd ~/example

cat > Dockerfile << EOF
FROM overlaybd/ubuntu:0705
COPY Dockerfile /Dockerfile
EOF

Attempt build:

../buildkit/bin/buildctl --addr tcp://127.0.0.1:9999 build     --frontend dockerfile.v0     --local context=.     --local dockerfile=. --progress plain

#1 [internal] load .dockerignore
#1 transferring context: 2B done
#1 DONE 0.0s

#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 151B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/overlaybd/ubuntu:0705
#3 DONE 0.4s

#4 [internal] load build context
#4 transferring context: 90B done
#4 DONE 0.0s

#5 [1/2] FROM docker.io/overlaybd/ubuntu:0705@sha256:a6cfec9b35890c6f615cbd51b027f8244aa9922142a3763083f57e97ac7df1b5
#5 resolve docker.io/overlaybd/ubuntu:0705@sha256:a6cfec9b35890c6f615cbd51b027f8244aa9922142a3763083f57e97ac7df1b5 0.0s done
#5 DONE 0.0s

#6 [2/2] COPY Dockerfile /Dockerfile
#6 ERROR: failed to prepare sha256:ec9d7d32d86b2d290c7c462c974ad1bb83e2912c8d8963de31b4531a86e4e38e as 6olip1fz0ssmloti1nf7gl96r: parent snapshot sha256:ec9d7d32d86b2d290c7c462c974ad1bb83e2912c8d8963de31b4531a86e4e38e does not exist: not found
------
 > [2/2] COPY Dockerfile /Dockerfile:
------
Dockerfile:2
--------------------
   1 |     FROM overlaybd/ubuntu:0705
   2 | >>> COPY Dockerfile /Dockerfile
   3 |
--------------------
error: failed to solve: failed to prepare sha256:ec9d7d32d86b2d290c7c462c974ad1bb83e2912c8d8963de31b4531a86e4e38e as 6olip1fz0ssmloti1nf7gl96r: parent snapshot sha256:ec9d7d32d86b2d290c7c462c974ad1bb83e2912c8d8963de31b4531a86e4e38e does not exist: not found

What is the version of your Accelerated Container Image?

1.0.2

What is your OS environment?

RHEL 9.3

Are you willing to submit PRs to fix it?

liulanzheng commented 9 months ago

@goller This version only support build FROM overlaybd image, and has some compatibility problems with normal OCI images. "overlaybd/ubuntu:0705" is not an overlaybd image. The images under overlaybd are not officially released overlaybd images, only those referenced in the documentation with the "_obd" suffix are overlaybd images. The overlaybd repository contains only some test images and it is recommended to convert your own overlaybd images for build.

We have another new version of buildkit compatible with normal OCI image. We're trying to merge it into mainline buildkit, but it's not progressing well and there hasn't been further follow-up recently. I will be placing this version on github.com/data-accelerator/buildkit.git shortly, and then I would kindly ask you to try it out again.

liulanzheng commented 9 months ago

@goller you may try branch 202312 in github.com/data-accelerator/buildkit.git This version is compatible with standard OCIv1 image builds. When building from an OCIv1 image, the output is an OCIv1 image, and when building from an OverlayBD image, the output is an OverlayBD image. You can determine whether it is built in overlaybd by observing whether the full FROM image is downloaded during the pull.

goller commented 9 months ago

@goller you may try branch 202312 in github.com/data-accelerator/buildkit.git This version is compatible with standard OCIv1 image builds. When building from an OCIv1 image, the output is an OCIv1 image, and when building from an OverlayBD image, the output is an OverlayBD image. You can determine whether it is built in overlaybd by observing whether the full FROM image is downloaded during the pull.

@liulanzheng I will try that out now. Thank you for the update!

goller commented 9 months ago

Hi @liulanzheng the new buildkit branch does indeed work ! Thank you !

I'm excited for this to be in mainline buildkit.