Open reisman234 opened 1 year ago
We are experiencing the same issue. We have configured stargz-snapshotter with containerd as described in the docs. The supplied pre built images are pulling lazily with no issues (can confirm from logs and also can see the fuse mount), but when pushing the pre built image to our own Docker repository it fails to pull lazily and prints the following error (this happens also when the repository is public, not just private):
Mar 19 22:48:33 ip-10-101-48-29 containerd-stargz-grpc[19605]: {"key":"k8s.io/203/extract-575453287-y9WT sha256:c00e77fc513e8777043d58387a011deb334b3976cc32445e2df11343a670d034","level":"info","mountpoint":"/var/lib/containerd/io.containerd.snapshotter.v1.stargz/snapshotter/snapshots/152/fs","msg":"Received status code: 401 Unauthorized. Refreshing creds...","parent":"","src":"docker.io/dadavan/python:3.10-esgz/sha256:918438fbf26b133ff20f857054d8ef6e7992fce12bd77d7b058ad89d739994c5","time":"2023-03-19T22:48:33.631845435Z"}
Mar 19 22:48:33 ip-10-101-48-29 containerd-stargz-grpc[19605]: {"error":"failed to resolve layer: failed to resolve layer \"sha256:5aadac84e9a45bee7a1bdc7a69c14245c9c2a858c38141a8da6270ae9e463146\" from \"docker.io/dadavan/python:3.10-esgz\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header: failed to resolve target","key":"k8s.io/203/extract-575453287-y9WT sha256:c00e77fc513e8777043d58387a011deb334b3976cc32445e2df11343a670d034","level":"warning","msg":"failed to prepare remote snapshot","parent":"","remote-snapshot-prepared":"false","time":"2023-03-19T22:48:33.902988919Z"}
We have configured CRI based authentication as described here.
containerd config:
version = 2
# Plug stargz snapshotter into containerd
# Containerd recognizes stargz snapshotter through specified socket address.
# The specified address below is the default which stargz snapshotter listen to.
[proxy_plugins.stargz]
type = "snapshot"
address = "/run/containerd-stargz-grpc/containerd-stargz-grpc.sock"
[plugins."io.containerd.grpc.v1.cri"]
sandbox_image = "registry.k8s.io/pause:3.6@sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db"
max_concurrent_downloads = 9
[plugins."io.containerd.grpc.v1.cri".containerd]
# Use stargz snapshotter through CRI
snapshotter = "stargz"
disable_snapshot_annotations = false
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
runtime_type = "io.containerd.runc.v2"
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
stargz config.toml:
# Stargz Snapshotter proxies CRI Image Service into containerd socket.
[cri_keychain]
enable_keychain = true
image_service_path = "/run/containerd/containerd.sock"
stargz process is running with the following flags:
/usr/bin/containerd-stargz-grpc --root=/var/lib/containerd/io.containerd.snapshotter.v1.stargz --config=/etc/containerd-stargz-grpc/config.toml
And kubelet is running with the --image-service-endpoint=unix:///run/containerd-stargz-grpc/containerd-stargz-grpc.sock
flag as instructed.
stargz-snapshotter service has been restarted, followed by containerd and finally kubelet.
Is there anything wrong with our setup or configuration? We really are out of ideas here and would appreciate your help.
Thank you very much
docker.io/dadavan/python:3.10-esgz\": 4 error(s) occurred:\n\t gzip: invalid header\n\t legacy: failed to get footer gzip reader: gzip: invalid header\n\t invalid magic number\n\t gzip: invalid header: failed to resolve target
@Dadavan Looks like the image isn't formatted as eStargz. Could you provide the commands used for creating and pushing the image?
docker.io/dadavan/python:3.10-esgz": 4 error(s) occurred:\n\t gzip: invalid header\n\t legacy: failed to get footer gzip reader: gzip: invalid header\n\t invalid magic number\n\t gzip: invalid header: failed to resolve target
@Dadavan Looks like the image isn't formatted as eStargz. Could you provide the commands used for creating and pushing the image?
In the above case I simply pulled one of the pre built images, tagged it and pushed to my own repo:
docker pull ghcr.io/stargz-containers/python:3.10-esgz
docker tag ghcr.io/stargz-containers/python:3.10-esgz dadavan/python:3.10-esgz
docker push dadavan/python:3.10-esgz
We did also try building our own images but encountered the same problem, in that case the command we used was:
docker buildx build -t bringg/hive:esgz -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true .
@Dadavan
docker
Does using other image tools (e.g. crane
) or other registry (e.g. ghcr.io) solve the issue?
docker buildx build -t bringg/hive:esgz -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true .
This should work even with Docker Hub. Which BuildKit version are you using? BuildKit >= v0.10 is needed.
@Dadavan
docker
Does using other image tools (e.g.
crane
) or other registry (e.g. ghcr.io) solve the issue?docker buildx build -t bringg/hive:esgz -o type=registry,oci-mediatypes=true,compression=estargz,force-compression=true .
This should work even with Docker Hub. Which BuildKit version are you using? BuildKit >= v0.10 is needed.
I tried performing the same steps with nerdctl
(pull, tag, push to dockerhub) and it does work, problem is we can't use nerdctl
in our automation pipeline currently. buildx
version is v0.10.3.
Hi folks, anything new with this?
@hagaibarel, I think there were two different issues in this thread, not sure which one you're looking for updates on:
When using the buildx build
command above and configuring the snapshotter to use my user's docker credentials, I've been able to successfully build and lazily pull eStargz images with a private repo. Happy to help troubleshoot if you're still stuck here.
Hello !!
I'm trying out this Snapshotter (Stargz-Store) service and ended up in a similar situation. All the prebuilt images are being pulled lazily from a public repo and however when I tag, push them to a public repo(say quay.io), the lazy image pull effect is not observed.
Following is the log snippet that has some of the error logs collected from the stargz service
..........
Apr 17 09:18:22 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"info","msg":"Received status code: 401 UNAUTHORIZED. Refreshing creds...","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:93871e503e5d40151edd1057993a543bda0fa35937192a7e9de4104df37b7627","time":"2024-04-17T09:18:22.676680752Z"}
Apr 17 09:18:22 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"info","msg":"Received status code: 401 UNAUTHORIZED. Refreshing creds...","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:476de2241a71380a6e9ba94e28e6c532dc42205815937111ec3127104fe65b10","time":"2024-04-17T09:18:22.686206281Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"error":"failed to resolve layer \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\" / \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"debug","msg":"failed to resolve layer","time":"2024-04-17T09:18:23.489656712Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"error":"failed to resolve layer: failed to resolve layer \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\" / \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","layerdigest":"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16","level":"debug","msg":"error resolving layer (context error: \u003cnil\u003e)","remote-snapshot-prepared":"false","time":"2024-04-17T09:18:23.489745500Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"error":"failed to resolve layer: failed to resolve layer \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\" / \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\": 4 error(s) occurred:\n\t* gzip: invalid header\n\t* legacy: failed to get footer gzip reader: gzip: invalid header\n\t* invalid magic number\n\t* gzip: invalid header","level":"warning","msg":"failed to mount layer \"diff\": \"sha256:6ff5afe634c730e94fd48e220f87d3ff312eb1be72210747b826612d578c4e16\"","time":"2024-04-17T09:18:23.489786958Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"debug","msg":"reusing manifest and config of \"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz\"","time":"2024-04-17T09:18:23.493605228Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"debug","msg":"resolving","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:593237bf049fec6b90aaad6ee3638f51f438872a50a17fe9a05dd27d53cca9cf","time":"2024-04-17T09:18:23.493771980Z"}
Apr 17 09:18:23 ci-ln-2mi86qb-72292-nxmcp-worker-a-dvpcj stargz-store[90060]: {"level":"debug","msg":"resolving","src":"quay.io/svanka/stargz-containers/wordpress:5.9.2-esgz/sha256:4ef55b3a7bb75e3e6be5c864d74087ea573a5c2c180701ec6292fedea2da183e","time":"2024-04-17T09:18:23.494002151Z"}
..........
I'm using cri-o as the container manager for my cluster and I also tried the workaround mentioned by @dmosdallas but didn't work.
@ktock do you have any leads on this?
Hi, I have now a setup with rootless podman and the stargz-store running, which works with the pre-converted-images The stargz-store creates for that pull the following log messages.
But when I pull my own created Image this will not work properly
After that, I pushed one of the pre-converted-images in my private registry and tried to pull that, but it end with similar messages.
Because of the
401 Unauthorized
Error, I did also switch the accessibility of that specific repository to public, but with the same result.Thanks for any help on this!