Open nathanpeck opened 2 years ago
Hi @nathanpeck Thanks for creating the issues.
To me issue looks duplicate of https://github.com/containers/buildah/issues/3866 and podman push --digestfile
should be useful for such use-cases.
Thanks @flouthoc. For more context (which I should have put in the first post) I am helping the team building AWS Copilot which is our official command line tool for building, pushing, and deploying images on AWS.
We have some customer requests to support Podman. Unfortunately right now our tool does a docker build
then docker push
and then uses the docker inspect
command to grab the image digest of the locally built image. It then uses that image digest as the "canonical" reference to the image when launching it in AWS App Runner or AWS Elastic Container Service. This works great with Docker Desktop because the local digest is the same as the remote digest, but the flow breaks when switching to Podman because the digests don't match up between local and the remote registry.
Our goal is to figure out a workaround where Podman can function as a drop-in replacement for Docker within our tooling, and right now this digest behavior difference is a blocker. It looks like right now the only way to fix this is to have our tooling build and push, then delete the local Docker image that was built by Podman and then repull off the registry to get the "real" digest. Or stop using the local digest value entirely and only rely on the remote digest from ECR.
It looks in the linked issue like --digestfile
won't help because they say "the --digestfile argument, prints out the 'unuseable' digest, so is not very helpful in the instance".
@nathanpeck Thanks for the detailed explanation i think a solution decided here https://github.com/containers/buildah/issues/3866 could be useful for your use-case where proposed solution is that on push the remote digest will be added to local storage so that it can be useful for regular podman commands.
Same problem with podman 4.1.1 and dockerhub, similar use case - need to have unique canonical reference to image after it's pushed to dockerhub registry
@mtrmac @vrothberg PTAL
Everything @flouthoc said is accurate.
Actually --digesfile finally did worked for me. But it was just learning scenario. And it really feels weird that calculation of "digest" is so unstable. If I'll need to use unique reference for several different registries then I it can get pretty messy.
That’s the trade-off: the digest is much more useful for security / attribution when it includes the compression representation, but that requires care while designing a workflow to preserve that value.
If there are multiple registries, it would almost always be better to podman push
only once (compressing the data only once, which is quite time-consuming), and then use skopeo copy --preserve-digests
to distribute the created image.
A friendly reminder that this issue had no activity for 30 days.
I'm facing the same problem. The only workaround I found (except going to registry and figuring out there) is to pull image from registry and then check SHA256. Interesting thing - after pulling image it has 2 different SHAs when I run inspect
@mtrmac should we think of a podman push --preserve-digests
?
No, that would force us to push an uncompressed image, which users typically don’t want.
podman push --digestfile
is the right approach, and I think the only right approach. An image does not have a single digest, it may have an unlimited number of different representations, so podman push && podman inspect
is fundamentally incorrect.
Consider that the image may be pushed to registries with different capabilities (e.g. an OCI image might be pushed to a non-OCI registry and might need to be converted), or it may be differently compressed (to one registry with zstd, to one with gzip), or there may be different gzip-compressed versions of the same layer (yes that happens in practice), and when pushing to two registries, registry A might already contain one version, and registry B might already contain another version; in that case podman push
, if it knows about those pre-existing layers, uses those pre-existing layers instead an expensive compress+upload.
So it is perfectly possible that
podman push --digestfile digest1 registry1.example.com/repo
podman push --digestfile digest2 registry2.example.com/repo
results in two different digits in digest1
and digest2
(even without any extra options, but also with extra options like --format
). Which one of these digests should podman inspect
show in the “digest” column? Would that potentially change after each push? How does any of this work reliably with multiple concurrent users / scripts creating images on the same computer?
Just use --digestfile
. Unfortunately that might result in a digest that is not resolvable locally (as tracked in https://github.com/containers/buildah/issues/3866 ) but it does reliably work for finding the image on the registry where it was pushed to.
I don’t see this as a case of whether it’s “fundamentally incorrect” but rather a case of whether you want to be drop in compatible with Docker. If you want to be drop in compatible with Docker then you have to fix this behavior to match Docker. If you say that we need to add a flag and use a digest file workaround then that’s not Docker compatible, which is okay as long as that’s what you are going for with podman.
Fundamentally incorrect or not, this issue is asking for the default behavior to match Docker CLI
On Mon, May 15, 2023 at 3:02 PM Miloslav Trmač @.***> wrote:
No, that would force us to push an uncompressed image, which users typically don’t want.
podman push --digestfile is the right approach, and I think the only right approach. An image does not have a single digest, it may have an unlimited number of different representations, so podman push && podman inspect is fundamentally incorrect.
Consider that the image may be pushed to registries with different capabilities (e.g. an OCI image might be pushed to a non-OCI registry and might need to be converted), or it may be differently compressed (to one registry with zstd, to one with gzip), or there may be different gzip-compressed versions of the same layer (yes that happens in practice), and when pushing to two registries, registry A might already contain one version, and registry B might already contain another version; in that case podman push, if it knows about those pre-existing layers, uses those pre-existing layers instead an expensive compress+upload.
So it is perfectly possible that
podman push --digestfile digest1 registry1.example.com/repo podman push --digestfile digest2 registry2.example.com/repo
results in two different digits in digest1 and digest2 (even without any extra options, but also with extra options like --format). Which one of these digests should podman inspect show in the “digest” column? Would that potentially change after each push? How does any of this work reliably with multiple concurrent users / scripts creating images on the same computer?
Just use --digestfile. Unfortunately that might result in a digest that is not resolvable locally as tracked in containers/buildah#3866 https://github.com/containers/buildah/issues/3866 ) but it does reliably work for finding the image on the registry where it was pushed to.
— Reply to this email directly, view it on GitHub https://github.com/containers/podman/issues/14779#issuecomment-1548405346, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA2VN6R4EDWLV5G3U4SH7Z3XGJ4U3ANCNFSM52HGDTSQ . You are receiving this because you were mentioned.Message ID: @.***>
(It’s ambiguous on Docker as well, unless that whole machine (VM, not just a container, except for Docker-in-Docker nesting), is dedicated to that push and can guarantee “the same” image won’t be pushed, or pulled, by any other user / process. docker image list
can easily list multiple entries with multiple digests, possibly for the same repo, with no way to tell which one was just pushed. See below how there can be several digest values for “the same” image after a push, with no way to disambiguate.)
I guess that’s a fair point. Actually the issue seems to be larger.
Currently (https://github.com/containers/podman/blob/4c399fc6fb2de70e0a197cd81fc3978d586e23ee/cmd/podman/images/list.go#L221-L248 ) , Podman lists one line per tag, and one line per all untagged references, with a .Digest
set to c/storage.Image.Digest
, which is set at the time of image creation, and ~never modified, and the same for all lines.
Whereas Docker (https://github.com/docker/cli/blob/935df5a59f59d5ee9291627be1b80b3cc7ad2b7e/cli/command/formatter/image.go#L113 ) lists one line per each RepoDigest
(possibly listing the same digest for multiple tags in the same repo).
So one part is that just that the behavior is clearly different here. I don’t know whether we can / want to change that.
Another is that RepoDigests
is not tracked accurately in Podman, IIRC. And yet another, and a specific subset of that, is https://github.com/containers/buildah/issues/3866 , where setting RepoDigests
to include the just-pushed image is a bit of an implementation effort.
@mtrmac is this issue still active?
@rhatdan The comment just above points at code. Did that code change?
Not that I know of, but if this issue can not be fixed should we close it?
Nothing in the above indicates it “can not be fixed”. A lot of that is just work that has not happened.
Also, Podman maintainers have a decision to make, whether Podman should change how images with multiple digests are presented in the UI.
Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)
/kind bug
Description
When a container image is built its local digest SHA as seen from
podman image list <image-url> --digests
does not match with what Amazon ECR reports as the image digest SHA.Steps to reproduce the issue:
Build a container image locally and tag it with an Amazon ECR registry URL:
Push the image up to Amazon ECR:
View the image digest locally:
Get the digest of the uploaded image according to Amazon ECR:
Describe the results you received:
According to podman locally the image digest is:
sha256:b578f272c7a9acef26188559c7c98af971d8724f320600e0a87042d01688f3fa
After pushing this image, according to Amazon ECR the image digest SHA is:sha256:74ea9e8924245a010fd254694b6c7667daab06fee986ac7b29960b2110cdaa6d
Describe the results you expected:
I would expect that the SHA digest would be the same locally as it is on Amazon ECR. Note that when using out of the box Docker Desktop as the image builder both the local image's digest and Amazon ECR end up with the same SHA digest.
Additional information you deem important (e.g. issue happens only occasionally):
After running
podman pull
and thenpodman inspect 209640446841.dkr.ecr.us-east-2.amazonaws.com/node/node-demo
when runningpodman inspect 209640446841.dkr.ecr.us-east-2.amazonaws.com/node/node-demo
the.RepoDigests[0]
entry does match with Amazon ECR's digest. Perhaps Docker Desktop is taking this remote digest value and applying it to the local image digest value upon push?Output of
podman version
:Output of
podman info --debug
:Package info (e.g. output of
rpm -q podman
orapt list podman
):Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/main/troubleshooting.md)
Yes Additional environment details (AWS, VirtualBox, physical, etc.):
Mac OS X, Podman installed via brew, using an AWS account to test upload to Amazon ECR