search

containers / podman

Podman: A tool for managing OCI containers and pods.
https://podman.io
Apache License 2.0
22.43k stars 2.31k forks source link

podman push does not preserve docker manifest type as of podman v5.1.0 #22976

Closed RichardShiao closed 3 weeks ago

RichardShiao commented 3 weeks ago

Issue Description

When building an image using the format flag, podman build --format=docker produces an image with ManifestType application/vnd.docker.distribution.manifest.v2+json.

Building is working as expected, but pushing is showing new behavior.

As of podman 5.0.3, podman push for this image preserved the type: application/vnd.docker.distribution.manifest.v2+json

As of podman 5.1.0 (also reproduced in latest version 5.1.1), podman push is overwriting the type: application/vnd.oci.image.manifest.v1+json

Is this a bug? Or new design?

Steps to reproduce the issue

Will show steps for both podman 5.0.3 and 5.1.1:

podman 5.0.3 shows expected behavior

$ podman version

Client:       Podman Engine
Version:      5.0.3
API Version:  5.0.3
Go Version:   go1.22.3
Git Commit:   d08315df35cb6e95f65bf3935f529295c6e54742
Built:        Fri May 10 08:37:14 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.0.3
API Version:  5.0.3
Go Version:   go1.22.2
Built:        Thu May  9 17:00:00 2024
OS/Arch:      linux/arm64

$ ls

Dockerfile

$ cat Dockerfile

FROM debian:latest
RUN echo hello

$ podman build --format=docker -t docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3 .

STEP 1/2: FROM debian:latest
STEP 2/2: RUN echo hello
hello
COMMIT docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3
--> b2c9e9e720fa
Successfully tagged docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3
b2c9e9e720fa719d3fac8a300ce895694093a1450e07ea3f7e627511f428c76c

$ podman images

REPOSITORY                                        TAG           IMAGE ID      CREATED             SIZE
docker.intuit.com/personal/rshiao/sample20240610  podman-5.0.3  b2c9e9e720fa  About a minute ago  144 MB
docker.io/library/debian                          latest        f1dc429dc895  4 weeks ago         144 MB

$ podman inspect docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3

[
     {
          "Id": "b2c9e9e720fa719d3fac8a300ce895694093a1450e07ea3f7e627511f428c76c",
          "Digest": "sha256:b6d3eba1828cdaa5999c2f680d82a71c96e86638036a34da4647daaa4821666b",
          "RepoTags": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3"
          ],
          "RepoDigests": [
               "docker.intuit.com/personal/rshiao/sample20240610@sha256:b6d3eba1828cdaa5999c2f680d82a71c96e86638036a34da4647daaa4821666b"
          ],
          "Parent": "f1dc429dc8955ea4e9bbd0049f311ba3c4c3a16cda479c7f1153ea8f125e2639",
          "Comment": "",
          "Created": "2024-06-12T05:33:01.561004692Z",
          "Config": {
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
               ],
               "Cmd": [
                    "bash"
               ],
               "Labels": {
                    "io.buildah.version": "1.35.4"
               }
          },
          "Version": "",
          "Author": "",
          "Architecture": "arm64",
          "Os": "linux",
          "Size": 143651177,
          "VirtualSize": 143651177,
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888/diff",
                    "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/047a6a57d14694cf5d3a69592eaf3f6f5dc4a61883424c9e088450020f8300b1/diff",
                    "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/047a6a57d14694cf5d3a69592eaf3f6f5dc4a61883424c9e088450020f8300b1/work"
               }
          },
          "RootFS": {
               "Type": "layers",
               "Layers": [
                    "sha256:9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888",
                    "sha256:086d5dc958e7907c1afdecdf8ead287faa365d2c20d8c60976f043a0b3510a69"
               ]
          },
          "Labels": {
               "io.buildah.version": "1.35.4"
          },
          "Annotations": {},
          "ManifestType": "application/vnd.docker.distribution.manifest.v2+json",
          "User": "",
          "History": [
               {
                    "created": "2024-05-14T00:39:32.979545676Z",
                    "created_by": "/bin/sh -c #(nop) ADD file:b7c55dda8ded4219a7cdc9fbd91dfb5996c4886d4394b85751c2f956defe3287 in / "
               },
               {
                    "created": "2024-05-14T00:39:33.862724032Z",
                    "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
                    "empty_layer": true
               },
               {
                    "created": "2024-06-12T05:33:01.561348026Z",
                    "created_by": "/bin/sh -c echo hello",
                    "comment": "FROM docker.io/library/debian:latest"
               }
          ],
          "NamesHistory": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3"
          ]
     }
]

$ podman push docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3

Getting image source signatures
Copying blob sha256:086d5dc958e7907c1afdecdf8ead287faa365d2c20d8c60976f043a0b3510a69
Copying blob sha256:9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888
Copying config sha256:b2c9e9e720fa719d3fac8a300ce895694093a1450e07ea3f7e627511f428c76c
Writing manifest to image destination

$ podman image rm docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3

Untagged: docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3
Deleted: b2c9e9e720fa719d3fac8a300ce895694093a1450e07ea3f7e627511f428c76c

$ podman images

REPOSITORY                TAG         IMAGE ID      CREATED      SIZE
docker.io/library/debian  latest      f1dc429dc895  4 weeks ago  144 MB

$ podman pull docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3

Trying to pull docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3...
Getting image source signatures
Copying blob sha256:348f243a425766ce6bcab09436d791a172ce5d243a663f608f7047d9c7169316
Copying blob sha256:91e301773f03e9e0fabc5c177fe6bfe8daf14e992ab816f151692b814ddc462c
Copying config sha256:b2c9e9e720fa719d3fac8a300ce895694093a1450e07ea3f7e627511f428c76c
Writing manifest to image destination
b2c9e9e720fa719d3fac8a300ce895694093a1450e07ea3f7e627511f428c76c

$ podman images

REPOSITORY                                        TAG           IMAGE ID      CREATED         SIZE
docker.intuit.com/personal/rshiao/sample20240610  podman-5.0.3  b2c9e9e720fa  33 minutes ago  144 MB
docker.io/library/debian                          latest        f1dc429dc895  4 weeks ago     144 MB

$ podman inspect docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3

[
     {
          "Id": "b2c9e9e720fa719d3fac8a300ce895694093a1450e07ea3f7e627511f428c76c",
          "Digest": "sha256:d54419757554713d8ab3c79b65ca1455363347b2ae2a89b495478987c7d88eb8",
          "RepoTags": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3"
          ],
          "RepoDigests": [
               "docker.intuit.com/personal/rshiao/sample20240610@sha256:d54419757554713d8ab3c79b65ca1455363347b2ae2a89b495478987c7d88eb8"
          ],
          "Parent": "f1dc429dc8955ea4e9bbd0049f311ba3c4c3a16cda479c7f1153ea8f125e2639",
          "Comment": "",
          "Created": "2024-06-12T05:33:01.561004692Z",
          "Config": {
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
               ],
               "Cmd": [
                    "bash"
               ],
               "Labels": {
                    "io.buildah.version": "1.35.4"
               }
          },
          "Version": "",
          "Author": "",
          "Architecture": "arm64",
          "Os": "linux",
          "Size": 143651193,
          "VirtualSize": 143651193,
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888/diff",
                    "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/047a6a57d14694cf5d3a69592eaf3f6f5dc4a61883424c9e088450020f8300b1/diff",
                    "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/047a6a57d14694cf5d3a69592eaf3f6f5dc4a61883424c9e088450020f8300b1/work"
               }
          },
          "RootFS": {
               "Type": "layers",
               "Layers": [
                    "sha256:9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888",
                    "sha256:086d5dc958e7907c1afdecdf8ead287faa365d2c20d8c60976f043a0b3510a69"
               ]
          },
          "Labels": {
               "io.buildah.version": "1.35.4"
          },
          "Annotations": {},
          "ManifestType": "application/vnd.docker.distribution.manifest.v2+json",
          "User": "",
          "History": [
               {
                    "created": "2024-05-14T00:39:32.979545676Z",
                    "created_by": "/bin/sh -c #(nop) ADD file:b7c55dda8ded4219a7cdc9fbd91dfb5996c4886d4394b85751c2f956defe3287 in / "
               },
               {
                    "created": "2024-05-14T00:39:33.862724032Z",
                    "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
                    "empty_layer": true
               },
               {
                    "created": "2024-06-12T05:33:01.561348026Z",
                    "created_by": "/bin/sh -c echo hello",
                    "comment": "FROM docker.io/library/debian:latest"
               }
          ],
          "NamesHistory": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.0.3"
          ]
     }
]

podman 5.1.1 shows new behavior

$ podman version

Client:       Podman Engine
Version:      5.1.1
API Version:  5.1.1
Go Version:   go1.22.3
Git Commit:   bda6eb03dcbcf12a5b7ae004c1240e38dd056d24
Built:        Tue Jun  4 12:54:07 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.1.1
API Version:  5.1.1
Go Version:   go1.22.3
Built:        Mon Jun  3 17:00:00 2024
OS/Arch:      linux/arm64

$ ls

Dockerfile

$ cat Dockerfile

FROM debian:latest
RUN echo hello

$ podman build --format=docker -t docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1 .

STEP 1/2: FROM debian:latest
Resolved "debian" as an alias (/etc/containers/registries.conf.d/000-shortnames.conf)
Trying to pull docker.io/library/debian:latest...
Getting image source signatures
Copying blob sha256:91e301773f03e9e0fabc5c177fe6bfe8daf14e992ab816f151692b814ddc462c
Copying config sha256:f1dc429dc8955ea4e9bbd0049f311ba3c4c3a16cda479c7f1153ea8f125e2639
Writing manifest to image destination
STEP 2/2: RUN echo hello
hello
COMMIT docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1
--> 72cfa659874a
Successfully tagged docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1
72cfa659874a57990c8f49c9f3b07653849a76db25ea722c0b5839c6e3b011a1

$ podman images

REPOSITORY                                        TAG           IMAGE ID      CREATED        SIZE
docker.intuit.com/personal/rshiao/sample20240610  podman-5.1.1  72cfa659874a  8 seconds ago  144 MB
docker.io/library/debian                          latest        f1dc429dc895  4 weeks ago    144 MB

$ podman inspect docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1

[
     {
          "Id": "72cfa659874a57990c8f49c9f3b07653849a76db25ea722c0b5839c6e3b011a1",
          "Digest": "sha256:8fb3b1b19933db53c3595d2db096d5cedd8e93654aeeda0d4c0b8e15a850b06c",
          "RepoTags": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1"
          ],
          "RepoDigests": [
               "docker.intuit.com/personal/rshiao/sample20240610@sha256:8fb3b1b19933db53c3595d2db096d5cedd8e93654aeeda0d4c0b8e15a850b06c"
          ],
          "Parent": "f1dc429dc8955ea4e9bbd0049f311ba3c4c3a16cda479c7f1153ea8f125e2639",
          "Comment": "",
          "Created": "2024-06-12T06:28:52.905615761Z",
          "Config": {
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
               ],
               "Cmd": [
                    "bash"
               ],
               "Labels": {
                    "io.buildah.version": "1.36.0"
               }
          },
          "Version": "",
          "Author": "",
          "Architecture": "arm64",
          "Os": "linux",
          "Size": 143651177,
          "VirtualSize": 143651177,
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888/diff",
                    "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/62d6c10ab4fd8b96950930b0ce07ca4721ff3bd647a8c92334a1db72266db208/diff",
                    "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/62d6c10ab4fd8b96950930b0ce07ca4721ff3bd647a8c92334a1db72266db208/work"
               }
          },
          "RootFS": {
               "Type": "layers",
               "Layers": [
                    "sha256:9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888",
                    "sha256:304486ee47e4d2aa67bf9153ff3150c380078bf6ddf39a0db036548e3a58a01e"
               ]
          },
          "Labels": {
               "io.buildah.version": "1.36.0"
          },
          "Annotations": {},
          "ManifestType": "application/vnd.docker.distribution.manifest.v2+json",
          "User": "",
          "History": [
               {
                    "created": "2024-05-14T00:39:32.979545676Z",
                    "created_by": "/bin/sh -c #(nop) ADD file:b7c55dda8ded4219a7cdc9fbd91dfb5996c4886d4394b85751c2f956defe3287 in / "
               },
               {
                    "created": "2024-05-14T00:39:33.862724032Z",
                    "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
                    "empty_layer": true
               },
               {
                    "created": "2024-06-12T06:28:52.906013094Z",
                    "created_by": "/bin/sh -c echo hello",
                    "comment": "FROM docker.io/library/debian:latest"
               }
          ],
          "NamesHistory": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1"
          ]
     }
]

$ podman push docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1

Getting image source signatures
Copying blob sha256:304486ee47e4d2aa67bf9153ff3150c380078bf6ddf39a0db036548e3a58a01e
Copying blob sha256:9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888
Copying config sha256:a92b1da67162194db6af591fbccf85d8829e835cb1913509c97da470463c1dc6
Writing manifest to image destination

$ podman image rm docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1

Untagged: docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1
Deleted: 72cfa659874a57990c8f49c9f3b07653849a76db25ea722c0b5839c6e3b011a1

$ podman images

REPOSITORY                TAG         IMAGE ID      CREATED      SIZE
docker.io/library/debian  latest      f1dc429dc895  4 weeks ago  144 MB

$ podman pull docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1

Trying to pull docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1...
Getting image source signatures
Copying blob sha256:9fc0e12d9c1c76fc4e959f26dd2ea331284da2589b5d4210b90e4a8266a4d6c9
Copying blob sha256:91e301773f03e9e0fabc5c177fe6bfe8daf14e992ab816f151692b814ddc462c
Copying config sha256:a92b1da67162194db6af591fbccf85d8829e835cb1913509c97da470463c1dc6
Writing manifest to image destination
a92b1da67162194db6af591fbccf85d8829e835cb1913509c97da470463c1dc6

$ podman images

REPOSITORY                                        TAG           IMAGE ID      CREATED        SIZE
docker.intuit.com/personal/rshiao/sample20240610  podman-5.1.1  a92b1da67162  2 minutes ago  144 MB
docker.io/library/debian                          latest        f1dc429dc895  4 weeks ago    144 MB

$ podman inspect docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1

[
     {
          "Id": "a92b1da67162194db6af591fbccf85d8829e835cb1913509c97da470463c1dc6",
          "Digest": "sha256:ba382018001f50ab1efbacce48206faf5e57e590543f0ceeea28b0375fda3421",
          "RepoTags": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1"
          ],
          "RepoDigests": [
               "docker.intuit.com/personal/rshiao/sample20240610@sha256:ba382018001f50ab1efbacce48206faf5e57e590543f0ceeea28b0375fda3421"
          ],
          "Parent": "f1dc429dc8955ea4e9bbd0049f311ba3c4c3a16cda479c7f1153ea8f125e2639",
          "Comment": "",
          "Created": "2024-06-12T06:28:52.905615761Z",
          "Config": {
               "Env": [
                    "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
               ],
               "Cmd": [
                    "bash"
               ],
               "Labels": {
                    "io.buildah.version": "1.36.0"
               }
          },
          "Version": "",
          "Author": "",
          "Architecture": "arm64",
          "Os": "linux",
          "Size": 143650782,
          "VirtualSize": 143650782,
          "GraphDriver": {
               "Name": "overlay",
               "Data": {
                    "LowerDir": "/var/home/core/.local/share/containers/storage/overlay/9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888/diff",
                    "UpperDir": "/var/home/core/.local/share/containers/storage/overlay/62d6c10ab4fd8b96950930b0ce07ca4721ff3bd647a8c92334a1db72266db208/diff",
                    "WorkDir": "/var/home/core/.local/share/containers/storage/overlay/62d6c10ab4fd8b96950930b0ce07ca4721ff3bd647a8c92334a1db72266db208/work"
               }
          },
          "RootFS": {
               "Type": "layers",
               "Layers": [
                    "sha256:9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888",
                    "sha256:304486ee47e4d2aa67bf9153ff3150c380078bf6ddf39a0db036548e3a58a01e"
               ]
          },
          "Labels": {
               "io.buildah.version": "1.36.0"
          },
          "Annotations": {},
          "ManifestType": "application/vnd.oci.image.manifest.v1+json",
          "User": "",
          "History": [
               {
                    "created": "2024-05-14T00:39:32.979545676Z",
                    "created_by": "/bin/sh -c #(nop) ADD file:b7c55dda8ded4219a7cdc9fbd91dfb5996c4886d4394b85751c2f956defe3287 in / "
               },
               {
                    "created": "2024-05-14T00:39:33.862724032Z",
                    "created_by": "/bin/sh -c #(nop)  CMD [\"bash\"]",
                    "empty_layer": true
               },
               {
                    "created": "2024-06-12T06:28:52.906013094Z",
                    "created_by": "/bin/sh -c echo hello",
                    "comment": "FROM docker.io/library/debian:latest"
               }
          ],
          "NamesHistory": [
               "docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1"
          ]
     }
]

Describe the results you received

As of podman 5.1.0 (also reproduced in latest version 5.1.1), podman push is overwriting the type: application/vnd.oci.image.manifest.v1+json

Describe the results you expected

As of podman 5.0.3, podman push for this image preserved the type: application/vnd.docker.distribution.manifest.v2+json

Was expecting the same behavior in later versions of podman as well.

podman info output

$ podman version

Client:       Podman Engine
Version:      5.0.3
API Version:  5.0.3
Go Version:   go1.22.3
Git Commit:   d08315df35cb6e95f65bf3935f529295c6e54742
Built:        Fri May 10 08:37:14 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.0.3
API Version:  5.0.3
Go Version:   go1.22.2
Built:        Thu May  9 17:00:00 2024
OS/Arch:      linux/arm64

$ podman version

Client:       Podman Engine
Version:      5.1.1
API Version:  5.1.1
Go Version:   go1.22.3
Git Commit:   bda6eb03dcbcf12a5b7ae004c1240e38dd056d24
Built:        Tue Jun  4 12:54:07 2024
OS/Arch:      darwin/arm64

Server:       Podman Engine
Version:      5.1.1
API Version:  5.1.1
Go Version:   go1.22.3
Built:        Mon Jun  3 17:00:00 2024
OS/Arch:      linux/arm64

Podman in a container

No

Privileged Or Rootless

None

Upstream Latest Release

Yes

Additional environment details

Additional environment details

Additional information

Additional information like issue happens only occasionally or issue happens with a particular architecture or on a particular setting

baude commented 3 weeks ago

@nalind do you figure this is on the Podman side or something vendored in?

nalind commented 3 weeks ago

I was able to build and push an image using the provided Dockerfile. If you're using Fedora, like I am, then pushing the built image with podman push produces an OCI-format manifest in the registry because my system is apparently configured to use zstd compression in /usr/share/containers/containers.conf. This compression format isn't something that can be expressed in the docker format, so it ends up being converted to OCI format on the fly. The result can be confirmed by inspecting the image in the registry using skopeo inspect --raw. Pushing with --format=docker writes an image to the registry in the docker format using gzip compression.

If you're not using Fedora, then more diagnostic information would be really helpful here. The full podman info output also provides information and inferences about configuration settings and the kernel version. Pushing with --log-level=debug is also helpful for getting some insight into what the underlying libraries are doing at the time.

@baude, I think the logic controlling what happens here is mostly in containers/image and partially in containers/storage, but I could see this as being an expected consequence of setting an [engine] compression_type in containers.conf.

RichardShiao commented 3 weeks ago

@nalind Thanks for checking! Are you also able to reproduce the old behavior when using 5.0.3 - that it does not change the Docker format to OCI?

Can you confirm if the new behavior as of 5.1.0 is a bug? If we have an image already built in Docker format, then I would expect pushing it without any flags to change the format should still retain the same format. That is what we see in earlier versions of podman.

Or is this behavior by design now to default pushes to OCI format? So to preserve Docker format, we should use --format=docker for both builds and pushes?

Here is my current podman info: $ podman info

host:
  arch: arm64
  buildahVersion: 1.36.0
  cgroupControllers:
  - cpu
  - io
  - memory
  - pids
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.10-1.fc40.aarch64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.10, commit: '
  cpuUtilization:
    idlePercent: 99.98
    systemPercent: 0.01
    userPercent: 0.01
  cpus: 5
  databaseBackend: sqlite
  distribution:
    distribution: fedora
    variant: coreos
    version: "40"
  eventLogger: journald
  freeLocks: 2048
  hostname: localhost.localdomain
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
    uidmap:
    - container_id: 0
      host_id: 503
      size: 1
    - container_id: 1
      host_id: 100000
      size: 1000000
  kernel: 6.8.11-300.fc40.aarch64
  linkmode: dynamic
  logDriver: journald
  memFree: 1334853632
  memTotal: 2044608512
  networkBackend: netavark
  networkBackendInfo:
    backend: netavark
    dns:
      package: aardvark-dns-1.11.0-1.20240531102943328308.main.4.g6838c50.fc40.aarch64
      path: /usr/libexec/podman/aardvark-dns
      version: aardvark-dns 1.12.0-dev
    package: netavark-1.11.0-1.20240606174759319307.main.8.gfebe31a.fc40.aarch64
    path: /usr/libexec/podman/netavark
    version: netavark 1.12.0-dev
  ociRuntime:
    name: crun
    package: crun-1.15-1.20240607090105650503.main.32.gea54402.fc40.aarch64
    path: /usr/bin/crun
    version: |-
      crun version UNKNOWN
      commit: 7cfd0aeb40e4605b6b0ee0afd9cfca80f9c5f68a
      rundir: /run/user/503/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +LIBKRUN +WASM:wasmedge +YAJL
  os: linux
  pasta:
    executable: /usr/bin/pasta
    package: passt-0^20240510.g7288448-1.fc40.aarch64
    version: |
      pasta 0^20240510.g7288448-1.fc40.aarch64-pasta
      Copyright Red Hat
      GNU General Public License, version 2 or later
        <https://www.gnu.org/licenses/old-licenses/gpl-2.0.html>
      This is free software: you are free to change and redistribute it.
      There is NO WARRANTY, to the extent permitted by law.
  remoteSocket:
    exists: true
    path: /run/user/503/podman/podman.sock
  rootlessNetworkCmd: pasta
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: true
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.2-2.fc40.aarch64
    version: |-
      slirp4netns version 1.2.2
      commit: 0ee2d87523e906518d34a6b423271e4826f71faf
      libslirp: 4.7.0
      SLIRP_CONFIG_VERSION_MAX: 4
      libseccomp: 2.5.3
  swapFree: 0
  swapTotal: 0
  uptime: 8h 15m 43.00s (Approximately 0.33 days)
  variant: v8
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  - ipvlan
  volume:
  - local
registries:
  search:
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions: {}
  graphRoot: /var/home/core/.local/share/containers/storage
  graphRootAllocated: 106769133568
  graphRootUsed: 4735725568
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "true"
    Supports d_type: "true"
    Supports shifting: "false"
    Supports volatile: "true"
    Using metacopy: "false"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 2
  runRoot: /run/user/503/containers
  transientStore: false
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 5.1.1
  Built: 1717459200
  BuiltTime: Mon Jun  3 17:00:00 2024
  GitCommit: ""
  GoVersion: go1.22.3
  Os: linux
  OsArch: linux/arm64
  Version: 5.1.1

Here is a push with debug: $ podman push --log-level=debug docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1

INFO[0000] podman filtering at log level debug          
DEBU[0000] Called push.PersistentPreRunE(podman push --log-level=debug docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1) 
DEBU[0000] SSH Ident Key "/Users/rshiao/.local/share/containers/podman/machine/machine" SHA256:***** ssh-ed25519 
DEBU[0000] DoRequest Method: GET URI: http://d/v5.1.1/libpod/_ping 
DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" 
DEBU[0000] Found credentials for docker.intuit.com in credential helper containers-auth.json in file /Users/rshiao/.config/containers/auth.json 
DEBU[0000] DoRequest Method: POST URI: http://d/v5.1.1/libpod/images/docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1/push 
Getting image source signatures
Copying blob sha256:304486ee47e4d2aa67bf9153ff3150c380078bf6ddf39a0db036548e3a58a01e
Copying blob sha256:9f6b5e87c1209b3370798b8f074361950ec07dbf7319cc662d4c1574413a3888
Copying config sha256:a92b1da67162194db6af591fbccf85d8829e835cb1913509c97da470463c1dc6
Writing manifest to image destination
DEBU[0001] Called push.PersistentPostRunE(podman push --log-level=debug docker.intuit.com/personal/rshiao/sample20240610:podman-5.1.1) 
DEBU[0001] Shutting down engines
nalind commented 3 weeks ago

I was not able to reproduce different behavior with a copy of 5.0.3 built from source, but if you're using a packaged version of 5.0.3, it may have come with a containers.conf that did include the configuration change. At any rate, a --format=docker shouldn't hurt when pushing an image.

rhatdan commented 3 weeks ago

Check to see if zstd:chunked is set as compression-format.

podman push --help | grep compression-format --compression-format string compression format to use (default "zstd:chunked")

RichardShiao commented 3 weeks ago

@nalind Thanks for checking your copy of 5.0.3. I thought it was something consistently reproducible in code, but sounds like it is packaging dependent.

For more context, the original place we saw impact was from our scripts that use images from quay.io podman/stable releases. We were using stable:latest, which has been working for us for a few years. But a few days ago, we started seeing this manifest type issue. I'm guessing it is correlated to when latest tag switched to 5.1.0? As a workaround, we pinned the image to stable:v5.0.2 and that restored the old behavior.

For further testing, I also tried podman on my local (Macbook), which was running podman v5.0.3 installed via brew. I then installed podman v5.1.1 via brew. And the test results are what I posted above.

@rhatdan , here is what I get from my Mac, now running v5.1.1 from brew: podman push --help | grep compression-format

      --compression-format string   compression format to use

Did something change with packaging for both these methods (quay.io repository and brew)?

Luap99 commented 3 weeks ago

see https://github.com/containers/podman/issues/22813 and https://github.com/containers/common/pull/2048

This was a packaging bug as zstd:chunked should only be enabled on fedora 41, so this will be fixed next time the machine image is rebuild