podman-docker integration

[GrahamDumpleton]

This issue seems to be similar to https://github.com/k14s/kbld/issues/19 but the setup is different.

Have a typical setup for kbld where it is doing image build and pushing image to remote registry. When it fills out the image digest reference in the deployment, it is using the digest which matched what was stored in local image build cache. The digest which the image was stored under in the remote image registry is different though, meaning the image cannot be found by the deployment.

Final step of kbld outputs:

resolve | final: custom-jupyterhub -> lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-
jupyterhub@sha256:0fc073a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6

Inspection of the image in the local build cache yields:

[
    {
        "Id": "769906fbf88ba54ba5a5267749edda68c2aea96a630b2e2c9aee3b7042eaa860",
        "Digest": "sha256:0fc073a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6",
        "RepoTags": [
            "localhost/kbld:custom-jupyterhub-769906fbf88ba54ba5a5267749edda68c2aea96a630b2e2c9aee3b7042eaa8
60",
            "lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub:kbld-rand-1586
680932345334383-93222136210142"
        ],
        "RepoDigests": [
            "lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc073a
041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6",
            "localhost/kbld@sha256:0fc073a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6"
        ],

The image reference in the deployment is:

metadata:
  annotations:
    kbld.k14s.io/images: |
      - Metas:
        - Path: /home/eduk8s/hub-v3
          Type: local
        URL: lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc073a
041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6
...
    spec:
      containers:
      - image: lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc07
3a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6

The error from kapp when deploying is:

8:47:22AM:     ^ Pending: ErrImagePull (message: rpc error: code = Unknown desc = Error response from daemon
: manifest for lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc07
3a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6 not found: manifest unknown: manifest unknown)

If you rerun just kbld with nothing changed, you get the error:

kbld: Error:
- Resolving image 'custom-jupyterhub': Expected to find same repo digest, but found []string{"lab-jupyter-on
-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub@sha256:0fc073a041e341ca3db2fe40a570d5c22e
b529b591fc90a490737b4d48ca53d6", "lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupy
terhub@sha256:484209beb53e3dae50fc828fa215ac596f9f6f35d6398c113b7ce45b59dec5d2", "localhost/kbld@sha256:0fc0
73a041e341ca3db2fe40a570d5c22eb529b591fc90a490737b4d48ca53d6", "localhost/kbld@sha256:484209beb53e3dae50fc82
8fa215ac596f9f6f35d6398c113b7ce45b59dec5d2"}

which also appears to be picking up the mismatch.

As far as the setup goes, the builds are being done with podman using the podman-docker wrapper on Fedora which results in podman being invoked when docker is run. Thus podman is used to push the image to the remote registry. The remote registry self hosted latest version of the standard Docker image registry.

The image registry was empty before starting. After kbld run, have:

$ curl -u xxx:xxx -X GET https://$REGISTRY_HOST/v2/_catalog
{"repositories":["custom-jupyterhub"]}

$ curl -u xxx:xxx -X GET https://$REGISTRY_HOST/v2/custom-jupyterhub/tags/list
{"name":"custom-jupyterhub","tags":["kbld-rand-1586680932345334383-93222136210142"]}

Running skopeo to inspect the remote image get:

    "Name": "lab-jupyter-on-k8s-03-w01-s003-registry.training.getwarped.org/custom-jupyterhub",
    "Digest": "sha256:cdaeaddf70987d0b895efccaa735ea6efcdb00228a0bdd9f586c78f663c42b3c",
    "RepoTags": [
        "kbld-rand-1586680932345334383-93222136210142"
    ],
    ...
    "Layers": [
        "sha256:462816986e4db9b3ba3ff89114709053ed939e75339c1bf64bb6c8c20bb0ac09",
        "sha256:8a14c0ec1aad906c9703b3be9a4807671a6367d6496d1941ff4a40f488887816",
        "sha256:ea892e6e4eade778ab7349dafb7a078676f4367c7bed5067c2456c204705aeaa",
        "sha256:a8abb9d778483d211484c35e61002e6dde8ca8755c0577a82fe265c65590a786",
        "sha256:f5266c3eb07a9a052ccde05abe2bd4e763a8ffe316dcaef1c3f1b2935515ec14",
        "sha256:255f20c7a19d5ec58bc992bcf3c56cd58c6cd9ffd144c3cbd854a5e4efc72a94",
        "sha256:3699fa7667edd14c6e99940406dd9fb4f2d33e0bced900623bd973bd822ab4d5",
        "sha256:cf5bbb6d527ad8bc3eb29a4b73fa7acfc1fcb2d4bbf17df1b3d5e4176aad33bd",
        "sha256:021fc1973d41319bdb76e723c4f89424d5a9c7a6d3bd5b0d79560722a70e53e3",
        "sha256:b971891c534ff0a70e03c988a9e254b6e33d192e61d1e1352061edb69bba7469",
        "sha256:5fd45b49b7e12c875ca5110ea78b8ff04ac90c79db1c0bbced7a13d55d6f820a",
        "sha256:f291ee79eb30aedf1cf37386b3c3233e193c3a95255a8e8b55e9d5ff41e07409",
        "sha256:028d992643957f2d6b967697e91e7c986eb919016aa88fd997e8b91237cb5fed",
        "sha256:716df0b1c063d005b933667274c2edd3a0d6bf5405ac1390812066543c812a36",
        "sha256:68d6bea7dd2ed44cb44cf70365c75f69e6b30877332dac5d08b333c4de56c6b1"
    ],

So looks like kbld should be working out what the digest is from the image on the remote image registry after it has been pushed, rather than assuming the image will have the same digest value.

cc @jorgemoralespou

[cppforlife]

hey @GrahamDumpleton,

If you rerun just kbld with nothing changed, you get the error: Expected to find same repo digest...

hmm, it seems that podman-docker returns multiple digests for the same locally tagged image. that's not the behaviour of docker inspect.

So looks like kbld should be working out what the digest is from the image on the remote image registry after it has been pushed

kbld expects docker to record a digest at the time of the push (it uses docker inspect to retrieve it). it also expects that digest matches what was pushed to registry. it looks like podman-docker does not offer same API guarantees as docker, hence it is not compatible with kbld today.

kbld does not ask registry for the digest because pushing process should know what the digest is (given that it's pushing content), and there is no good way to ask registry in a "non-racy" way. theoretically asking registry for the digest for a tag is racy because someone else can override the tag meanwhile.

i havent run podman before, but it might be worth adding it as a builder.

[jorgemoralespou]

@GrahamDumpleton can it be a problem on podman side? Related to this (https://github.com/containers/libpod/issues/3761)(https://github.com/containers/libpod/issues/3761) maybe?