Image Digest mismatch when using Jib Builder with/without Docker

[willsu]

When building an image using Skaffold and the Jib builder, the image digest will be different when pushed to the remote image registry depending on whether the image was built and pushed with or without Docker.

A scenario where the digest mismatch issue can cause problem begins when an image in a remote registry is pushed by Skaffold using the Jib Builder and the "push" flag set to true (which does not use Docker). When that same remote image is then pulled into a local Docker daemon, and pushed (with no changes) through the Docker CLI to the same remote registry, an image with a new digest will be created in the remote registry. While there is not much utility in using Docker to "re-push" the same image that already exists in a remote repository, users may have the requirement to update properties of the image (such as Docker metadata) that do not effect the image digest. For example, Cloud Build Build Provenance is designed to update the Docker images metadata and push the image to the remote repository on pipeline where it is enabled. In the Cloud Build scenario, calling 'skaffold build --push=true' at the beginning of the build pipeline will generate an image with a different digest in the remote repository than Cloud Build will generate when it pushes the image from it's local Docker daemon (provided that you the image had been pulled from the remote registry into your Cloud Build pipeline's local Docker daemon)

Note: This may entirely be a Jib issue and there may not be much to be done here. I wanted to report this issue to see if you all felt like anything should come out of this (i.e. documentation, code changes, etc).

Expected behavior

Skaffold is expected to generate container images with consistent digests when pushed to a remote image registry regardless whether the image was built using a local Docker daemon or directly through Jib (without using Docker).

Note: I'm not sure that this premise is strictly true, but I'm using it as a way to describe the issue.

Actual behavior

Skaffold generates container images with different digests when pushed to a remote image registry regardless whether the image was built using a local Docker daemon or directly through Jib (without using Docker).

Information

• Skaffold version: v2.9.0

• Operating system: Debian GNU/Linux 11

• Installed via: Google Cloud Shell

• Contents of skaffold.yaml:

apiVersion: skaffold/v4beta6
kind: Config
build:
  artifacts:
    - image: spring-java-hello-world
      jib: {}
deploy:
  kubectl: {}
manifests:
  rawYaml:
    - kubernetes/deployment.yml

profiles:
  - name: dev
    # automatically activate this profile when current context is "minikube"
    activation:
      - kubeContext: minikube
    build:
      local:
        push: true 
      # Generated artifact
      artifacts:
        - image: spring-java-hello-world
          jib: {}
    manifests:
      rawYaml:
        - kubernetes/deployment.yml
    portForward:
      - resourceType: service
        resourceName: spring-java-hello-world-service
        port: 8080
        localPort: 9000

Steps to reproduce the behavior

1. Clone Repo: https://github.com/willsu/jib-skaffold-docker-digest-mismatch
2. Build and push the image with Skaffold using the Jib (without Docker):

   skaffold build -p dev --default-repo us-central1-docker.pkg.dev/[YOUR-PROJECT]/hello-world-docker-repository --push=true

3. View the images in Artifact Registry and see a container with the digest. You should see one container with a digest starting with: aaaa163f8012

4. Verify that there are no Docker images with the name "spring-java-hello-world"

   docker images

5. Use skaffold to build the image locally

   skaffold build -p dev --default-repo us-central1-docker.pkg.dev/[YOUR-PROJECT]/hello-world-docker-repository --push=false

6. Push the image back to the remote registry

   docker push us-central1-docker.pkg.dev/[YOUR-PROJECT]/hello-world-docker-repository/spring-java-hello-world:6d6d69

7. View the images in Artifact Registry and see a container with the digest. You should see two containers with the digests starting with: afc49cdee161 and aaaa163f8012

Logs

See attached files step-2.log step-5.log step-6.log

[ericzzzzzzz]

There are two problems relating to inconsistent image digests: Here is a short summary about the investigation result which has been shared with will offline.

Build Tool Differences: Different image building tools (Jib, Docker) might layer or configure images slightly differently, leading to mismatched digests (the SHA256 hash of the image's manifest).

Registry Client Behavior: Image registry clients can format image manifests (metadata files) differently. This formatting change, even without content changes, alters the digest. Using crane copy may preserve the digest during image transfers. "