search

vmware-labs / distribution-tooling-for-helm

Helm Distribution plugin is is a set of utilities and Helm Plugin for making offline work with Helm Charts easier. It is meant to be used for creating reproducible and relocatable packages for Helm Charts that can be moved around registries without hassles. This is particularly useful for distributing Helm Charts into airgapped environments.
Apache License 2.0
71 stars 12 forks source link

Failed to verify Helm chart Images.lock when unwrapping #23

Closed RasmusGodske closed 8 months ago

RasmusGodske commented 10 months ago

Describe the bug

This might be a intentional beheaviour and me simply using this plugin wrong. When I'm unwrapping a chart, I'm getting the following error:

Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:                                    
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
+ sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
+ sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7

Reproduction steps

Setup a OCI-based registry locally using docker:

docker run -d -p 5000:5000 --restart=always --name registry registry:2

Run the following script

#!/bin/bash

helm repo add temp --force-update https://benthosdev.github.io/benthos-helm-chart/
helm repo update

helm pull temp/benthos --version 0.8.0 --untar --untardir .

# This helm chart isn't annotated by default, so I'm manually doing this here
images_override=$(cat <<-END
- name: benthos
  image: docker.io/jeffail/benthos:4.11.0
END
)
yq -i e ".annotations.images |= \"$images_override\"" ./benthos/Chart.yaml

helm dt wrap  ./benthos
helm dt unwrap  benthos-0.8.0.wrap.tgz oci://localhost:5000 --yes

After this script has ran, a new directory named benthos is present in the current working directory.

. └── benthos/ │ ├── images/ │ │ ├── 244e9c77b90ec18...tar (linux/arm64) │ │ └── e2fa170d1d0c2e7...tar (linux/amd64) │ └── Images.lock │ └── .... └── benthos-0.8.0.wrap.tgz

When untaring benthos-0.8.0.wrap.tgz you get the same content as ./benthos.

Expected behavior

  1. The helm chart and docker images is being relocated and pushed to the local registry

  2. The benthos-0.8.0.wrap.tgz wound have the Image.lock relocated and Image.lock look like this:

    apiVersion: v0
kind: ImagesLock
metadata:
generatedAt: "2023-11-30T09:27:39.313962408Z"
generatedBy: Distribution Tooling for Helm
chart:
name: benthos
version: 0.8.0
appVersion: 4.11.0
images:
- name: benthos
image: localhost:5000/jeffail/benthos:4.11.0
chart: benthos
digests:
  - digest: sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
    arch: linux/amd64
  - digest: sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
    arch: linux/arm64

    instead of this:

    apiVersion: v0
kind: ImagesLock
metadata:
generatedAt: "2023-12-05T10:32:00.611186491Z"
generatedBy: Distribution Tooling for Helm
chart:
name: benthos
version: 0.8.0
appVersion: 4.11.0
images:
- name: benthos
image: docker.io/jeffail/benthos:4.11.0
chart: benthos
digests:
  - digest: sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
    arch: linux/amd64
  - digest: sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
    arch: linux/arm64

Additional context

I'm able to produce the digests found in the error, when doing the following addtional steps(after the bash script previously mentioned):

> helm dt charts relocate benthos/ oci://localhost:5000
> helm dt images lock ./benthos
> cat Images.lock

apiVersion: v0
kind: ImagesLock
metadata:
  generatedAt: "2023-12-05T13:48:53.107936871Z"
  generatedBy: Distribution Tooling for Helm
chart:
  name: benthos
  version: 0.8.0
  appVersion: 4.11.0
images:
  - name: benthos
    image: localhost:5000/jeffail/benthos:4.11.0
    chart: benthos
    digests:
      - digest: sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
        arch: linux/amd64
      - digest: sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
        arch: linux/arm64

So this seems to me that the relocate changes the digest of the images. I'm assuming this is some metadata within the images that is changed.

However this change seems to be messing with the actual digest which is used when comparing.

mpermar commented 10 months ago

Hi @RasmusGodske. Verification usually will fail when the digests that are in the lock file don't match the digests of the images that the user has asked dt to push.

When running relocate , what dt does is to modify the container image references found at different places like Chart.yaml or values.yaml, so indeed it modifies the Chart. When you then run lock afterwards, that generates a lock from the relocated chart which well, it will cause such inconsistency.

This is probably our bad at documentation. wrap and unwrap are meant to be used together, and not with the "finer grained" commands. Also lock is always the first command you would run after annotate. Only then you should run relocate and not the other way around.

Technically, you can reproduce a wrap -> unwrap with: annotate -> lock -> verify -> pull -> relocate -> push -> verify and then manually pushing the chart to a registry with helm or crane. Wrap and unwrap just try to do all that for you.

RasmusGodske commented 10 months ago

Hello @mpermar. Thanks for the quick response, that makes perfect sense. However I have tried using wrap and unwrap together, which caused the same problem. This led me down the path of using the "fine grained" commands.

This is the modified bash script, using just wrap and unwrap.

#!/bin/bash

helm repo add temp --force-update https://benthosdev.github.io/benthos-helm-chart/
helm repo update

helm pull temp/benthos --version 0.8.0 --untar --untardir .

######### Manually Annotating Chart.yaml #########
images_override=$(cat <<-END
- name: benthos
  image: docker.io/jeffail/benthos:4.11.0
END
)

yq -i e ".annotations.images |= \"$images_override\"" ./benthos/Chart.yaml

helm dt wrap ./benthos
helm dt unwrap  benthos-0.8.0.wrap.tgz oci://localhost:5000 --yes

output:

"temp" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "temp" chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "benthos" chart repository
...Successfully got an update from the "openebs-jiva" chart repository
...Successfully got an update from the "vmware-tanzu" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
 »  Wrapping Helm chart "./benthos"
    ✔  Images.lock file written to "/home/godske/Documents/repositoriesoci-container-registry-storage/test/benthos/Images.lock"                          
    »  Pulling images into "/home/godske/Documents/repositories/oci-container-registry-storage/test/benthos/images"
       ✔  All images pulled successfully                                                                                                                       
    ✔  Compressed into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/benthos-0.8.0.wrap.tgz"                                   

 🎉  Helm chart wrapped into "/home/godske/Documents/repositories/oci-container-registry-storage/test/benthos-0.8.0.wrap.tgz"
 »  Unwrapping Helm chart "benthos-0.8.0.wrap.tgz"
    ✔  Helm chart uncompressed to "/tmp/chart-3226158309/at-wrap3259428596"                                                                                    
    ✔  Helm chart relocated successfully                                                                                                                       
    »  The wrap includes the following 1 images:

       localhost:5000/jeffail/benthos:4.11.0

    »  Pushing Images
       ✔  All images pushed successfully                                                                                                                       
    ✘  Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:                                                          
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
      + sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
      + sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
+ sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
+ sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Error: plugin "dt" exited with error

Worth mentioning that I have tried running it with a redis helm chart from a OCI based registry which runs without any problems.

Example:

#!/bin/bash

helm pull oci://registry-1.docker.io/bitnamicharts/redis --version 18.1.5 --untar --untardir .

helm dt wrap ./redis
helm dt unwrap  redis-18.1.5.wrap.tgz oci://localhost:5000 --yes

Output: 

Pulled: registry-1.docker.io/bitnamicharts/redis:18.1.5
Digest: sha256:892a93be15b28f4a03870c31afbaa73578c5bcf1eeb3f32d8d5649d1be9e51ee
 »  Wrapping Helm chart "./redis"
    ✔  Images.lock file written to "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis/Images.lock"                                                              
    »  Pulling images into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis/images"
       ✔  All images pulled successfully                                                                                                                                                         
    ✔  Compressed into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis-18.1.5.wrap.tgz"                                                                      

 🎉  Helm chart wrapped into "/home/godske/Documents/repositories/work/oci-container-registry-storage/test/redis-18.1.5.wrap.tgz"
 »  Unwrapping Helm chart "redis-18.1.5.wrap.tgz"
    ✔  Helm chart uncompressed to "/tmp/chart-1844294399/at-wrap3814855721"                                                                                                                      
    ✔  Helm chart relocated successfully                                                                                                                                                         
    »  The wrap includes the following 4 images:

       localhost:5000/bitnami/os-shell:11-debian-11-r90
       localhost:5000/bitnami/redis-exporter:1.55.0-debian-11-r0
       localhost:5000/bitnami/redis-sentinel:7.2.1-debian-11-r26
       localhost:5000/bitnami/redis:7.2.1-debian-11-r26

    »  Pushing Images
       ✔  All images pushed successfully                                                                                                                                                         
       ✔  Chart "/tmp/chart-1844294399/at-wrap3814855721" lock is valid                                                                                                                          

    ✔  Helm chart successfully pushed                                                                                                                                                            

 🎉  Helm chart unwrapped successfully: You can use it now by running "helm install oci://localhost:5000/redis --generate-name"
RasmusGodske commented 10 months ago

After some more tinkering and experimenting, I got a working script. This works with the non OCI-based benthos helm chart, which did not work using wrap and unwrap. This time I only made use of the "fine-grade" commands, in the order you mentioned(Much appreciated). I can however not explain why the wrap and unwrap commands did not work for me.

#!/bin/bash

helm repo add temp --force-update https://benthosdev.github.io/benthos-helm-chart/
helm repo update

helm pull temp/benthos --version 0.8.0 --untar --untardir .

images_override=$(cat <<-END
- name: benthos
  image: docker.io/jeffail/benthos:4.11.0
END
)

yq -i e ".annotations.images |= \"$images_override\"" ./benthos/Chart.yaml

helm dt images lock ./benthos

helm dt images verify ./benthos

helm dt images pull ./benthos

helm dt charts relocate ./benthos oci://xragoregistryforgiggles.azurecr.io

helm dt images push ./benthos

helm dt images lock ./benthos

helm dt images verify ./benthos

helm package ./benthos

helm push ./benthos-0.8.0.tgz oci://xragoregistryforgiggles.azurecr.io
mpermar commented 10 months ago

I will try to replicate once back from holidays. I wonder if it might be related to be using a local registry. Have you tried the wrap/unwrap with azurecr?

RasmusGodske commented 10 months ago

No rush this can wait, holidays are important.

I have tried using ACR, however with the same result.

mpermar commented 9 months ago

@RasmusGodske I could reproduce the issue with the exising dt release 0.2.3. Then decided to build from source to debug it and it turned out that I was able to successfully wrap/unwrap the benthos Helm chart. So my theory is that you were hitting a bug that somehow @juamedgod had already fixed but that we hadn't release yet.

Can you try updating to the latest release (v0.3.0) with helm plugin update dt and give it a go again?

By the way, in this release, we have changed how the wraps are structured internally as the way they were it could cause some side-effects with certain Helm charts. This is an improvement but breaks backwards compatibility with older wraps. So you will have to wrap and unwrap for it to work.

Let me know if doing an upgrade fixes it 🙏

RasmusGodske commented 9 months ago

@mpermar I tried to update to the v0.3.0 however the error still persists.

The following script:

#!/bin/bash

helm dt version

helm repo add temp --force-update https://benthosdev.github.io/benthos-helm-chart/
helm repo update

helm pull temp/benthos --version 0.8.0 --untar --untardir .

# This helm chart isn't annotated by default, so I'm manually doing this here
images_override=$(cat <<-END
- name: benthos
  image: docker.io/jeffail/benthos:4.11.0
END
)
yq -i e ".annotations.images |= \"$images_override\"" ./benthos/Chart.yaml

helm dt wrap  ./benthos
helm dt unwrap  benthos-0.8.0.wrap.tgz oci://localhost:5000 --yes

Gives me the following error:

./scripts/test.sh    
Distribution Tooling for Helm v0.3.0
Built on: 2023-12-11T17:44:44Z

"temp" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "benthos" chart repository
...Successfully got an update from the "temp" chart repository
...Successfully got an update from the "stable" chart repository
...Successfully got an update from the "openebs-jiva" chart repository
...Successfully got an update from the "vmware-tanzu" chart repository
...Successfully got an update from the "traefik" chart repository
...Successfully got an update from the "grafana" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈Happy Helming!⎈
 »  Wrapping Helm chart "./benthos"
    ✔  Images.lock file written to "/tmp/chart-3552518746/wrap/chart/Images.lock"                                                                
    »  Pulling images into "/tmp/chart-3552518746/wrap/chart/images"
       ✔  All images pulled successfully                                                                                                         
    ✔  Compressed into "/home/godske/Documents/repositories/work/oci-container-registry-storage/benthos-0.8.0.wrap.tgz"                          

 🎉  Helm chart wrapped into "/home/godske/Documents/repositories/work/oci-container-registry-storage/benthos-0.8.0.wrap.tgz"
 »  Unwrapping Helm chart "benthos-0.8.0.wrap.tgz"
    ✔  Helm chart uncompressed to "/tmp/chart-2496835767/at-wrap2084825586"                                                                      
    ✔  Helm chart relocated successfully                                                                                                         
    »  The wrap includes the following 1 images:

       localhost:5000/jeffail/benthos:4.11.0

    »  Pushing Images
       ✔  All images pushed successfully                                                                                                         
    ✘  Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:                                            
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
      + sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
      Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
      - sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
      + sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Failed to push images: failed to verify Helm chart Images.lock: Images.lock does not validate:
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:e2fa170d1d0c2e7bf5c89986b6e5de471fe6499f231d6f8e9c1f31c4350353b0
+ sha256:85689f09ddac1449188c9b9c50138027ff476bab81e39d4229f28eb72793314b
Helm chart "benthos": image "localhost:5000/jeffail/benthos:4.11.0": digests do not match:
- sha256:244e9c77b90ec18e692acf162c55060156b1d36a820d19a8606f0a98e55d4ee3
+ sha256:f794f3b17252736664244d508c6ffc10f05619cc6d09bf4c4d465d1815e0afa7
Error: plugin "dt" exited with error
mpermar commented 9 months ago

@RasmusGodske definitely a bug. @juamedgod figured out. We are using Crane (go-containerregistry) to relocate the packages, and the way we are doing it actually modifies the format of the manifest. Not the content, but the format.

That does not affect any images that were put into registries using go-containerregistry, like Bitnami's for example. And that's why we hadn't seen this earlier. But your image manifest in DockerHub is pretty-printed. Therefore when we relocate it, go-containerregistry flattens the format and the manifest ends up with a different digest which is what fails the verification.

We will come up with a bug fix soon. But if you try using crane to upload something into DockerHub and then try to wrap/unwrap it, then it should work flawlessly.

RasmusGodske commented 9 months ago

That's an unexpected error, I'm happy that you manage to find it. I'm off this project soon, however feel free to contact/mention me if you need further testing. I'll be more than happy to help in my spare time.

juamedgod commented 8 months ago

@RasmusGodske We just merged #30, which should fix your use case. In case you want to give it a try.