dt
, is a set of utilities available in a standalone mode and as a Helm Plugin for making offline work with Helm charts easier. It is meant to be used for creating reproducible and relocatable packages for Helm charts that can be easily moved across registries without hassles. This is particularly useful for distributing Helm charts into air-gapped environments like those used by Federal governments.
Distribute your Helm charts with two easy commands
# Wrap
$ helm dt wrap oci://docker.io/bitnamicharts/kibana
...
π Helm chart wrapped into "/tmp/workspace/kibana/kibana-10.4.8.wrap.tgz"
# Unwrap
$ helm dt unwrap kibana-10.4.8.wrap.tgz demo.goharbor.io/helm-plugin/ --yes
...
π Helm chart unwrapped successfully: You can use it now by running "helm install oci://demo.goharbor.io/helm-plugin/kibana --generate-name"
This tool builds on HIP-15 and the, currently proposed, images lock file HIP (PR) as a foundation. Hence, it does require Helm charts to contain an annotation that provides the full list of container images that a Helm chart might need for its usage independently of the bootstrapping configuration.
Bitnami Helm charts are now fully annotated to support this tooling, but you can also use this set of utilities with any other Helm charts that might use any other alternative image listing annotation, like for example, Helm charts relying on artifact.io/images.
Provided you have Helm then you can install this tool as a plugin:
$ helm plugin install https://github.com/vmware-labs/distribution-tooling-for-helm
Note: Windows installation
If installing on Windows, the above command must be run in a bash emulator such as Git Bash.
Fetch the latest available release from the Releases section.
Note that all the examples below use this tool as a Helm plugin but you can just run it as standalone. Just remove the helm
command from all those examples.
You can build this tool with the following command. Golang 1.20 or above is needed to compile. golangci-lint is used for linting.
$ make build
You can also verify the build by running the unit tests:
$ make test
The following sections list the most common commands and their usage. This tool can be used either standalone or through the Helm plugin.
For the sake of following this guide, let's pull one of the Bitnami Helm charts into an examples folder:
$ git clone git@github.com:vmware-labs/distribution-tooling-for-helm.git
$ cd distribution-tooling-for-helm
$ bash -c "mkdir examples & helm pull oci://docker.io/bitnamicharts/mariadb -d examples --untar"
The two simplest and most powerful commands on this tool are wrap
and unwrap
. With these two commands you can relocate any Helm chart to any OCI registry in two steps.
Wrapping a chart consists of packaging the chart into a tar.gz, including all container images that this chart depends on, independently of values. Everything gets wrapped together into a single file. This will include also all the subcharts and their container images. That new file, the wrap, can be distributed around in whatever way you want (e.g. USB stick) to then later be unwrapped into a destination OCI registry. This process is commonly referred to as relocating a Helm chart.
Even more exciting, we don't need to download the Helm chart for wrapping it. We can point the tool to any reachable Helm chart and the tool will take care of packaging and downloading everything for us. For example:
$ helm dt wrap oci://docker.io/bitnamicharts/kibana
Β» Wrapping Helm chart "oci://docker.io/bitnamicharts/kibana"
β Helm chart downloaded to "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/kibana"
β Images.lock file "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/kibana/Images.lock" does not exist
β Images.lock file written to "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/kibana/Images.lock"
Β» Pulling images into "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-1177363375/chart-1516625348/kibana/images"
β All images pulled successfully
β Helm chart wrapped to "/tmp/workspace/kibana/kibana-10.4.8.wrap.tgz"
π Helm chart wrapped into "/tmp/workspace/kibana/kibana-10.4.8.wrap.tgz"
Note that depending on the number of images needed by the Helm chart (remember, a wrap has the full set of image dependencies, not only the ones set on values.yaml) the size of the generated wrap might be considerably large:
$ ls -l kibana-10.4.8.wrap.tgz
-rw-r--r-- 1 martinpe staff 731200979 Aug 4 15:17 kibana-10.4.8.wrap.tgz
If you want to make changes on the Helm chart, you can pass a directory to the wrap command. For example, if we wanted to wrap the previously pulled mariadb Helm chart, we could just do:
$ helm dt wrap examples/mariadb/
Β» Wrapping Helm chart "examples/mariadb/"
β Images.lock file "/tmp/workspace/distribution-tooling-for-helm/examples/mariadb/Images.lock" does not exist
β Images.lock file written to "/tmp/workspace/distribution-tooling-for-helm/examples/mariadb/Images.lock"
Β» Pulling images into "/tmp/workspace/distribution-tooling-for-helm/examples/mariadb/images"
β All images pulled successfully
β Helm chart wrapped to "/tmp/workspace/distribution-tooling-for-helm/mariadb-13.0.0.wrap.tgz"
π Helm chart wrapped into "/tmp/workspace/distribution-tooling-for-helm/mariadb-13.0.0.wrap.tgz"
If your chart and docker images include artifacts such as signatures or metadata, you can also include them in the wrap using the --fetch-artifacts
flag.
Currently, dt
supports moving artifacts that follow certain conventions. That is:
sha256-digest.metadata
OCI entryFor example:
$ helm dt wrap --fetch-artifacts oci://docker.io/bitnamicharts/kibana
...
π Helm chart wrapped into "/tmp/workspace/distribution-tooling-for-helm/kibana-10.4.8.wrap.tgz"
$ tar -tzf "/tmp/workspace/distribution-tooling-for-helm/kibana-10.4.8.wrap.tgz" | grep artifacts
kibana-10.4.8/artifacts/images/kibana/kibana/8.10.4-debian-11-r0.sig
kibana-10.4.8/artifacts/images/kibana/kibana/8.10.4-debian-11-r0.metadata
kibana-10.4.8/artifacts/images/kibana/kibana/8.10.4-debian-11-r0.metadata.sig
...
Note: Signatures
Chart signatures are not bundled as they would be invalidated at chart unwrap because of the relocation. All the container images wrapped will maintain their signatures and metadata.
Sometimes you might actually want to wrap your Helm charts without fetching the container images. This is not exactly a wrap from the point of view of the initial conception of this tool but we acknowledge that this case can be useful when you already know that the Helm charts exist in the target registry. Hence you can skip wrapping the container images by using the --skip-pull-images
flag:
$ helm dt wrap oci://docker.io/bitnamicharts/magento --skip-pull-images
Β» Wrapping Helm chart "oci://docker.io/bitnamicharts/magento"
β Helm chart downloaded to "/var/folders/cr/jn5532p51390yx_6ctd7c6_40000gn/T/chart-2437949055/chart-1972871498/magento"
β Images.lock file written to "/var/folders/cr/jn5532p51390yx_6ctd7c6_40000gn/T/chart-2437949055/wrap/chart/Images.lock"
β Compressed into "/tmp/workspace/distribution-tooling-for-helm/magento-28.0.4.wrap.tgz"
π Helm chart wrapped into "/tmp/workspace/distribution-tooling-for-helm/magento-28.0.4.wrap.tgz"
Unwrapping a Helm chart can be done either to a local folder or to a target OCI registry, being the latter the most powerful option. By unwrapping the Helm chart to a target OCI registry the dt
tool will unwrap the wrapped file, proceed to push the container images into the target registry that you have specified, relocate the references from the Helm chart to the provided registry and finally push the relocated Helm chart to the registry as well.
At that moment your Helm chart will be ready to be used from the target registry without any dependencies to the source. By default, the tool will run in dry-run mode and require you to confirm actions but you can speed everything up with the --yes
parameter.
$ helm dt unwrap kibana-10.4.8.wrap.tgz demo.goharbor.io/helm-plugin/ --yes
Β» Unwrapping Helm chart "kibana-10.4.8.wrap.tgz"
β Helm chart uncompressed to "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258"
β Helm chart relocated successfully
Β» The wrap includes the following 2 images:
demo.goharbor.io/helm-plugin/bitnami/kibana:8.9.0-debian-11-r9
demo.goharbor.io/helm-plugin/bitnami/os-shell:11-debian-11-r25
Β» Pushing Images
β All images pushed successfully
β Chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" lock is valid
β Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (0 β Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (0 β Ή Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (0 β Έ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (1 β Ό Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (1 β ΄ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (1 β ¦ Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (1 β § Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (1 β Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (2 β Pushing Helm chart "/var/folders/mn/j41xvgsx7l90_hn0hlwj9p180000gp/T/chart-586072428/at-wrap2428431258" to "oci://demo.goharbor.io/helm-plugin/" (2 β Helm chart successfully pushed
π Helm chart unwrapped successfully: You can use it now by running "helm install oci://demo.goharbor.io/helm-plugin/kibana --generate-name"
If your wrap includes bundled artifacts (if you wrapped it using the --fetch-artifacts
flag), they will be also pushed to the remote registry.
That was all as per the basic most basic and powerful usage. If you're interested in some other additional goodies then we will dig next into some specific finer-grained commands.
An images lock file, a.k.a. Images.lock
is a new file that gets created inside the directory as per this HIP submission to Helm community. The Images.lock
file contains the list of all the container images annotated within a Helm chart's Chart.yaml
manifest, including also all the images from its subchart dependencies. Along with the images, some other metadata useful for automating processing and relocation is also added.
So, for example, the mariadb Helm chart that we downloaded earlier, has an images
annotation like this:
$ cat examples/mariadb/Chart.yaml | head -n 10
annotations:
category: Database
images: |
- image: docker.io/bitnami/mariadb:11.0.2-debian-11-r2
name: mariadb
- image: docker.io/bitnami/mysqld-exporter:0.15.0-debian-11-r5
name: mysqld-exporter
- image: docker.io/bitnami/os-shell:11-debian-11-r22
name: os-shell
licenses: Apache-2.0
We can run the following command to create the Images.lock
for the above Helm chart:
$ helm dt images lock examples/mariadb
INFO[0005] Images.lock file written to "/tmp/workspace/distribution-tooling-for-helm/examples/mariadb/Images.lock"
And it should look similar to this:
$ cat examples/mariadb/Images.lock
apiVersion: v0
kind: ImagesLock
metadata:
generatedAt: "2023-08-04T13:36:09.398772Z"
generatedBy: Distribution Tooling for Helm
chart:
name: mariadb
version: 13.0.0
images:
- name: mariadb
image: docker.io/bitnami/mariadb:11.0.2-debian-11-r2
chart: mariadb
digests:
- digest: sha256:d3006a4d980d82a28f433ae7af316c698738ba29a5a598d527751cb9139ab7ff
arch: linux/amd64
- digest: sha256:3ec78b7c97020ca2340189b75eba4a92ccb0d858ee62dd89c6a9826fb20048c9
arch: linux/arm64
- name: mysqld-exporter
image: docker.io/bitnami/mysqld-exporter:0.15.0-debian-11-r5
chart: mariadb
digests:
- digest: sha256:6f257cc719f5bbde118c15ad610dc27d773f80216adabf10e315fbcaff078615
arch: linux/amd64
- digest: sha256:e0c141706fd1ce9ec5276627ae53994343ec2719aba606c1dc228f9290698fc1
arch: linux/arm64
- name: os-shell
image: docker.io/bitnami/os-shell:11-debian-11-r22
chart: mariadb
digests:
- digest: sha256:7082ebf5644cf4968ac635986ded132dd308c0b9c13138f093834f343cd47d7b
arch: linux/amd64
- digest: sha256:232ca2da59e508978543c8b113675c239a581938c88cbfa1ff17e9b6e504dc1a
arch: linux/arm64
By default Images.lock
creation expects an images
annotation in your Helm chart. However, this can be overridden by the annotations-key
flag. This is useful for example when dealing with Helm charts that rely on a different annotation like artifacthub.io/images
which has existed for a while. You can use this flag with most of the commands in this guide.
$ helm dt images lock ../charts/jenkins --annotations-key artifacthub.io/images
The above lock
command can be constrained to specific architectures. This is pretty useful to create lighter wraps as many of the images will be dropped when wrapping.
$ helm dt images lock ../charts/jenkins --platforms linux/amd64
If we now look at generated Images.lock
we will notice that it contains only linux/amd64
digests:
apiVersion: v0
kind: ImagesLock
metadata:
generatedAt: "2023-08-04T14:24:18.515082Z"
generatedBy: Distribution Tooling for Helm
chart:
name: mariadb
version: 13.0.0
images:
- name: mariadb
image: docker.io/bitnami/mariadb:11.0.2-debian-11-r2
chart: mariadb
digests:
- digest: sha256:d3006a4d980d82a28f433ae7af316c698738ba29a5a598d527751cb9139ab7ff
arch: linux/amd64
- name: mysqld-exporter
image: docker.io/bitnami/mysqld-exporter:0.15.0-debian-11-r5
chart: mariadb
digests:
- digest: sha256:6f257cc719f5bbde118c15ad610dc27d773f80216adabf10e315fbcaff078615
arch: linux/amd64
- name: os-shell
image: docker.io/bitnami/os-shell:11-debian-11-r22
chart: mariadb
digests:
- digest: sha256:7082ebf5644cf4968ac635986ded132dd308c0b9c13138f093834f343cd47d7b
arch: linux/amd64
The verify
command can be used to validate the integrity of an Images.lock
file in a given Helm chart. This command will try to validate that all upstream container images that will be pulled from the Helm chart match actually the image digests that exist in the actual lock file.
With this command, you can make sure that when you distribute a Helm chart with its corresponding Images.lock
then any customer will be able to validate that just exactly the images defined in the lock will be pulled. Note that this is exactly part of what the unwrap
command does, to make sure that only exactly what was wrapped gets into the target registry. Signing and other types of provenance are out of the scope of this tool for the time being and need to be added manually with external tooling. This is an area that we are very eager to improve soon.
$ helm dt images verify examples/mariadb
INFO[0004] Helm chart "examples/mariadb" lock is valid
Based on the Images.lock
file, this command downloads all listed images into the images/
subfolder.
$ helm dt images pull examples/mariadb
INFO[0000] Pulling images into "/tmp/workspace/distribution-tooling-for-helm/examples/mariadb/images"
INFO[0022] All images pulled successfully
INFO[0022] Success
Then, in the images
folder we should have something like
$ ls -1 examples/mariadb/images
232ca2da59e508978543c8b113675c239a581938c88cbfa1ff17e9b6e504dc1a.tar
3ec78b7c97020ca2340189b75eba4a92ccb0d858ee62dd89c6a9826fb20048c9.tar
6f257cc719f5bbde118c15ad610dc27d773f80216adabf10e315fbcaff078615.tar
7082ebf5644cf4968ac635986ded132dd308c0b9c13138f093834f343cd47d7b.tar
d3006a4d980d82a28f433ae7af316c698738ba29a5a598d527751cb9139ab7ff.tar
e0c141706fd1ce9ec5276627ae53994343ec2719aba606c1dc228f9290698fc1.tar
This command will relocate a Helm chart rewriting the Images.lock
and all of its subchart dependencies locks as well. Additionally, it will change the Chart.yaml
annotations, and any images used inside values.yaml
(and all those on subchart dependencies as well).
For example
$ helm dt charts relocate examples/mariadb acme.com/federal
INFO[0000] Helm chart relocated successfully
And we can check that references have indeed changed:
$ cat examples/mariadb/Images.lock |grep image
images:
image: acme.com/federal/bitnami/mariadb:11.0.2-debian-11-r2
image: acme.com/federal/bitnami/mysqld-exporter:0.15.0-debian-11-r5
image: acme.com/federal/bitnami/os-shell:11-debian-11-r22
Note that in some scenarios one might actually not be interested in relocating the images. Perhaps one is only interested in pushing the Helm chart to a different registry but retaining the images. For such scenarios the --skip-relocation
flag can be used when unwrapping the chart.
Based on the Images.lock
file, this command pushes all images (that must have been previously pulled into the images/
folder) into their respective registries. Note that this command does not relocate anything. It will simply try to push the images to wherever they are pointing.
Obviously, this command only makes sense when used after having pulled the images and executed the relocate
command.
# .. should have pulled images first ..
# .. then relocate to a target registry ..
# and now...
$ helm dt images push examples/mariadb
INFO[0033] All images pushed successfully
It is sometimes useful to obtain information about a wrapped chart before unwrapping it. For this purpose, you can use the info command:
$ helm dt info wordpress-16.1.24.wrap.tgz
Β» Wrap Information
Chart: wordpress
Version: 16.1.24
Β» Metadata
- generatedBy: Distribution Tooling for Helm
- generatedAt: 2023-08-18T12:52:55.824345304Z
Β» Images
docker.io/bitnami/apache-exporter:0.13.4-debian-11-r12 (linux/amd64, linux/arm64)
docker.io/bitnami/bitnami-shell:11-debian-11-r132 (linux/amd64, linux/arm64)
docker.io/bitnami/wordpress:6.2.2-debian-11-r26 (linux/amd64, linux/arm64)
docker.io/bitnami/bitnami-shell:11-debian-11-r123 (linux/amd64, linux/arm64)
docker.io/bitnami/mariadb:10.11.4-debian-11-r0 (linux/amd64, linux/arm64)
docker.io/bitnami/mysqld-exporter:0.14.0-debian-11-r125 (linux/amd64, linux/arm64)
docker.io/bitnami/bitnami-shell:11-debian-11-r130 (linux/amd64, linux/arm64)
docker.io/bitnami/memcached:1.6.21-debian-11-r4 (linux/amd64, linux/arm64)
docker.io/bitnami/memcached-exporter:0.13.0-debian-11-r8 (linux/amd64, linux/arm64)
If you are interested in getting the image digests, you can use the --detailed
flag:
$ helm dt info --detailed wordpress-16.1.24.wrap.tgz
Β» Wrap Information
Chart: wordpress
Version: 16.1.24
Β» Metadata
- generatedBy: Distribution Tooling for Helm
- generatedAt: 2023-08-18T12:52:55.824345304Z
Β» Images
Β» wordpress/apache-exporter
Image: docker.io/bitnami/apache-exporter:0.13.4-debian-11-r12
Digests
- Arch: linux/amd64
Digest: sha256:0b4373c3571d5640320b68f8d296c0a4eaf7704947214640b77528bb4d79d23c
- Arch: linux/arm64
Digest: sha256:895ba569e4db3188798e445fe3be2e4da89fd85cb8ae0c5ef0bd2a67cfe4305c
...
Β» mariadb/bitnami-shell
Image: docker.io/bitnami/bitnami-shell:11-debian-11-r123
Digests
- Arch: linux/amd64
Digest: sha256:13d8883d4f40612e8a231c5d9fa8c4efa74d2a62f0a1991f20fc32c5debdd2b1
- Arch: linux/arm64
Digest: sha256:74579dc63b3ae7d8ec21a6ffcd47d16781582fef8dd5a28e77844fcbcb1072c1
...
It is also possible to get a YAML dump if the Images.lock
in case you need to feed it to another process:
$ helm dt info --yaml wordpress-16.1.24.wrap.tgz
apiVersion: v0
kind: ImagesLock
metadata:
generatedAt: "2023-08-18T12:52:55.824345304Z"
generatedBy: Distribution Tooling for Helm
chart:
name: wordpress
version: 16.1.24
images:
- name: apache-exporter
image: docker.io/bitnami/apache-exporter:0.13.4-debian-11-r12
chart: wordpress
digests:
- digest: sha256:0b4373c3571d5640320b68f8d296c0a4eaf7704947214640b77528bb4d79d23c
arch: linux/amd64
- digest: sha256:895ba569e4db3188798e445fe3be2e4da89fd85cb8ae0c5ef0bd2a67cfe4305c
arch: linux/arm64
...
Images.lock
creation relies on the existence of the special images annotation inside Chart.yaml
. If you have a Helm chart that does not contain any annotations, this command can be used to guess and generate an annotation with a tentative list of images. It's important to note that this list is a best-effort as the list of images is obtained from the values.yaml
file and this is always an unreliable, often incomplete, and error-prone source as the configuration in values.yaml
is very variable.
$ helm dt charts annotate examples/mariadb
INFO[0000] Helm chart annotated successfully
From dt
v0.2.0 we have introduced a new command to create a Carvel bundle from any Helm chart.
$ helm dt charts carvelize examples/postgresql
β Helm chart "examples/postgresql" lock is valid
Β» Generating Carvel bundle for Helm chart "examples/postgresql"
β Validating Carvel images lock
β Carvel images lock written to "examples/postgresql/.imgpkg/images.yml"
β Carvel metadata written to "examples/postgresql/.imgpkg/bundle.yml"
π Carvel bundle created successfully
It is also possible to login and logout from OCI registries using the dt
command. For example:
$ helm dt auth login 127.0.0.1:5000 -u testuser -p testpassword
β log in to 127.0.0.1:5000 as user testuser
π logged in via /Users/home/.docker/config.json
$ helm dt auth logout 127.0.0.1:5000
β logout from 127.0.0.1:5000
π logged out via /Users/home/.docker/config.json
Error: Unable to update repository: exit status 1
This can happen when somehow the plugin process installation or removal breaks and the Helm plugin's cache gets corrupted. Try removing the plugin from the cache and reinstalling it. For example on MAC OSX it would be:
$ rm -rf $HOME/Library/Caches/helm/plugins/https-github.com-vmware-labs-distribution-tooling-for-helm
$ helm plugin install https://github.com/vmware-labs/distribution-tooling-for-helm
Good question. Both projects come from VMware and should be able to continue using relok8s if you want to. Although, our expectation is to gradually build more and more tooling around the HIP-15 proposal as it does have a substantial number of benefits when compared to the relocation approach followed by relok8s.
So as the community adopts this new proposal and this plugin becomes more mature we would suggest anyone using relok8s to move its platform scripts to start using this helm plugin. We expect this move to be pretty much straightforward, and actually a great simplification for anyone using relok8s or even chart-syncer.
Yes, still support chart-syncer and we don't have any short-term plans right now about it. But as this tool gains adoption, it becomes natural to think that it should be fairly straightforward to implement Helm chart syncing on top of it.