Adopt KRM for `apko` file format

[marshall007]

Once you have your image configs and build scripts setup, the majority of the work involved in operating a secure software factory is config management.

Kubernetes ecosystem has great config management tooling and libraries, which comes as no surprise given that the Kubernetes Resource Model (KRM) was designed with that intent.

Virtually all manipulation that needs to take place in the build systems for actual catalogs like chainguard-images/images can easily be achieved with off-the-shelf tooling that knows how to manipulate KRM configs.

We can make this a backwards-compatible change to start (make apiVersion and kind optional) and eventually provide a migration command to rewrite existing configs once we settle on the APIs:

# before
contents:
  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
  repositories:
    - https://packages.wolfi.dev/os
  packages:
    - ca-certificates-bundle
    - wolfi-base

entrypoint:
  command: /bin/sh -l

archs:
- x86_64

annotations:
  org.opencontainers.image.authors: "Chainguard Team https://www.chainguard.dev/"
  org.opencontainers.image.url: https://edu.chainguard.dev/chainguard/chainguard-images/reference/wolfi-base/
  org.opencontainers.image.source: https://github.com/chainguard-images/images/tree/main/images/wolfi-base

# after
apiVersion: apko.chainguard.dev/v1alpha1
kind: ImageConfig
metadata:
  name: wolfi-base
  annotations:
    org.opencontainers.image.authors: "Chainguard Team https://www.chainguard.dev/"
    org.opencontainers.image.url: https://edu.chainguard.dev/chainguard/chainguard-images/reference/wolfi-base/
    org.opencontainers.image.source: https://github.com/chainguard-images/images/tree/main/images/wolfi-base

contents:
  keyring:
    - https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
  repositories:
    - https://packages.wolfi.dev/os
  packages:
    - ca-certificates-bundle
    - wolfi-base

entrypoint:
  command: /bin/sh -l

archs:
- x86_64

Relates to #637

[marshall007]

As far as config management tooling goes, folks can use whatever they want, but kpt would be a great fit in particular for managing image catalogs.

It can clone remote configurations, you can apply manual edits, and continue incorporating upstream changes with its resource-merge update strategy. kpt also supports function execution both imperatively on the command line and in declarative pipeline configurations, promoting sharable validation/mutation functions. kpt always mutates resources in-place on the filesystem, which facilitates a pure GitOps model.

Down the road it might make sense to adopt kpt pkg and kpt fn sub-commands natively in some form.

[kaniini]

We have been talking about adopting some aspects of KRM internally. Between apko and melange, I think it is helpful already. Plus we have other tools that we want to build which would also benefit from standardizing on KRM.

We likely want explicitly declared feature-flag opt-ins however, rather than just depending on apiVersion to be magical.

[marshall007]

@kaniini I noticed some recent work has been done to refactor the matrixed variant definitions out of the YAML config, now using Terraform instead. Is this a stop-gap to more integrated tooling or a direction you all are committing to?

[mattmoor]

@marshall007 the terraform flows still render concrete apko configurations under the hood (I'm actually writing a blog post on that topic literally right now), and there is nothing stopping these from rendering KRM-compatible versions of things (we just treat the yaml as an API to target, so if anything the versioning helps).

In fact, there are a lot of parallels to KRM where the apko_config block is a lot like a partial resource specification that may be blended with more global settings (e.g. provider "apko" in terraform, but possibly something like a controller configmap in a cluster/KCP/KRM setup), and the KRM resource status is the rendered and locked output.

The terraform flow attests these rendered and locked configurations to enable reproducibility (this is the post I'm writing, you can see one here).

We likely want explicitly declared feature-flag opt-ins however, rather than just depending on apiVersion to be magical.

To echo this, apiVersion isn't everything. Consider that K8s itself continually launches alpha/beta features within "stable" API surfaces, so the apiVersion is sensitive to the context of apko tool version (analogous to the version of K8s you are running) and feature flag settings (analogous to alpha/beta settings on the K8s API server).

We have a lot of K8s controller/reconciler nerds on staff (myself included), so I'd be happy to nerd out about this.

[marshall007]

In fact, there are a lot of parallels to KRM where the apko_config block is a lot like a partial resource specification that may be blended with more global settings

@mattmoor exactly! What I was getting at is that these patterns of (1) blending/rewriting/patching resource-specific config with global config and (2) generating variant resource configs are quite common and a number of tools in the K8s ecosystem have been authored to deal with that.

I'm not actually interested in doing any sort of in-cluster reconciliation of apko configs (though that may be interesting for some folks). The primary motivation was simply to unlock the usage of tools like kustomize and kpt for managing/rendering configs (for both apko and melange).

Tools that manipulate KRM documents (like kustomize and kpt) are (mostly) all based on kyaml transformers. Looking ahead, my thinking was that kyaml might provide a more straight forward path to native integration (via wolfictl?) than Terraform/HCL.

[mattmoor]

(this got a little rambly, but hopefully it makes sense)

I don't see them as mutually exclusive TBH. Given that JSON is valid yaml, there's nothing stopping the use of kyaml transformers prior to apko_config's config_contents or even between apko_config and apko_build.

I would say the most interesting manipulation of the config (apko_config) is the production of a "locked" file form which is a task that has a level of semantic depth beyond what something like kustomize or kpt could do, since it requires a fairly deep semantic understanding. apko show-packages can be used to get a similar package list directly from apko FWIW.

That said, in my mind the main feature of terraform isn't the templating, it is enabling an orchestrated composition of tools, e.g. describing how we take an apko SBOM and attest it with cosign, or tag the resulting digest based on package versions, ... This sort of composition is enabled by treating the apko.yaml as an API, which many tools can read and write, so they can be spliced into the edges of the graph.

So I think what we're arguing is actually very complementary:

• I'm saying that the main feature of terraform we're pursuing (beyond lock files) is composition of tools,
• You're saying that by aligning with KRM we will have more tools capable of acting on apko.yaml.

That isn't to say that either requires the other, just that they could compose nicely.

[marshall007]

I don't see them as mutually exclusive TBH.

@mattmoor I don't either, sorry if I gave that impression in my last comment. I think folks should be able to use any config management tooling they wish.

I would say the most interesting manipulation of the config (apko_config) is the production of a "locked" file form which is a task that has a level of semantic depth beyond what something like kustomize or kpt could do, since it requires a fairly deep semantic understanding.

I only briefly looked at the apko terraform provider, and I'm sure there is more to it than this, but primarily "package version resolution" is what you are referring to as requiring a semantic understanding, right?

This is still rather clunky in kustomize right now but both kustomize and kpt leverage the KRM Functions specification for extending behavior.

That said, in my mind the main feature of terraform isn't the templating, it is enabling an orchestrated composition of tools [...] This sort of composition is enabled by treating the apko.yaml as an API, which many tools can read and write, so they can be spliced into the edges of the graph.

KRM functions are just executables (packaged as container images) that read ResourceList from stdin and write them back to stdout, enabling exactly the same sort of composition. Just as you say, the resource configs themselves are the API.

Here are some examples: https://catalog.kpt.dev/

[marshall007]

I keep thinking it would be really nice to "soft fork" a single apko and/or melange config from the official images repo in order to either experiment or build non-general variants with specific requirements. In cases where you want to own the day-two maintenance, it becomes very important to keep up with upstream changes to the build configs.

Here is how kpt enables that workflow and I think it could be a great solution here:

1. mkdir my-configs && cd $_ && git init
2. kpt pkg get https://github.com/kubernetes/ingress-nginx.git/deploy/static/provider/cloud@controller-v1.7.0 nginx-controller - fetch v1.7.0 manifests and put them in a package dir named nginx-controller
3. initial commit
4. apply any manual edits you want to the resource configs and commit
5. kpt pkg update ./nginx-controller@controller-v1.8.0 - perform a 3-way structural comparison between the original upstream, local changes, and updated remote. then merge changes into the local package