[Explore]: Distributing Korifi using Carvel, Helm or Kustomize

[gcapizzi]

Background

Currently, we distribute Korifi as a set of YAML files that the users need to manually edit before they can be applied. This is inconvenient and error-prone. Our hacking flow is also not great.

Instead, we'd like to distribute a package that can be easily customised via simple input values before being applied. This should simplify our automation scripts, our CI, and our installation instructions. Ideally it should be able to use the same flow for installing a local copy of Korifi just by running kbld locally.

Here's an overview of the tools I'm aware of that could solve our problem:

• Kustomize: the tool we currently use to generate our YAML files. It hasn't proved very flexible so far and it makes it particularly tricky to interpolate values inside ConfigMaps. On the other hand, it's is built into kubectl, making it automatically available to every user. It's worth exploring wether it could be used to provide a user-friendly experience to users installing Korifi.
• Helm: a very popular tool for distributing Kubernetes applications. It allows to distribute applications as charts which can be customised via values. Values are available for interpolation in the YAML via a simple but powerful templating language. Helm also supports features like hook scripts, tests and dependencies.
• Carvel: historically the tool of choice of the CF-on-Kubernetes community. It comes as a set of decoupled tools for templating (ytt), installation (kapp), image building (kbld), dependency management (vendir). It's quite powerful and it would easily cover our use case.

Acceptance Criteria

An assessment of how our distribution story might look like using each one of the tools above, including:

• how good you think it would be for the job;
• how we're going to migrate to it;
• how the release installation experience might look like;
• how the local copy installation experience might look like.

Dev Notes

• A particularly interesting issue is the one of building our images and pinning our releases to specific versions of those images. We're currently using kbld for this. Once our YAMLs have been generated, they get passed to kbld which will:
  • find all image references in the YAML;
  • build the correct image for each one of them;
  • inject the exact image reference in the YAML, pointing to the correct SHA;
  • produce a kubectl applyable YAML file.
• We want to be able to carry on doing this, but this might not be possible if we turn our config YAMLs into templates. Templates with non-YAML syntax (e.g. Helm) will just brake kbld, while YAML comment-based syntax (e.g. ytt) will get stripped.
• For Helm, the solution we found for Eirini was to have values for each container image, and run kbld on the Values.yml file, which is standard YAML.
• For ytt, I think the most elegant solution would be to use lock files, which could then be fed to ytt as data values file.
• Kustomize should not need any of this as it only deals with plain YAML.

[kieron-dev]

I've just done an experiment with @danail-branekov regarding templating config maps using kustomize.

To setup the files:

mkdir -p base overlay
cat <<EOF >base/kustomization.yaml
configMapGenerator:
- name: the-map
  literals:
  - altGreeting="Have a pineapple!"
  - enableRisky="true"
  - something-common="ok"
EOF

cat <<EOF >overlay/config-values.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: the-map
data:
  altGreeting: "Have a banana!"
  enableRisky: "false"
EOF

cat <<EOF >overlay/kustomization.yaml
resources:
- ../base

patchesStrategicMerge:
- config-values.yaml
EOF

To see the base configmap:

kubectl kustomize base

To see the overlay one:

kubectl kustomize overlay

Notice that the overlay can replace individual values from the base config map, but will preserve unmodified values. One downside is that we can't do nested structures this way. But this approach seems better for tweaking values than our existing file-based config map approach.

[danail-branekov]

Here are some general notes and observation that me and @kieron-dev made:

• There is duplication between configmap values and kustomization yamls

  • E.g. patching contour HTTPProxy to set spec.virtualhost.fqdn to the same fqdn value from API configmap
  • This duplication is error-prone
  • We should have a single source of truth, which is probably the config map

• install-dependencies mixes korifi dependency configuration with dependency installation

  • Might be better to do the korifi configuration of kpack, for example, alongside controllers installation?
  • cf-setup creates the root namespace and cf-admin rolebinding. This could be part of controllers installation.
  • Removing configuration means install-dependencies.sh becomes straightforward kubectl apply (or kapp deploy if you want easier uninstall)

• ConfigMap values can be overridden simply in our existing code by taking advantage of file load ordering

  • E.g. we supply default values in 00-cfgmap-foo.yaml, 00-cfgmap-bar.yaml
  • User supplies 99-cfgmap-override.yaml with some modified properties
  • The values are merged into the config struct, with the values from the latest loaded file taking precedence

• Kelsey Hightower env var suggestion (https://medium.com/@kelseyhightower/12-fractured-apps-1080c73d481c)

  • Accept config files in your app, if you like, but
  • (in code) allow overriding values using env vars
  • This would mean people would not need to edit our config maps
  • However, there are other bits of config that require values from the config maps, so maybe this doesn't help too much

[danail-branekov]

Notes on Kustomize

• HTTPProxy seems a bit of a problem (controllers/config/base/ingress.yaml)

  • It needs serviceName and serviceNamespace for (cert generation perhaps?). The namespace comes from kustomize config, and the service name from a service yaml
  • It needs fqdn to match a config setting
  • We could consider creating the HTTPProxy within API code (or a setup job), but we don't know the service name and namespace without configuring those, and that would be further duplication?
  • We don't seem to be able to use the configmap value to set fqdn when using kustomize
  • You probably couldn't combine the kustomize output with a YTT templated value, as kustomize would strip the ytt comment (probably)

Overall, it seems that using plain kustomize to configure our redistributable seems awkward. However, moving away from kustomize is awkward as well as kubebuilder generates kustomize overlays when introducing new stuff (e.g. new CRD or new webhooks). Therefore, from developers' point of view, having kustomize around is useful.

[danail-branekov]

Branch explore/1509-ytt-kustomize experiments combining ytt and kustomize.

Here are the goals of the experiment:

• Keep kustomize around as it is useful for development
• Introduce a ytt overlay that replaces values in the kustomize files from provided property file
• In order to produce deployable yamls from the property file one first runs ytt followed by running kustomize, for example

ytt -f api/config/base -f api/config/ytt -f api/config/values/kind-local-registry.yml --output-files $(TMPDIR)
kubectl kustomize build $(TMPDIR) | kubectl apply -f -

Therefore, in a release tarball we could just ship the kustomize that have proper image SHA filled by the CI and the ytt overlay. Users are not expected to mess with those files, they are just expected to come up with a properties one.

The approach above has the following benefits

• Makes our existing kustomize overlays obsolete as they can be replaced by a single property file, i.e. we could use the property file for our various CI deployments as well
• Centralised properties management, i.e. one should not set a property on multiple places (see the HTTPProxy FQDN discussion above)
• The local copy installation and the release installation is the same - run ytt with your own properties file, then run kustomize
• Controllers debug option however gets in the way of this approach:
  • It creates an additional Service
  • It patches the controller deployment command and args

Migrating to this way of doing things requires:

• replacing existing kustomize overlays with property files (mostly trivial)
• Consider how to deal with the controllers debug overlay. Maybe we should get rid of the overlay alltogether and just enable the debug option by uncommenting bits in the kustomization files. That seems to be quite a standard way of optionally enabling stuff in kustomize, e.g. to enable CertManager
• Simplify install-depencies.sh to just installing dependencies without performing kpack configuration or user setup
• Manual steps from INSTALL.MD should be ideally automated and parameterised via the properties file

With regards to automating the configuration bits, there might be an issue that kustomize is putting prefixes on all the objects it produces. That might be a problem for cases where we would need stable predictable names...

[georgethebeatle]

After gaining confidence that the kustomize + ytt approach described above would work for us we set out to explore if we can achieve similar dev and user experience by using just kustomize. The benefits of this approach would be the following:

• we as developers can continue to use the kubebuilder scaffolding
• the users of korifi will be able to edit a single file with key-value pairs overriding only what they care about
• the users of korifi will not have to install any tooling since kustomize is embedded in kubectl
• they will be able to either apply the resulting yamls directly using kubectl or pipe it to a tool like kapp for better state management

It turned out that it is actually possible to achieve the goals above. The key things we will need to chage are:

• replace configmaps that get mounted into deployments by env vars
• introcude one configmap that holds all deployment properties
• replace all kustomize patcher by substitutions that take values from the configmap and set them wherever they are needed in the resource specs.

Here is a WIP branch that demonstrated how this works: https://github.com/cloudfoundry/korifi/tree/explore/1509-kustomize

Users only need to edit the api/config/overlays/kind-local-registry/apiconfig/korifi_api_config.env file. The rest of the deployment procedure is unchanged.

This approach has the additional benefit of removing all overlays that we have had up to now with all the duplications in them and replaceing them with just one overlay that takes a simple key-value file.

We could even leverage the fact that kustomize can refer to bases from the internet and release a single overlay that refers to a release tag in github.

[georgethebeatle]

Today we carried on with the kustomize-only experiment. Here is what we tried:

• Replaced controllers configmap with env vars and set their values throug an env file like we did for api.
• Made sure that the end user properties file can contain just a subset of all values, the values in the base layer being the defaults

• Experimented with the potential release format

  • Created a sample release overlay that has the following structure:

  - korifi
    - api
      - config
        - api.env
        - kustomization.yml
    - controllers
      - config
        - controllers.env
        - kustomization.yml
    - ...

  In the kustomization file we are including the base from github via reference and have set the released image sha using kustomize edit set image

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
- github.com/cloudfoundry/korifi/api/config/base/?ref=14f3d14d5aa201500622e03e871f14d815d642c2

configMapGenerator:
- behavior: merge
  envs:
  - apiconfig/korifi_api_config.env
  name: config

images:
- name: cloudfoundry/korifi-api
  newName: cloudfoundry/korifi-api
  newTag: dev-0.3.0-bd46d89cb53bafc5b96d9ab5a9754f9172b1dac0

replacements:
- source:
    fieldPath: data.externalFQDN
    kind: ConfigMap
    name: config
    version: v1
  targets:
  - fieldPaths:
    - spec.template.spec.containers.[name=korifi-api].env.[name=EXTERNAL_FQDN].value
    select:
      kind: Deployment
      name: deployment
      version: v1
  - fieldPaths:
    - spec.virtualhost.fqdn
    select:
      group: projectcontour.io
      kind: HTTPProxy
      name: proxy
      version: v1
    ...

In this setup each component is installed like this:

kustomize build api | kubectl apply -f -

[georgethebeatle]

Investigating removing kustomize

The project structure created by kubebuilder uses kustomize to optionally configure various resources. This is problematic if we wish to use a templating solution around the korifi yaml, such as helm or ytt templates. So we are considering the impact of moving from kubebuilder generated yaml requiring kustomize, to plain yaml, potentially with templating code.

Approach

We picked the statefulset-runner as a representative component to experiment on.

Although we subsequently realised this does not define its own resources!

We generated the full distribution yaml with kubectl kustomize config/default inside statefulset-runner. Then we saved each yaml object in the output to a separate file in the korifi/dist/templates/statefulset-runner/ directory.

Roles

We can continue to generate RBAC cluster role definitions by modifying the controller-gen command in the makefile, e.g.

./bin/controller-gen \
    rbac:roleName=korifi-statefulset-runner-appworkload-manager-role \
    output:rbac:artifacts:config=../dist/templates/statefulset-runner paths="./..."

This uses the fully prefixed name for the cluster-role, and switches its output directory. The output file will be called role.yaml

CRDs

CRDs can be generated as before. Again we just need to switch the output directory, similar to above.

Webhook generation

There is a problem here. We use object selectors to make sure we only modify korifi statefulset pods. However the +kubebuilder:webhook annotation does not allow us to specific object selector rules. Therefore this is implemented with a kustomize patch, since each time the webhook manifest is regenerated, it would overwrite the object selector changes.

Therefore we propose to stop the auto-generation of the webhook configurations.

Helm

It seems like helm is up to the task of the amount of templating we require. It can insert namespaces into the middle of strings, which is required for certificate DNS entries.

It will also suit the current configmap configuration approach, avoid the need to switch to env vars.

ytt templates would also work, though we note the ubiquity of helm and its ease of templating for this use case.

[tcdowney]

Therefore we propose to stop the auto-generation of the webhook configurations.

Could ytt do this still though? I feel a ytt overlay provides the same amount of capability here as the kustomize patch, right? That said, I don't think it's a big deal to move away from auto-generation of webhook config. We could even just generate once for new ones and then make the object selector change ourselves.

[gcapizzi]

@tcdowney Yes, I think the pair tried to do this with pure interpolation and without overlays, but we could use overlays for this special case. This is an extra point for ytt vs Helm which doesn't have overlays.

[kieron-dev]

Kustomize makes the config directory rather hard to understand. If you look at a yaml, you also need to check all the kustomization.yamls in the whole directory structure to see if it gets modified elsewhere. I don't think that is great. And keeping that sort of practice by overlaying object selectors onto the webhook configuration also makes the configuration less explicit and more magic.

When we extract the kustomize output into plain yaml files, it's surprising how simple the resultant configuration is (although I'm about to explore converting the controllers component, so it maybe that opinion is premature). The goal of this part of the explore is to have a set of simple yamls containing minimal templating (with helm, or inline ytt - not overlayed ytt), and see how that affects our current practices for local development, testing and release.

There should be a branch available to look at later, and we'll update the explore doc outlining the pros and cons of this approach compared with the pure kustomize approach we've already described.

[gcapizzi]

Yeah I totally agree that templating is a lot easier to understand and to use. It feels like Kustomize was designed so that users could customise (duh) YAML from third parties without having to fork it (their homepage emphasises the "use of off-the-shelf applications", "without forking"), but our use case is different.

That said, it might be worth having just one overlay if that still allows us to autogenerate webhook configurations - it really depends on how complex those are to write from scratch and keep up-to-date.