Decision to support KRM functions with builtin kustomize plugin

[snuggie12]

Summary

Starting in version 3.9.X of kustomize you can now run a custom plugin which is a container image-based KRM function. This is great since before you needed to have a plugins directory storing your code and having all your engineers have that setup is a pain.

However, it requires you have a program called "docker" (doesn't necessarily actually need to be docker, just as long as commands like docker pull or docker run does the same thing as docker,) in your $PATH so the container can be fetched and run.

I think this would present a problem for the repo-server. There's a few things out there documenting security concerns and what not of running docker in a docker container in your environment. I personally prototyped using https://github.com/mgoltzsche/podman-static on the repo-server and it looks like it would work. FWIW, I also filed https://github.com/kubernetes-sigs/kustomize/issues/3536 but it looks like this will be closed with no change.

So yeah, essentially asking for official documentation saying whether this will be supported or not and if not maybe a work around such as a custom argo plugin or perhaps something with workflows or events can be done (I don't use those myself so not really sure.)

This should also probably be linked to #5293 since kpt is mentioned which also uses KRM functions in containers.

Motivation

The main use case I have for a KRM function is rendering dynamic data from one resource into another, but you can make a KRM function that does any style of generation or transformation in kustomize.

Proposal

If you did decide to support it, I see two options though open to any others:

• My POC did work and I feel could be production worthy if something like SELinux was added to ensure only the needed commands could be executed.
• The other setup I read about is a docker sidecar. If that turns out to be better for you all from a security profile standpoint great, but the first option was easier to prototype.
• Perhaps there is an even slimmer docker runtime which is secure? I think all it needs is docker pull and docker run and I already proved the symlink concept works.

[epcim]

What is the status as of today? Kustomize added KRM functions recently.

[epcim]

@snuggie12 @jannfis any update? What are Argo plans to support Kustomize/KPT KRM Fn ?

[snuggie12]

The issue I referenced is closed now and I believe the documentation is located here. I think the best bet would do the sidecar approach and use the same setup I referenced with a statically built podman symlinked to docker.

Disappointingly, currently kustomize is preparing to remove their helm chart plugin and revamp it into a KRM function. If this goes through I think the best option will be to fork kustomize. Essentially my plan is to use/create golang plugins but treat them as if they were built-in. This removes any need for laptop setup or mucking with the repo server.

Slightly ranty, but all of this might get solved automatically as docker is changing up licensing and what not. Fingers crossed someone just invents a better tool to run containers that doesn't require the security exceptions.

[ciis0]

What about building a small program that interprets docker run commands and converts them to appropriate kubectl run commands? Then the repo-server can intercept the calls kustomize tries make to docker binary and, instead, (given the appropriate permissions) run containers in new temporary pods and forward stdout/stderr from there. (I hope a pod running creating other temporary pods is not a antipattern.)

This would put some limits on the containerized KRM function plugins because the mounts and network options that are present in kustomize cannot work the same as in actual docker (or aliased podman), but I think that's a sensible limitation when CD is involved. --security-opt I'd consider not necessary as kubernetes already isolates pods properly. And while kubectl run does not support --user it might be possible to patch it in via --overrides.

I don't see bundling kubectl as the only option, probably instead the repo-server itself could integrate logic for running pods based on docker run parameters, with the actual docker command simply being a bridge to that logic in repo server.

[epcim]

There are multiple options implementation can take. One is that Kustomize will offer an interface what is being called to run function) another is that Argo will implement role of orchestrator - so it will run the functions on its own.

I would even consider that ArgoCD Application spec.source.plugin could accept an KRM Fn as plugin. Some global template processing could be easily done this way on CD level.

In other word it would be even nice to chain source engines what ever way is needed/per app, ie: helm -> ... -> KRM Fn -> Kustomize -> ... -> ... etc whatever way/order. I would mention here this project that as far as I do understnad offload some logic to plugin instead of argo: https://github.com/crumbhole/argocd-lovely-plugin

I have spent some time with the KRM Fn. Mostly some template rendering (both configuration for apps and manifests).

• mounts are sort of needed (private repos, cache repos)
• env variables - Kustomize is going to remove the option to pass them to Fn :( - and as such there is no way to pass token/key to get private repos etc..

Overall, it would be great if ArgoCD team would update how they do see that and what is essentially going to be placed on the roadmap as of 2022.

[tedtramonte]

@snuggie12 Have you made any headway with Containerized KRM functions in ArgoCD? I went through the trouble of creating one (https://github.com/kubernetes-sigs/kustomize/issues/4555) and I'm extremely disappointed it doesn't "just work" in ArgoCD. You say you got a proof of concept working, but I'm struggling to piece it all together.

[epcim]

@snuggie12 Have you made any headway with Containerized KRM functions in ArgoCD? I went through the trouble of creating one (kubernetes-sigs/kustomize#4555) and I'm extremely disappointed it doesn't "just work" in ArgoCD. You say you got a proof of concept working, but I'm struggling to piece it all together.

@tedtramonte have you made any progress?

[tedtramonte]

@epcim I got the podman image running as a sidecar to ArgoCD, and that's about where I stopped. I've had a rough week, so it could be that, but I just can't grok the ArgoCD plugin documentation. Nothing I tried would get my Application to run through the sidecar.

I have a branch in my Ansible repo with a dump commit that has what I've configured so far, if you want to take a look: https://gitlab.com/door-to-darkness/ansible/-/commit/f37f6af49fbdf8fa3beef787e05b118c4b0601fb

[snuggie12]

We don't really have this issue anymore so did this on personal time, but I spent a few hours and am almost there. I just need to solve one or two more podman/permissions issues and it should be working.

However, given that every other piece is working such as the sidecar itself with argocd, perhaps someone can figure out how to make the correct Dockerfile which finishes this:

Dockerfile

``` FROM REDACTED/circleci/buildpack-deps as builder USER root COPY kustomize-downloader.sh /bin/. RUN /bin/kustomize-downloader.sh # currently trying out non-minimal. No luck AFAIK FROM mgoltzsche/podman:4.1 as sidecar USER root COPY --from=builder /kustomize-files/* /bin/ # Makes 3.9.3 the default kustomize being run COPY --from=builder /kustomize-files/kustomize-v3.9.3 /usr/local/bin/kustomize RUN ln -s /usr/local/bin/podman /usr/local/bin/docker \ # This is me trying to get rootless podman working # "ping" already had GID 999 which argocd says you need UID 999 so unsure whether a matching GID is also necessary && adduser -h /home/argocd -s /bin/sh -G ping -u 999 -D argocd \ && echo 'argocd:165536:65536' >> /etc/subuid \ && echo 'argocd:165536:65536' >> /etc/subgid ```

kustomize-downloader.sh

``` # We already had this file sitting around for something else so figured I'd just bring it. obviously you can do these steps manually in the builder or use this as well # don't forget to make executable #!/bin/bash set -x kust_versions=" \ v3.9.3,6066ecf96e337fc30491daa027579edd8788bb21b92a84c02df53bc345745f70 \ v4.2.0,2eac02d7b1a9901305e96be3ea500ad22cd123080f9a98bcd666e846440fd6ab \ " for version_info in $kust_versions; do version=$(echo $version_info | cut -d',' -f 1) hash=$(echo $version_info | cut -d',' -f 2) curl -fSL "https://github.com/kubernetes-sigs/kustomize/releases/download/kustomize%2F${version}/kustomize_${version}_linux_amd64.tar.gz" \ | tar -xzC /tmp \ && echo "${hash} /tmp/kustomize" \ | sha256sum -c \ && mkdir -p "/custom-tools/kustomize-${version}" "/kustomize-files" \ && cp /tmp/kustomize "/kustomize-files/kustomize-${version}" \ && cp /tmp/kustomize "/custom-tools/kustomize-${version}/kustomize" \ && rm /tmp/kustomize done ```

cmp-cm

``` apiVersion: v1 kind: ConfigMap metadata: name: kustomize-docker-plugin data: plugin.yaml: | apiVersion: argoproj.io/v1alpha1 kind: ConfigManagementPlugin metadata: name: kustomize-docker-plugin spec: discover: find: # this seems to be very touchy. I made it so any argo apps which contain this file and is set to be a plugin # will use this sidecar. I've seen others simply do a "touch" on a /tmp file, but this worked for me and suggest you do something similar command: [sh, -c, find . -name template-generator.yaml] generate: command: ["sh", "-c"] args: ["kustomize build --enable_alpha_plugins"] # init (or the command I'm trying) doesn't seem to be working for me. # I might need to add this to my command before executing the sidecar init: command: ["sh", "-c"] args: ["mkdir -p /tmp/docker"] allowConcurrency: true lockRepo: false ```

argocd-repo-server-deploy.patch.yaml

``` # I use kustomize so adjust as necessary. apiVersion: apps/v1 kind: Deployment metadata: name: argocd-repo-server namespace: argocd spec: template: spec: containers: - name: kustomize-docker # This is where I might create /tmp/docker or perhaps a different mountpoint command: [/var/run/argocd/argocd-cmp-server] # gets patched in via image transformer image: app:latest securityContext: # capabilities is me trying to solve podman permissions issues capabilities: add: - SETGID - SETUID runAsNonRoot: true # Not sure privileged works with rootless and subgid/subuid business privileged: true runAsUser: 999 # maybe I should try uncommenting this and only UID 999 matters? # runAsGroup: podman volumeMounts: - mountPath: /var/run/argocd name: var-files - mountPath: /home/argocd/cmp-server/plugins name: plugins # This changes in 2.4 and you'll want to create a new volume called cmp-tmp separate from repo-server's tmp volume. More on this below - mountPath: /tmp name: tmp - mountPath: /home/argocd/cmp-server/config/plugin.yaml subPath: plugin.yaml name: kustomize-docker-plugin volumes: - configMap: name: kustomize-docker-plugin name: kustomize-docker-plugin # Ephemeral volumes are only available in k8s 1.21+ # Will need to switch to this in 2.4 # - ephemeral: # volumeClaimTemplate: # metadata: # creationTimestamp: null # labels: # type: argocd-repo-server-cmp-sidecar-tmp # spec: # accessModes: # - ReadWriteOnce # resources: # requests: # storage: 10Gi # storageClassName: gcp-zone-aware-ssd # volumeMode: Filesystem # name: cmp-tmp ```

argo app

``` # This part is very important on getting podman working. Figuring out these env vars has been brutal # It's probably better to load these in your container spec, # but this is the one file not under version control and so it's been easy iteration for me. # feel free to change up as needed apiVersion: argoproj.io/v1alpha1 kind: Application metadata: name: test-kustomize-docker namespace: argocd spec: destination: namespace: argocd server: https://kubernetes.default.svc project: infra source: repoURL: REDACTED targetRevision: main path: REDACTED plugin: # there is a super easy part to miss here! If using sidecar mode do not use the "name" field # it's in the instructions, but I def missed it and so have others when looking at issues. # if you don't use any env's then simply put "plugin: {}" env: # despite coming from a docker container you still need to have a plugin home specified # I think XDG_CONFIG_HOME also works but at the time I went with this one - name: KUSTOMIZE_PLUGIN_HOME value: /tmp/docker # All of these are trying to give various podman components a directory they can write to # one mistake I might be making is not making a separate volume for these files # do NOT confuse this with the "discover" section though. Pre-2.4 you need to share the same volume - name: XDG_RUNTIME_DIR value: /tmp/docker - name: XDG_DATA_HOME value: /tmp/docker - name: XDG_CONFIG_HOME value: /tmp/docker - name: TMPDIR value: /tmp/docker ```

kustomization.yaml for app

``` --- apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization # Obviously should be "transformers" if that's what your plugin is generators: # Note the file name matches what's in the discover of the plugin config - template-generator.yaml ```

template-generator.yaml

``` # straight forward and very cool krm function by airshipctl # makes a configmap that renders golang templating # sounds funny saying this for the first time, but # it works on my laptop lol # I just mean that hopefully once podman subgidmapping works # this won't throw anymore hurdles at me apiVersion: airshipit.org/v1alpha1 kind: Templater metadata: name: cm-template annotations: config.kubernetes.io/function: | container: image: quay.io/airshipit/templater:latest values: hello: world template: | apiVersion: v1 kind: ConfigMap metadata: name: test-rendered-cm namespace: argocd data: hello: {{ .hello }} ```

All of this gets you to the point where the sidecar is def executing kustomize build but various issues involving mixing argocd's user with something setup for the podman user to work are coming into conflict. If I have some time I'll add the app back and paste the error I'm currently at.

[snuggie12]

Well good news/slightly embarrassing, but turns out the final change I made last night/this morning pushed it over the edge. I have it working!

That still leaves a lot of work to do to figure out which one of all my hacks are absolutely necessary. The final one that took it over the edge was using the non-minimal image of podman-static. I guess one could start from there and see if things are still busted.

If I get some free time I'll try to list out the rest of the changes, but they're all in the comments above.

[criztovyl]

I suspect the limitation of your approach is running the sidecar privileged or getting /dev/fuse into the container, see https://www.redhat.com/sysadmin/podman-inside-kubernetes, "Rootless Podman without the privileged flag":

To eliminate the privileged flag, we need to do the following:

• Devices: /dev/fuse is required to use fuse-overlayfs inside of the container, this option tells Podman on the host to add /dev/fuse to the container so that containerized Podman can use it.
• Disable SELinux: SELinux does not allow containerized processes to mount all of the file systems required to run inside a container. So we need to disable SELinux on the host that is running the Kubernetes cluster.

I am still intrigued whether kubectl run (or similar functionality baked into repo-server) might be a easier solution with less external dependencies, but did not took time to further look into this yet.

[heyleke]

Hey, I will be pitching an alternative approach for running KRM funtions in containers at ArgoCon Europe 2023: https://colocatedeventseu2023.sched.com/event/1Jo9s/cl-lightning-talk-using-kustomize-krm-functions-to-enhance-argo-cd-application-deployments-jan-heylen-nokia

[renaudguerin]

Hi, I'm considering KRM functions to work around Kustomize limitations with replacements. Did this ever get somewhere ?

[criztovyl]

last time i checked containerized KRM functions reliance on stdin/stdout makes them hard to use on kubernetes.

https://github.com/kubernetes-sigs/kustomize/pull/4921

Edit: https://github.com/argoproj/argo-cd/issues/5553#issuecomment-2118988826

[criztovyl]

there also is the hurdle that kustomize currently expects "docker run" to run containers

https://github.com/criztovyl/automatic-umbrella

[aabouzaid]

I believe that the issue title is a bit out of date or not so accurate because it states support KRM functions where the body covers only Containerized KRM Functions.

There is no mention of Exec KRM functions which is still possible to run KRM functions and it only needs the binary in the system.

[criztovyl]

true, regular exec krm functions should work fine!