Images present as args to a container command are not included nor substituted with kbld

[absoludity]

What steps did you take: For kubernetes native apps, it's not uncommon (I don't think?) to have deployments which trigger Jobs using a specific image. In this case, the image to use is configured with the executable, as in the following small example:

$ cat /tmp/kbld-issue/example.yml 
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
  name: kbld-test
spec:
  template:
    metadata:
    spec:
      containers:
      - args:
        - --some-img-arg=docker.io/bitnami/kubeapps-asset-syncer:2.0.1-scratch-r0
        command:
        - /some-command
        image: docker.io/bitnami/kubeapps-apprepository-controller:2.0.1-scratch-r0
        name: controller

What happened:

When running kbld only the container image is found:

$ kbld inspect -f /tmp/kbld-issue/example.yml
Images

Image     docker.io/bitnami/kubeapps-apprepository-controller:2.0.1-scratch-r0  
Metadata  -  
Resource  deployment/kbld-test (apps/v1) cluster  

1 images

Succeeded

What did you expect: I (naively) expected both images to be found, but see that could be tricky without something marking the CLI arg as an image.

Anything else you would like to add: Not sure if this is a feature request or an issue, sorry in advance if I chose the wrong one :)

Environment:

• kbld version (use kbld --version): 0.28.0
• Docker registry used (e.g. Docker HUB): N/A
• OS (e.g. from /etc/os-release): VERSION="20.04.2 LTS (Focal Fossa)"

[absoludity]

Another thought related to this: even if it's difficult to capture images not in a standard location, it would half-solve the issue if kbld ensured that when my image lock file already includes a specific image to match that it substitutes that image wherever it is found rather than only in standard locations.

I verified that currently kbld does not substitute the image in the command arg of the example above even if it has a substitution in the image lock file.

[gcheadle-vmware]

Hi @absoludity,

kbld cannot replace strings like "--some-img-arg=docker.io/bitnami/kubeapps-asset-syncer:2.0.1-scratch-r0" with a resolved image while retaining the prefix. We feel it would be too risky if kbld was able to replace every substring that looks like an URL.

However, kbld has custom search rules that you can set with a kbld configuration to match in non-standard locations. Here is an example of a search rule that finds your image:

---
apiVersion: kbld.k14s.io/v1alpha1
kind: Config
searchRules:
- valueMatcher:
    image: --some-img-arg=docker.io/bitnami/kubeapps-asset-syncer:2.0.1-scratch-r0

This does not work in your case because kbld thinks that "-some-img-arg=" is a part of the URL, and it will not resolve. We could think about extending this feature to include an alternate matcher for search rules, such as a regular expression: regex: --some-img-arg=(.+). This could be an option (seems a little too powerful for your use case), but we are open to other suggestions and/or contributions and we will bring this feature request up to our team next community meeting 2/8/2021.

[absoludity]

Thanks, I'll have a play with the config to get a feel for what's possible now.

[DennisDenuto]

Hey we (@cari-lynn and I) were thinking about this more in our community meeting and have a workaround you can try out:

At a high level, you could try setting an environment variable on your Deployment/PodSpec, and configure kbld to resolve the value configured in the environment variable.

for e.g.

deployment.yml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-deployment
  namespace: default
spec:
  replicas: 2
  selector:
    matchLabels:
      k8s-app: kube-app-label
  template:
    metadata:
      labels:
        k8s-app: kube-app-label
    spec:
      containers:
      - args:
        - --some-img-arg=$(IMAGE_TO_USE)
        command:
        - /some-command
        image: docker.io/bitnami/kubeapps-apprepository-controller:2.0.1-scratch-r0
        name: controller
        env:
        - name: IMAGE_TO_USE
          value: "docker.io/bitnami/kubeapps-asset-syncer:2.0.1-scratch-r0"

kbld-config.yml

---
apiVersion: kbld.k14s.io/v1alpha1
kind: Config
searchRules:
- keyMatcher:
    name: value

And then run: kbld -f .

Let us know if this works for your use case

[absoludity]

Thanks @DennisDenuto . I don't think that would work generally (but to be clear, this isn't a pressing issue for me - I was just playing with kbld and helm post rendering ). Often I'd be rendering charts or yaml configs over which I don't have control - so I can't just update it to switch in an env var. I'd need to be able to match on the specific arg, --some-img-arg=(?<img regex>) or similar, I'd think.

[cppforlife]

so I can't just update it to switch in an env var

a common option here is to rely on ytt to transform the yaml so that it's more friendlier to other tools.

[cppforlife]

one possible feature addition i see here is that we could teach kbld to accept ytt functions as a way to deal with more complex input. at some point trying to describe in yaml how to replace parts of arbitrary content becomes too complex and inflexible (if possible at all), so providing "break glass" feature for user to search and replace image ref within chunk of content seems to be a very appealing solution.

[cari-lynn]

A similar feature to accepting ytt functions or Starlark snippets has been discussed in kapp #11 as well. It makes sense to think of this possibility in a larger context across the tools so we can provide a consistent experience. Starlark also already supports a regex library we could utilize here.

However, we expose configuration SearchRules in kbld to configure how images are found. This seems like a more lightweight ask to support regex matching than what is being discussed in kapp, in that case, does it makes sense to go with a more lightweight solution like adding support for regex?

A solution could look like

apiVersion: kbld.k14s.io/v1alpha1
kind: Config

searchRules:
- valueMatcher:
    regex: /((\d{3})(?:\.|-))?(\d{3})(?:\.|-)(\d{4})/

[vishrantgupta]

I have a deployment that gets rendered as:

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    kbld.k14s.io/images: |
      - Metas:
        - Tag: 1.7.2-debian-10-r0
          Type: resolved
          URL: harbor-repo.vmware.com/tcx/snapshot/cert-manager:1.7.2-debian-10-r0
        URL: harbor-repo.vmware.com/tcx/snapshot/cert-manager@sha256:e48338841039ab23983e9a461add0a6815381bab5a837163eb365fea7a00769b
spec:
    spec:
      containers:
      - args:
        - --v=2
        - --cluster-resource-namespace=$(POD_NAMESPACE)
        - --leader-election-namespace=kube-system
        - --acme-http01-solver-image=harbor-repo.vmware.com/tcx/snapshot/acmesolver:1.7.2-debian-10-r0

I used the following regex:

apiVersion: kbld.k14s.io/v1alpha1
kind: Config
searchRules:
  - valueMatcher:
      regex: --acme-http01-solver-image=(.*)

But I got this error:

kbld: Error: Validating config/ (kbld.k14s.io/v1alpha1) cluster:
  Validating SearchRules[0]:
    Expected ValueMatcher.Image or ValueMatcher.ImageRepo to be non-empty

The values.yaml file looks like this:

cert-manager:
  ...
  controller:
    ...
    acmesolver:
      image:
        registry: harbor-repo.vmware.com/tcx/snapshot
        repository: acmesolver
        tag: 1.7.2-debian-10-r0
        debug: false

[vishrantgupta]

Ah! I see the regex is not supported.

[cjnosal]

---

apiVersion: kbld.k14s.io/v1alpha1 kind: Config searchRules:

• keyMatcher: name: value

We're trying a similar config. In our case we have other env vars, some of which aren't images. KeyMatcher appears to require all matching keys to be images. We tried using {index: n} in the search path but that didn't seem to work.

In our case the non-image env vars come from a ytt overlay, so we worked around it by invoking kbld on the base file before piping to ytt.