Declare a variable dynamically in one file that is available in next file

damienleger commented 2 years ago

Hello,

I'm working at the moment on a big backend repository which contains many applications manifests (app1.yml ... appN.yml). They have their config externalized in a common config.yml. I want to add various labels/annotations on all apps, and to be sure they are all set correctly I use an overlay.yml to set them.

My problem is that I don't want to split config.yml files into several config files per application but I need the overlay.yml to be aware of which app the ytt is invoked. I come up with the idea below but I would prefer to be able to declare the app var directly in app1.yml ... appN.yml to avoid to add an extra var to the command lines and those ugly if in overlay.yml. Do you have any ideas how/if this is possible? I tried add a new variable in appN.yml files with and without "data" values unsuccessful.

Here is my working workaround (with files shorten and simplified):

generation with: ytt -f config.yml -f app1.yml -f overlay.yml -v app=app1 ytt -f config.yml -f app2.yml -f overlay.yml -v app=app2

config.yml

#@data/values
---
common:
  env: "12345" #! filled dynamically via preliminar envsubst envvar substitution
  #! etc.

app1:
  name: app1_name
  #! etc.

app2:
  name: app2_name
  #! etc.

#! appN:
#! etc.

app1.yml

#@ load("@ytt:data", "data")
---
kind: Deployment
metadata:
  name: #@ data.values.app1.name
spec:
#! etc.

app2.yml

#@ load("@ytt:data", "data")
---
kind: Deployment
metadata:
  name: #@ data.values.app2.name
spec:
#! etc.

overlay.yml

#@ load("@ytt:data", "data")
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:regexp", "regexp")

#@ def metadata(app):
labels:
  #@ if app == "app1":
  app.kubernetes.io/component: app1
  #@ name=data.values.app1.name

  #@ elif app == "app2":
  app.kubernetes.io/component: app2
  #@ name=data.values.app2.name
  #@ end

  app: #@ name
  app.kubernetes.io/name: #@ name
  app.kubernetes.io/instance: #@ name

  #@ if regexp.match("[0-9]+", data.values.common.env):
  #@ env="feature-"+data.values.common.env
  #@ else:
  #@ env=data.values.common.env
  #@ end
  env: #@ env
  app.kubernetes.io/part-of: company-backend

annotations:
  a8r.io/repository: "https://bitbucket.org/company/company_backend/"
#@ end

#@overlay/match by=overlay.all, expects="0+"
---
#@overlay/match-child-defaults missing_ok=True
metadata: #@ metadata(data.values.app)
...

#!overlay/match by=overlay.map_key("kind"), expects="0+"
#@overlay/match by=lambda _, l, r: l["kind"] in ('Deployment', 'Job'), expects="0+"
---
#@overlay/match-child-defaults missing_ok=True
spec:
  template:
    metadata: #@ metadata(data.values.app)
...

which result in:

$ ytt -f config.yml -f app1.yml -f overlay.yml -v app=app1
kind: Deployment
metadata:
  name: app1_name
  labels:
    app.kubernetes.io/component: app1
    app: app1_name
    app.kubernetes.io/name: app1_name
    app.kubernetes.io/instance: app1_name
    env: feature-12345
    app.kubernetes.io/part-of: company-backend
  annotations:
    a8r.io/repository: https://bitbucket.org/company/company_backend/
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: app1
        app: app1_name
        app.kubernetes.io/name: app1_name
        app.kubernetes.io/instance: app1_name
        env: feature-12345
        app.kubernetes.io/part-of: company-backend
      annotations:
        a8r.io/repository: https://bitbucket.org/company/company_backend/

$ ytt -f config.yml -f app2.yml -f overlay.yml -v app=app2
kind: Deployment
metadata:
  name: app2_name
  labels:
    app.kubernetes.io/component: app2
    app: app2_name
    app.kubernetes.io/name: app2_name
    app.kubernetes.io/instance: app2_name
    env: feature-12345
    app.kubernetes.io/part-of: company-backend
  annotations:
    a8r.io/repository: https://bitbucket.org/company/company_backend/
spec:
  template:
    metadata:
      labels:
        app.kubernetes.io/component: app2
        app: app2_name
        app.kubernetes.io/name: app2_name
        app.kubernetes.io/instance: app2_name
        env: feature-12345
        app.kubernetes.io/part-of: company-backend
      annotations:
        a8r.io/repository: https://bitbucket.org/company/company_backend/

pivotaljohn commented 2 years ago

As a first iteration, we can absorb the if statement but using the bracket notation for accessing the app's data values:

#@ load("@ytt:data", "data")
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:regexp", "regexp")

#@ def metadata(app):
labels:
  app.kubernetes.io/component: #@ app
  #@ name=data.values[app].name

  app: #@ name
  app.kubernetes.io/name: #@ name
  app.kubernetes.io/instance: #@ name

  #@ if regexp.match("[0-9]+", data.values.common.env):
  #@ env="feature-"+data.values.common.env
  #@ else:
  #@ env=data.values.common.env
  #@ end
  env: #@ env
  app.kubernetes.io/part-of: company-backend

annotations:
  a8r.io/repository: "https://bitbucket.org/company/company_backend/"
#@ end

#@overlay/match by=overlay.all, expects="0+"
---
#@overlay/match-child-defaults missing_ok=True
metadata: #@ metadata(data.values.app)
...

#!overlay/match by=overlay.map_key("kind"), expects="0+"
#@overlay/match by=lambda _, l, r: l["kind"] in ('Deployment', 'Job'), expects="0+"
---
#@overlay/match-child-defaults missing_ok=True
spec:
  template:
    metadata: #@ metadata(data.values.app)
...

But, I presume you might want to generate all manifests for all your applications in one ytt invocation. Further, I presume that you might have a different set of manifests for each application. And even different content for the Deployment for each application.

If that's true, ...

One approach would be to:

capture each application as a ytt library.
capture the metadata overlays as a function, so that we can a) apply it multiple times, b) configure it with the specific values we want it to use each time we apply it, c) control which files are affected.
loop over the set of libraries, one at a time, applying a customized version of the overlay on each (i.e. with their individual data values.

I could look something like this:

├── _ytt_lib
│   ├── app1
│   │   └── deployment.yml
│   └── app2
│       └── deployment.yml
├── apps.yml
├── config.yml
└── overlay.lib.yml

where:

config.yml is unchanged
app1.yml got renamed to _ytt_lib/app1/deployment.yml and slightly adjusted
app2.yml got renamed to _ytt_lib/app2/deployment.yml with the same edit
overlay.yml got renamed to overlay.lib.yml and wrapped in a function
apps.yml pulls it all together

By creating this structure underneath a _ytt_lib directory, we are declaring a pair of libraries.

When we declare a library, it gets its own data values. So, when we look at the app1 and app2 libraries, they reference data values they expect for themselves:

#! _ytt_lib/app1/deployment.yml

#@ load("@ytt:data", "data")
---
kind: Deployment
metadata:
  name: #@ data.values.name
spec:
#! etc.

app1 library does not need to know about any other data values other than data.values.app1.
so when when we write templates inside the app1 library, we'll assume the data values for that library has the same shape as they appear in the root library's data values (i.e. config.yml).
so, instead of data.values.app1.name, the app1 library will assume it's own data values will be the section underneath data.values.app1... and it can then just simply refer to data.values.name.

#! _ytt_lib/app2/deployment.yml

#@ load("@ytt:data", "data")
---
kind: Deployment
metadata:
  name: #@ data.values.name
spec:
#! etc.

app2 library makes the same assumption about its data values.
I suspect that _ytt_lib/app1/deployment.yml and _ytt_lib/app2/deployment.yml are identical only because we've reduced this example to the bare minimum... that likely these two files have lots of differences.

#! overlay.lib.yml

#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:regexp", "regexp")
#@ load("@ytt:data", "data")

#@ def metadata(app):
labels:
  app.kubernetes.io/component: #@ app
  #@ name=data.values[app].name

  app: #@ name
  app.kubernetes.io/name: #@ name
  app.kubernetes.io/instance: #@ name

  #@ if regexp.match("[0-9]+", data.values.common.env):
  #@ env="feature-"+data.values.common.env
  #@ else:
  #@ env=data.values.common.env
  #@ end
  env: #@ env
  app.kubernetes.io/part-of: company-backend

annotations:
  a8r.io/repository: "https://bitbucket.org/company/company_backend/"
#@ end

#@ def metadata_overlay(app):
#@overlay/match by=overlay.all, expects="0+"
---
#@overlay/match-child-defaults missing_ok=True
metadata: #@ metadata(app)
...

#!overlay/match by=overlay.map_key("kind"), expects="0+"
#@overlay/match by=lambda _, l, r: l["kind"] in ('Deployment', 'Job'), expects="0+"
---
#@overlay/match-child-defaults missing_ok=True
spec:
  template:
    metadata: #@ metadata(app)
#@ end
...

a file that has the .lib.yml extension is called a "loadable module". It does not contribute to the output. Rather, it defines one or more variables/functions that can be load()ed and used in templates. See load() for details.
we've wrapped both the overlays in the function metadata_overlay().
this function takes the name of the application and customizes the overlay based on that argument rather than a data value.

#! apps.yml

#@ load("@ytt:data", "data")
#@ load("@ytt:library", "library")
#@ load("@ytt:overlay", "overlay")
#@ load("overlay.lib.yml", "metadata_overlay")
#@ load("@ytt:template", "template")

#@ for app_id in ["app1", "app2"]:
#@   app = library.get(app_id)
#@   app_with_dvs = app.with_data_values(data.values[app_id])
#@   app_raw_output = app_with_dvs.eval()
#@   app_with_metadata = overlay.apply(app_raw_output, metadata_overlay(app_id))
--- #@ template.replace(app_with_metadata)
#@ end

this new file loads and integrates the output of all the ytt libraries we've defined above (one for each app)
1. load a library of the same id as the app
2. we set data values for that library to be just the application's section (e.g. app1:...) of the root library's data values (config.yml)
3. we evaluate that library which means rendering all of its templates, given its data values into a document set.
4. we create a copy of the metadata overlay, customized for this app, and overlay that document set with it.
5. we render that result into our root library's output, by first creating a document (that's what the --- literal is: an empty document), and then replacing that with the document set we just created (see template.replace())

...

Please let us know whether this works for you (or not). We'd love to hear more details about what you're up to: please share as much as you can. This way we can offer advice that's fitting for your situation.

In case you don't know, we do have a Slack channel on the Kubernetes workspace. Find us a #carvel.

pivotaljohn commented 2 years ago

@damienleger — that was a lot to digest. Was it explained well?

damienleger commented 2 years ago

Hello @pivotaljohn, sorry for the lack of news. I was experimenting with an easier repo before going back to work on that one. I haven't finished yet but I've fixed all my blockers. Thanks a lot for that detailed answer that make me consider lib.

During implementation I discovered new blockers:

labels on resources and on selector aren't the same (e.g. on a Service/Deployment, selector must be immutable so version label must be removed on those ones)
some labels aren't needed for k8s resources shared (e.g. serviceAccount, roleBinding)
I had Cronjob/Ingress generated by looping on array, so to label each resource separately inside this loop from external overlay like my first idea was a headache
Common labels and specifics annotations on some resources (e.g. annotations for cert-manager on Ingress, annotations for datadog open-metric scraping on Pod)

I found a nice approach imo and rather easy to read by using:

lib with an overlay function that took many parameters (solve the ugly "if" problem, and the variable scope problem)
overlay.apply() to call the overlay inside my k8s resources and to merge common labels with built labels from above overlay function
template.replace() to add annotations after existing ones in the yml doc

Here is an extract that illustrate my solution.

To run with ytt -f config.yml -f ytt.lib.yml -f app1.yml

config.yml (envvar replaced via envsubst call prior to ytt call)

#@data/values
---
acme:
  dnsMode: true #! if false = httpMode

config:
  namespace: ${NAMESPACE}
  appName: company-frontend-app-${ENV_ID}
  component: frontend
  labels:
    env: "${ENV_ID_LONG}"
    app: "company-frontend"
    app.kubernetes.io/version: "${IMAGE_NAME}"

ytt.lib.yml

#@ load("@ytt:data", "data")

#@ def labels(name, component, instance, env_specific=True, selector=False):
#@overlay/match missing_ok=True
app.kubernetes.io/component: #@ component
#@overlay/match missing_ok=True
app: #@ name
#@overlay/match missing_ok=True
app.kubernetes.io/name: #@ name

#@ if env_specific:
#@overlay/match missing_ok=True
app.kubernetes.io/instance: #@ instance
#@ else:
#@overlay/remove
env:
#@ end

#@ if (not env_specific) or selector:
#@overlay/remove
app.kubernetes.io/version:
#@ end

#@overlay/match missing_ok=True
app.kubernetes.io/part-of: company-frontend
#@ end

#@ def annotations():
#@overlay/match missing_ok=True
a8r.io/repository: "https://bitbucket.org/company/company_frontend_web/"
#@ end

app1.yml

#@ load("@ytt:data", "data")
#@ load("@ytt:struct", "struct")
#@ load("ytt.lib.yml", "labels", "annotations")
#@ load("@ytt:overlay", "overlay")
#@ load("@ytt:template", "template")

#@ def ingressName(appName):
#@   return appName+"-ingress"
#@ end

#@ name=data.values.config.labels.app
#@ component=data.values.config.component
#@ instance=data.values.config.appName

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: #@ instance
  namespace: #@ data.values.config.namespace
  labels: #@ overlay.apply(data.values.config.labels, labels(name, component, instance))
  annotations: #@ annotations()
spec:
  selector:
    matchLabels: #@ overlay.apply(data.values.config.labels, labels(name, component, instance, selector=True))
#! truncated

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: #@ ingressName(instance)
  namespace: #@ data.values.config.namespace
  labels: #@ overlay.apply(data.values.config.labels, labels(name, component, instance))
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/ssl-redirect: "true"
    nginx.ingress.kubernetes.io/proxy-body-size: 256m
    #@ if data.values.acme.dnsMode:
    cert-manager.io/cluster-issuer: letsencrypt-production-dns
    #@ else:
    acme.cert-manager.io/http01-edit-in-place: "true"
    cert-manager.io/cluster-issuer: letsencrypt-production
    #@ end
    _: #@ template.replace(annotations())
#! truncated

I think this combination of (overlay library + overlay.apply + template.replace) should really need to be showed on the documentation somehow. This "yml anchor" on steroid approach will solve 90% of my templating needs. For the 10% others, ie. the yml I don't own (e.g. ingress controller, cert-manager) I prefer keep the original yml manifest and add an extra overlay.yml to patch it (keeping the original yml manifest ease the upgrade process).

pivotaljohn commented 2 years ago

Oh, yeah, lookin' sweet. 👍🏻

Great use of overlay.apply() for focused edits... alongside the simpler template.replace() when needed.

This "yml anchor" on steroid approach will solve 90% of my templating needs.

That's great to hear. I agree, this all seems rather readable/maintainable by my eye.

I think this combination of (overlay library + overlay.apply + template.replace) should really need to be showed on the documentation somehow

Couldn't agree more. One of the very next tracks of development we're taking on as a team is developing guides for how to use all this stuff. Heads-up: you're likely to find a section in there that's eerily "inspired" by your example, here. ;)

I prefer keep the original yml manifest and add an extra overlay.yml to patch it (keeping the original yml manifest ease the upgrade process).

Yes, we recommend this approach for exactly the reasons you give.

...

Really enjoying seeing this all come together for you, @damienleger. 🎉

Also we really appreciate the feedback as well. Keep that coming as you run into limits/awkwardness/inspiration.

I'm going to leave this open for now; a space to either decide you're done with this thread or continue the conversation — whichever makes more sense to you.

carvel-dev / ytt

Declare a variable dynamically in one file that is available in next file #601