How to overlay arbitrary packages from a YAML file.

[Hugal31]

I am building a robot and I am using YTT to configure it. Each robot has specific "instance" parameters, as well as "series" parameters, common to a series of almost identical robots, and some general parameters that are usually true for every robot.

My (a bit simplified) file hierarchy looks like this:

config/
  init.yaml # Empty file that serves as a root for all the overlaying
  global/  # Config valid for all the robot, but may be overlayed by the series-specific values.
    control.yaml
    limits.values.yaml
    ...
  series/  # Config valid for a series of robot. Can also be overlayed
    1.0/  
        limits.values.yaml
        ...
    1.1/
        ...
  instance/
    robot-0001/  # Config for a specific instance of a robot.
        hardware.values.yaml
    robot-0002/
        hardware.values.yaml
        limits.values.yaml

All the files are overlay files, except for init.yaml.

Then I invoke ytt like this:

ytt -f config/init.yaml -f config/global -f config/series/1.1 -f config/instance/robot-0002

However, I often find myself duplicating things between series. For instance, series 1.1 might have similar values with 1.0. So I could overlay it like this:

ytt -f config/init.yaml -f config/global -f config/series/1.0 -f config/series/1.1 -f config/instance/robot-0002

However that can quickly make the ytt command line long and confusing. Is there a way to for example, make it explicit in robot-0002 that it overlays series/1.1, which overlays series/1.0, which overlays global? I tried using the library, overlay and template, but I can't get my head around it.

[pivotaljohn]

If I'm tracking well, the core of what we're trying to accomplish, here, is conditionally apply overlays...

That one can accomplish this at the command-line by either including or omitting directories that contain those overlays.

But this puts the onus on the person running ytt to generate the robot's config: they have to remember to include all of the prior versions (and name each one).

So, there are a couple of strategies that come to mind:

1. write a shell script that — given a series number and instance — builds up the ytt command-line for you (or collects such config, copies in a separate directory (e.g. build/ and you can ytt -f build/)
2. include all overlays, but wrap them in #@ if / #@ end where the conditional effects that dependency relationship.

If you want to talk through option 1, happy to do so, but it's an approach that doesn't involve ytt-specific logic.

Option 2, conditional overlays.

so if...

config/init.yaml

---
name: ""
serial_no: 0
sensors: []

config/global/control.yaml

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.all
---
sensors:
- name: temp
  driver: DHT22

config/series/1.0/limits.values.yaml

#@ load("@ytt:data", "data")
#@ load("@ytt:overlay", "overlay")

#@ if/end data.values.series >= 1.0:
#@overlay/match by=overlay.all
---
sensors:
#@overlay/match by=overlay.subset({"name": "temp"})
- 
  #@overlay/match missing_ok=True
  min: 40
  #@overlay/match missing_ok=True
  max: 80

config/series/1.1/limits.values.yaml

#@ load("@ytt:data", "data")
#@ load("@ytt:overlay", "overlay")

#@ if/end data.values.series >= 1.1:
#@overlay/match by=overlay.all
---
sensors:
#@overlay/match by=overlay.subset({"name": "temp"})
- 
  #@overlay/match missing_ok=True
  burst: 120

...then...

$ ytt -f . --data-value-yaml series=1.0
name: ""
serial_no: 0
sensors:
- name: temp
  driver: DHT22
  min: 40
  max: 80

and

$ ytt -f . --data-value-yaml series=1.1
name: ""
serial_no: 0
sensors:
- name: temp
  driver: DHT22
  min: 40
  max: 80
  burst: 120

[pivotaljohn]

In case you we not aware, we also have a Slack channel on the Kubernetes workspace: #carvel (if you're not already signed-up, you can quickly get an invite here: http://slack.k8s.io/). You can also ask your question, there, which can allow for a little more back-and-forth. 👍🏻

Of course, we can continue here as well.

[Hugal31]

Thanks for your quick reply, I'll try the second solution. I'll go on the Slack channel if I need more help.