derive data from other data

[schmurfy]

I have been looking for a way to make this works:

data1.yaml

#@data/values
---
key1: value1

data2.yaml

#@ load("@ytt:data", "data")
#@data/values
---
key2: #@ data.values.key1

config.yaml

#@ load("@ytt:data", "data")

key2: #@ data.values.key2

expected output:

key2: value1

My problem is that on one side I have a value file generated from consul keys which contain the environment name as envName and on the other side an ytt template which expect that name to be passed in some_key.env_name, I don't really want to update the final template to make this work unless there is another way but I can't find any...

is there a way to achieve that ?

[gcheadle-vmware]

Hi @schmurfy,

Data values are calculated and stored into the data.values struct, prior to templating. In the example you provided above, your data values files merge together, and #@ data.values.key1 is not an accessible value.

To work around this, you could make envName accessible via function, so it can be called and stored as a data value with the name env_name. Example: https://carvel.dev/ytt/#gist:https://gist.github.com/gcheadle-vmware/c6dfb26fb44342e24efd66ecea2a5618

There are other work arounds that you could use, but that is what I initially thought of. Let us know if that works for you, or if you have any questions.

[cppforlife]

I don't really want to update the final template to make this work unless there is another way but I can't find any...

can you say a bit more about why you don't want to update existing template. this may help us understand the problem a bit better.

[cppforlife]

(@schmurfy i moved this issue into vmware-tanzu/carvel-ytt)

[schmurfy]

I created our current pipeline with already existing constraints so first I have terraform creating the cluster and all the low-level entities and then writing some of these config data into consul. After that when deploying an application our pipelines pulls data from multiple places and aggregates them (all data values):

• default-values (from the app itself)
• yaml export of the particular consul kv tree filled by terraform which have amongst other thing envName: something
• env specific values for this deployment

the problem is that I find myself with data which looks like:

#@ load("@ytt:data", "data")
---
envName: dev
externalIP: 1.2.3.4
# ...

and if I want to have a clean manifest for an app with multiple parts I have this:

#@ load("@ytt:data", "data")
---
part1:
  enabled: true

part2:
  enabled: true
  envName: #@ data.does.not.work.here

of course I could just change the expected data values structure and expect envName at the root where it will be but I was hoping for a cleaner way to do it, thanks for mentioning functions I did not think about them in that context and it might work. When exporting the data from the consul kv tree I could create functions as well as flat data (for legacy apps).

PS: the legacy apps are using helm and so the ytt data will essentially be used to produce the values.yaml that's why I inherited the previously existing structure on this pipeline. All our new apps uses ytt to define their manifests.

[schmurfy]

using a function does work well for me, thanks for the help :)

[groundnuty]

@gcheadle-vmware do you maybe have an idea how to resolve such situation with? function shall not work here... https://carvel.dev/ytt/#gist:https://gist.github.com/groundnuty/a62e1c2d4f5cde65954ddbb7a3ce6844

It's a simplified scenario of a situation where there is a bunch of libs, one of them is a util lib that checks some conditions and like here returns a list of something. The idea was that users when wanting to overwrite some values in their data values can use those utility libs. I hope my example is clear enough :-)

[gcheadle-vmware]

Hi @groundnuty, sorry for the late reply, I was out for the back half of last week. Your playground example is definitely on the right track, although I'd like to point out two things: 1) I do not see a function definition #@ def FunctionName(): in the example. 2) there are circular load statements in data.lib.yml and dataValues.yml which is causing the error that you are seeing.

I've created a gist example that addresses these two points: https://carvel.dev/ytt/#gist:https://gist.github.com/gcheadle-vmware/849f70d6c219ac84f7f5bc899b73cd0b Hopefully that helps, but I'm happy to try again if this doesn't meet your needs.