[overlay] short syntax for overlaying structures within document

[phil9909]

Feature request

Add a mechanism to merge computed and YAML values.

Example use-case

Starting with the following ytt-template (context: the result is a concourse pipeline):

#@ def github(repo):
username: ((GITHUB_USER))
password: ((GITHUB_PASS))
#@yaml/text-templated-strings
uri: https://github.com/phil9909/(@= repo @).git
#@ end

resources:
  - name: ytt-github
    type: git
    source: #@ github('ytt')
  - name: ytt-lint-github
    type: git
    source: #@ github('ytt-lint')

Now I would like to use the branch stable of ytt-github, which can be achieved when setting source.branch to the desired value.

Proposed syntax

Intuitively I assumed I could achieve this like so:

 #@ def github(repo):
 username: ((GITHUB_USER))
 password: ((GITHUB_PASS))
 #@yaml/text-templated-strings
 uri: https://github.com/phil9909/(@= repo @).git
 #@ end

 resources:
   - name: ytt-github
     type: git
     source: #@ github('ytt')
+     branch: stable
   - name: ytt-lint-github
     type: git
     source: #@ github('ytt-lint')

But ytt gives me Error: Compiling YAML template 'intutive.yaml': Expected YAML node at line intutive.yaml:12 to have either computed or YAML value, but found both

An alernative syntax I could imagine is:

 #@ def github(repo):
 username: ((GITHUB_USER))
 password: ((GITHUB_PASS))
 #@yaml/text-templated-strings
 uri: https://github.com/phil9909/(@= repo @).git
 #@ end

 resources:
   - name: ytt-github
     type: git
-    source: #@ github('ytt')
+    source:
+      <<: #@ github('ytt')
+      branch: stable
   - name: ytt-lint-github
     type: git
     source: #@ github('ytt-lint')

This uses the standard yaml merge syntax: https://yaml.org/type/merge.html But ytt rejects this with: Error: Unmarshaling YAML template 'alternative.yaml': yaml: map merge requires map or sequence of maps as the value

[cppforlife]

Intuitively I assumed I could achieve this like so:...

that's an interesting idea of combining computed and inlined value together as an overlay. which one would be considered as left and which one right?

This uses the standard yaml merge syntax: https://yaml.org/type/merge.html

it seems that "In YAML 1.3, loaders will be encouraged not to support merge keys by default. It will become an optional feature." (https://github.com/yaml/yaml/issues/35#issuecomment-270271980). i would much rather not rely on "programming" concept of yaml.

[phil9909]

which one would be considered as left and which one right?

I would reason that the computed value is the left and the literal is right, because the computed value comes first. But this might not be intuitive in RTL-languages.

i would much rather not rely on "programming" concept of yaml.

Fair. Might still be worth considering that (or a similar) syntax as it would enable the user to choose which side is right and which is left:

source:
  <<: #@ github('ytt')
  branch: stable

vs.

source:
  branch: stable
  <<: #@ github('ytt')

[antmarot]

Just a small comment to say that I started using ytt today and I believe this is a much needed feature. Like @phil9909, I naively tried that syntax as well, believing it was certainly supported. It's just that intuitive.

[cppforlife]

im still chewing on some of this info and checking out some implementation difficulties.

just to let yall know, following works for posed example in the first comment.

#@ load("@ytt:template", "template")

#@ def github(repo):
username: ((GITHUB_USER))
password: ((GITHUB_PASS))
#@yaml/text-templated-strings
uri: https://github.com/phil9909/(@= repo @).git
#@ end

resources:
 - name: ytt-github
   type: git
   source:
     _: #@ template.replace(github('ytt'))
     branch: stable
 - name: ytt-lint-github
   type: git
   source:
     _: #@ template.replace(github('ytt-lint'))
     branch: master

[MPV]

Any ideas on supporting this similar use case (with merged lists, not maps)?

#@ def items(items):
#@yaml/text-templated-strings
- (@= items @)
#@ end
#@ end

resources:
 - name: ytt-github
   items: 
   #@ items('one')
 - name: ytt-lint-github
   items:
   #@ items('one')
   - two
   - three

Doing something like this today just gives: Error: Compiling YAML template 'intutive.yaml': Expected YAML node at line intutive.yaml:1234 to have either computed or YAML value, but found both

[danielhelfand]

Any ideas on supporting this similar use case (with merged lists, not maps)?

@MPV Similar to what was shown in the comment here, you could use a - #@ template.replace(items('one')) approach as shown below to add the result of the items function to an array.

One thing to point out is the second #@ end is not needed in your example. Also, the document start --- is required in order to separate the sequence context from the mapping context.

Does this help address your use case?

#@ load("@ytt:template", "template")

#@ def items(items):
#@yaml/text-templated-strings
- (@= items @)
#@ end
---
resources:
 - name: ytt-github
   items: 
   - #@ template.replace(items('one'))
 - name: ytt-lint-github
   items:
   - #@ template.replace(items('one'))
   - two
   - three

[pivotaljohn]

Fascinating conversation...

I can imagine a way to combine these ideas into a rather feasible approach (from @phil9909 's original example)....

 #@ def github(repo):
 username: ((GITHUB_USER))
 password: ((GITHUB_PASS))
 branch: main
 #@yaml/text-templated-strings
 uri: https://github.com/phil9909/(@= repo @).git
 #@ end

 resources:
   - name: ytt-github
     type: git
-    source: #@ github('ytt')
+    source:
+      _: #@<< github('ytt')
+      branch: stable
   - name: ytt-lint-github
     type: git
     source: #@ github('ytt-lint')

Which:

• introduces a new annotation @<< that precisely supersedes template.replace() (looooong deprecation period).
• maps that contain such meta (i.e. one or more of it's items is annotated with a shovel-shovel) would infer an @overlay/merge on all its children.
  • as @phil9909 suggested, such implied overlays would execute as normal: top-to-bottom.