Overlaying list on schema renders incorrectly

[henry-spanka]

What steps did you take:

I expect b.yaml to append item "var1" to vars list in each app. However the item is appended twice (or more depending on how many items are in apps).

schema.yaml

#@data/values-schema
---
apps:
  - name: ""
    vars:
      - ""

a.yaml

#@data/values
---
apps:
  - name: app1
  - name: app2

b.yaml

#@data/values
#@ load("@ytt:overlay", "overlay")
---
apps:
  #@overlay/match by=overlay.all, expects="1+"
  - vars:
    - var1

ytt -f schema.yaml -f a.yaml -f b.yaml --data-values-inspect

What happened:

"var1" is appended to each vars list twice.

apps:
- name: app1
  vars:
  - var1
  - var1
- name: app2
  vars:
  - var1
  - var1

What did you expect:

"var1" appended once to vars list.

apps:
- name: app1
  vars:
  - var1
- name: app2
  vars:
  - var1

Anything else you would like to add:

Setting vars to an empty list in a.yaml explicitly resolves this issue, however it's not the same semantically (default value from schema file gets lost).

#@data/values
---
apps:
  - name: app1
    vars: []
  - name: app2

Additionally, defining b.yaml as following to only match on name=app1, appends item to every list item (behaving like overlay.all).

#@data/values
#@ load("@ytt:overlay", "overlay")
---
apps:
  #@overlay/match by=overlay.map_key("name"), expects="1+"
  - name: app1
    vars:
    - var1

Environment:

• ytt version (use ytt --version): ytt version 0.49.0
• OS (e.g. from /etc/os-release): macOS 14.4.1 M1

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[prembhaskal]

https://carvel.dev/ytt/#gist:https://gist.github.com/prembhaskal/626f1264d3ab6783d7c42ccf92eaeafd added a gist for simpler reproducing of issue.

[prembhaskal]

few more illustrating similar issues

• https://carvel.dev/ytt/#gist:https://gist.github.com/prembhaskal/e9fc5954c90eecd980c99533e3770070
  • duplicated for all key not just matched
• https://carvel.dev/ytt/#gist:https://gist.github.com/prembhaskal/cffd92d9678d4109d13d9f27b15e4e2a
  • duplicated for only cases where array was already empty.

[praveenrewar]

@henry-spanka You can update your b.yaml to something like this

#@ load("@ytt:overlay", "overlay")

#@overlay/match by=overlay.all
---
apps:
  #@overlay/match by=overlay.all, expects="1+"
  - vars:
    - var1

• Remove #@data/values as this is an overlay
• Add an overlay match at apps level to match all apps

Now if you run ytt -f config --data-values-inspect | ytt -f ovr -f-, you should get

apps:
- name: app1
  vars:
  - var1
- name: app2
  vars:
  - var1

(The need to use ytt twice is because you don't have a config to load the data values and you are using data-values-inspect)

[henry-spanka]

While this might work you lose schema validation by piping the ytt output to a new invocation of ytt.

What I actually want is to overwrite values of data values with overlays while still keeping data types and validation. This works perfectly so far except the issue I described. I don't really think this is expected behaviour.

[praveenrewar]

Could you elaborate your use case a little bit. Data values are a way for authors to allow their consumers to easily configure things (templating). While overlays provide a way for the consumers to update something that is not made configurable by the authors (not templated). In your case you are trying to overlay the data values, so I think what I am trying to understand is why should the values be made configurable first via data values and then override those values via the overlays. If you could share your use case, it will be great.

[henry-spanka]

Hi, please see this project https://github.com/henry-spanka/ytt-demo

I want to have default values and then override them in an environment specific values file. Ytt by default only merges keys, however I want to modify lists and work with overlays. This works well in 99% of cases but when overlaying lists of lists or dict of list it breaks with some weird behaviour (as explained before).