Unable to merge data values into private library when they contain values not declared in the private library

[blc1996]

What steps did you take: TLDR: The annotations like #@overlay/append #@overlay/match-child-defaults missing_ok=True are ignored when applying data values to ytt lib using

#@ def some_values():
#@   return data.values
#@ end

#@ --- #@ template.replace(some_lib.with_data_values(some_values()).eval())

The folder structure is like:

config
config/_ytt_lib
config/_ytt_lib/foo/templates/*
config/_ytt_lib/foo/values.yaml
config/foo.yaml
config/values.yaml

config/foo.yaml

#@ load("@ytt:data", "data")
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")

#@ some_lib = library.get("foo")

#@ def some_values():
#@   return data.values
#@ end

--- #@ template.replace(some_lib.with_data_values(some_values()).eval())

config/_ytt_lib/foo/values.yaml

#@data/values
#@overlay/match-child-defaults missing_ok=True

---
foo:
    tolerations: []

config/values.yaml

#@data/values
#@overlay/match-child-defaults missing_ok=True
---
foo:
    tolerations:
    #@overlay/append
    - key: CriticalAddonsOnly
      operator: Exists
    #@overlay/append
    - effect: NoSchedule
      key: node-role.kubernetes.io/master

What happened: When using ytt ytt --ignore-unknown-comments -f config

Got the following error

$ ytt --ignore-unknown-comments -f config
ytt: Error: 
- library.eval: Evaluating library 'addons/packages/addons-manager/bundle/config': Overlaying data values (in following order: _ytt_lib/addons/packages/addons-manager/bundle/config/values.yaml, additional data values): Overlaying additional data values on top of data values from files (marked as @data/values): Document on line ?: Map item (key 'foo') on line ?: Map item (key 'tolerations') on line ?: Array item on line ?: Expected array item to have 'overlay/match' annotation
    in <toplevel>
      addons-manager.yaml:11 | --- #@ template.replace(addons_manager_lib.with_data_values(addons_manager_values()).eval())

What did you expect: The template rendering should finish successfully. The annotations in values.yaml outside the ytt lib should not be ignored.

Anything else you would like to add: If I do config/foo.yaml

#@ load("@ytt:data", "data")
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")

#@ some_lib = library.get("foo")

#@ def some_values():
foo:
    tolerations:
    #@overlay/append
    - key: CriticalAddonsOnly
      operator: Exists
    #@overlay/append
    - effect: NoSchedule
      key: node-role.kubernetes.io/master
#@ end

--- #@ template.replace(some_lib.with_data_values(some_values()).eval())

The template renders without issue

Environment:

• ytt version (use ytt --version):
• OS (e.g. from /etc/os-release):

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[pivotaljohn]

Thanks for reporting this, @blc1996. What version of ytt are you seeing this with?

[pivotaljohn]

I believe this issue was resolved in a separate channel.

@blc1996, if I am mistaken, please comment here to re-open this issue.

[pivotaljohn]

Reported in another channel:

Hi @cppforlife, I tried your suggestion to add missing_ok

#@ load("@ytt:data", "data")
#@ load("@ytt:library", "library")
#@ load("@ytt:template", "template")
#@ kapp_controller_lib = library.get("addons/packages/kapp-controller/bundle/config")
#@ def kapp_controller_values():
#@overlay/match-child-defaults missing_ok=True
--- #@ data.values
#@ end
--- #@ template.replace(kapp_controller_lib.with_data_values(kapp_controller_values()).eval())

but got the following error message

ytt: Error: 
- library.eval: Evaluating library 'addons/packages/kapp-controller/bundle/config': Overlaying data values (in following order: _ytt_lib/addons/packages/kapp-controller/bundle/config/values.yaml, additional data values): Document on line ?: Expected docset, but was *yamlmeta.Map
    in <toplevel>
      kapp-controller.yaml:12 | --- #@ template.replace(kapp_controller_lib.with_data_values(kapp_controller_values()).eval())

Any suggestions on how can I make it work? Thanks!

A Zoom session ensued from which came these outcomes ...

some take aways:

• maintaining backward compat rules for package data values will become increasingly important (in a coming ytt schema world this will become more explicit)
• ytt will add a feature to support plain data values as part of library module (similar to what root library allows with --data-values-file)
• there is a unblocking workaround for now which involved removing "removed" data values via overlays before giving it to library module

ref: #418

[cari-lynn]

To add context as to why this is not a bug and is working as intended:

Data value overlay annotations are only used for merging data values files in the current library. The annotations are used to merge all the data values files in the current library into one structure using the overlay annotations to create the data.values struct. When passing the data.values into a library via some_lib.with_data_values(some_values()), it is at that point a structure that is a result of the overlay operations and does not have any annotations.

A library's data value file is responsible for stating which values the library accepts. Any given values not specified there would be considered as errors when trying to evaluate the library.