Create a tool for merging YAML files *with* comments

[jandubois]

Is your feature request related to a problem? Please describe.

We want to be able to have separate values.yaml files for each component in the mixins/ directory. Unlike the config/*.yaml files, the values.yaml files cannot be mixed in a single target directory, where they all use separate names; we need a single final values.yaml file for the helm chart. Obviously comments in both the main values.yaml file and in the ones to be merged in should be retained.

Another use case is updating YAML files via CI, like bumping the version of the Eirini image in mixins/eirini/config/eirini.yaml #841

Describe the solution you'd like

I would like to see a tool in kubecf-tools that merges multiple YAML files, e.g.

yaml-merge chart/values.yaml mixins/bits/values.yaml mixins/eirini/values.yaml > output/helm/values.yaml

There may be existing code in create_sample_value.rb that can be reused for this project.

To have predictable merging semantics I think it is sensible to expect all input YAML files to be well-formatted, and also have all keys in sorted sequence. Additional constraints may become obvious once implementation starts.

Each comment block belongs to the key below it.

Instead of writing a spec I'll just try to give an example with file1.yaml:

# Some comment
aaa:
  a1:
  # here comes a3
  a3:
# Nothing to see
ccc: foo

and file2.yaml:

# Another comment
aaa:
  # this is a2
  a2:
bbb: "Better Business Bureau"

Merging:

$ merge-yaml file1.yaml file2.yaml
# Some comment
# Another comment
aaa:
  a1:
  # this is a2
  a2:
  # here comes a3
  a3:
bbb: "Better Business Bureau"
# Nothing to see
ccc: foo

Running merge-yaml on a single file will effectively lint the file (indentation is proper, keys are sorted). If it doesn't throw an error, it will just copy the input file to STDOUT.

Optional/special rules:

• A comment block at the top of the file, until the first empty line, is a file level comment and does not belong to any key (could be implemented with a special empty key that will sort to the top, but the key is suppressed on output.

• Comments starting with ## (or some other sequence) are internal and will be omitted from the output file.

Describe alternatives you've considered

I've tried to use ruamel.yaml to merge YAML files, but the library is buggy. Sometimes the indentation was messed up, sometimes comments appeared in the wrong places, and I think once even the data itself was mangled.

[mook-as]

Because this is so finicky, I'd like to add to the acceptance criteria:

• There must be a rather extensive test suite, to clearly indicate what the rules we picked are.

[viccuad]

I am looking also forward to this tool, to merge values.yaml that would be used for testing harnesses.

Example: merge-yaml kubecf-diego-values.yaml kubecf-ingress-values.yaml kubecf-develop-values.yaml

Where:

• kubecf-diego-values.yaml example production values for a diego deployment
• kubecf-ingress-values.yaml example production values for a kubecf deployment with ingress
• kubecf-develop-values.yaml develop values, enabling testsuites and configuring them (cats, gingko nodes, etc), changing docker_registry, etc.

Also, I'm looking forward to this tool, to merge values.yaml for k8s providers in catapult. Example: merge-yaml default-catapult-values.yaml your-catapult-caasp-ecp-values.yaml (this usecase is simpler as it could be done without comments)