ytt should support providing arguments via a file

[StevenLocke]

This would benefit ytt users who want to template a variable list of files with a single repeated command.

ie. One could provide a file which indicates the -f arguments so that in a pipeline (such as argoCD) you could always provide the same command (maybe ytt --arg-file file.yaml) and the contents of the file could be updated to reflect the desired templating.

Further discussion could be useful to determine

1. Is this valuable for users?
2. The name of the flag

[joaopapereira]

Is the idea of what we are proposing here the creation of a flag that would pick up a file like the following:

file:
- file1.yml
- file2.yml

and then we would be able to run ytt --arg-file args.yml. The expected result would be the same as running ytt -f file1.yml -f file2.yml?

If so is a YAML file the best option here or should it be a straight bash file with the arguments that would get processed by cobra?

[dherbrich]

yes this is the idea. it should also be possible to add more or less all the other command line flags. not sure if one is better than the other.

[cppforlife]

last time this idea came up, i couldnt justify need for yaml flags config file vs just using basic script. example above is just:

#!/bin/sh
ytt -f file1.yml -f file2.yml

it sounds like one use case is when it's not possible to integrate script with existing tool. in that case i do wonder what distinction would be between ytt --args-file x.yml vs establishing a convention (like we recommend in other places) to have a config directory. this would mean that invocation would always be ytt -f config/.

[caphrim007]

@cppforlife @StevenLocke I can provide my use case fwiw.

tl;dr. I would +1 adding this.

The way we organize our yaml files is as a base directory and a set of N overlays in an overlays directory.

ex.

.
|- base
|  |- ... deeply nested directory structure of yaml
|- overlays
|  |- overlay1
|  |- overlay2
|  |- overlay3
|- envs
  |- dev
  |- prd

for any specific environment that we deploy to, it will receive the base and then it can choose to receive N of the overlays. For example, dev might choose (making up these overlay names)

-f base -f overlays/not-very-many-controls -f overlays/looser-resource-limits

where-as prod might choose

-f base -f overlays/strict-controls -f overlays/hpa overlays/canaried-deployment

It would would be helpful to limit the amount of "stuff" we need to do within our CICD system, by just invoking ytt pointing to the environment that was changed. For example,

ytt -f envs/dev/

But to do that, I would need a declarative way to codify what stuff I want ytt to read. for example,

-f base/ -f overlays/not-very-many-controls/ -f overlays/looser-resource-limits/ -f envs/dev/

But I would need to, in my CICD system, now know which overlays to apply to which environments. The people that work on our product can barely manage their understanding of YAML, let-alone grok'ing a Jenkinsfile.

I imagine I could persuade them to understand a proposed thing below

.ytt.yaml

file:
  - base
  - overlays/not-very-many-controls/
  - overlays/looser-resource-limit/
  - envs/dev/
file_mark:
  - **/*.enc.*:exclude=true
  - **/*.ytt.yaml:exclude=true

The above would be the args used to assemble the specific env.

1. We could tailor these args per env
2. We could keep from complicating the CICD pipelines (as they would just invoke the config file for what they found; ytt -f envs/dev/.ytt.yaml
3. (with all due respect) I wouldn't have to try to sell the following example to a squadron of finicky developers --- #@ template.replace(library.get("app1").with_data_values(custom_vals()).eval())
4. I can avoid the nuances of the multiple shells used by my devs as well as the ones used, or not used, by my CICD system.
5. We can avoid nuance with relative paths in shells. ex, ytt could choose to base its "root" from where the specified args file lives. This would imply only a single arg file be supported.

I see that ytt uses the cobra library, and I'm familiar with cobra's sister library viper insofar as viper can read from a config file provided and populate the available args. https://github.com/spf13/viper#reading-config-files . However, I don't know if the config file itself is something that is fixed or if it can be specified. Any-hoo, may be a thing.

Since I also highly value @cppforlife's thoughts and opinions on the topic, I'm going to go put together his suggested solution using scripts. Thanks for reading and soliciting use cases.

[caphrim007]

thought i would follow-up with the solution that we chose to go with in case others might be interested.

We opted for using the carvel vendir tool

https://carvel.dev/vendir/

We organize our YAML in a series of sub-directories which we affectionately call modules. for example,

.
├── ambassador-mappings
│   └── mappings.yaml
├── base
│   ├── defaults.yaml
│   ├── deployment.yaml
│   ├── namespace.yaml
│   ├── service-account.yaml
│   └── service.yaml
├── cloud-resources
│   └── deployment.yaml
├── flagger-canary
│   ├── canary.yaml
│   └── service.yaml
├── horizontal-pod-autoscaler
│   └── horizontal-pod-autoscaler.yaml
├── irsa-service-account-annotation
│   └── service-account.yaml
├── kapp-config
│   └── configmap.yaml
└── legacy-annotations
    └── main.yaml

We consider this our code.

Then we have an environment specific directory which contains our config. For example,

.
├── secrets.enc.yaml
├── values.yaml
├── vendir.lock.yml
└── vendir.yml

Note our use of the vendir.yml file.

The content of the vendir file lists all of the k8s modules as well as the version which we want to pull. I won't bore you with the full file because it can be mighty long, but here's a snippet from it.

---
apiVersion: vendir.k14s.io/v1alpha1
kind: Config
directories:
- path: .vendir
  contents:
    - path: base
      git:
        url: git@github.com:org/some-service.git
        ref: "4b6ea0"
      includePaths:
        - k8s/modules/base/*
      newRootPath: k8s/modules/base

We have separate paths for each k8s module even though we pull repeatedly from the same repo. We do this because the modules can rev their code at different rates. This also provides us the flexibility to carve out a module and open source it if necessary. In this case we would just change the git url to the new location; wouldn't require code changes.

Our deployment process is then 2 steps.

1. change to the env directory in question and run vendir sync
2. run ytt/kapp on the env directory ytt -f k8s/envs/dev-00

We've found this works well for us.