kustomize vars - enhance or replace?

monopole commented 4 years ago

Kustomize has a vars feature that shipped early in the project. The project's view of its purpose and direction has focussed since then, and the feature now appears as a outlier in the kustomize feature set for reasons discussed below.

The vars feature is also a common topic in issues being filed, not necessarily because of actual bugs but because vars appear to be template variables, but don't function as such.

This issue is a parent issue to gather var related issues, identify common problems and consider solutions to these problems that don't require vars.

What’s a var?

A var in kustomize is a reflection mechanism, allowing a value defined in one YAML configuration field (e.g. an IP address) to be copied to other locations in the YAML. It has a source spec and any number of targets.

source spec: a field in a kustomization.yaml file associating an uppercase var name like VAR with a specific field in a specific resource instance, e.g. the image name in the Deployment named production. This field is the source of the var’s value.
targets: instances of the string $VAR in resource instance fields, identifying where to put the var’s value. The placement of $VAR is constrained to a particular set of fields in a particular set of resources.

It’s a don’t-repeat-yourself feature. The overall effect is similar to the reflection provided by YAML anchors, except that kustomize manages it up the overlay stack.

Isn't this templating?

A kustomize var smells like a template variable, but stops short of full templating.

Full templating, with a distinct key:value (KV) file, has drawbacks.

A template isn't YAML; it must be rendered to make YAML.

Kustomize allows a $VAR only in a limited set of string fields like container command arguments (similar to the contraints of the downward API), so the source material remains usable with generic YAML tools.
KV files become a template-driven API wrapping the real API.

This is fine in simple cases, but an API emerging from templates scales poorly to real world production setups - large environments, disparate configuration owners, etc. Kustomize vars avoid the distinct KV file by requiring reflection.
The ugliness arising from shared templates - everything gets parameterized.
The difficulty of rebasing templates to capture upstream changes into your template fork.

These drawbacks (and others) are discussed in more detail in Brian Grant's Declarative application management in Kubernetes.

Kustomize vars directly avoid the first two, and help avoid other problems by simply not being the core means to generate and customize configuration.

Kustomize vars, however, share one glaring flaw with template variables. Their use makes the raw configuration data unusable in an apply operation - the config data must be passed through kustomize first before being applied.

This violates an explicit goal of kustomize; provide a means to manipulate configuration data without making the raw configuration unusable by kubernetes. kustomize vars would not now be accepted as a new feature in their current form.

Issue Survey

The following attempts to capture and categorize issues related to kustomize vars.

I want to put `$VAR` in some (currently disallowed) field

kustomize doesn't allow unstructured $VAR placement. A $VAR must go into a field, and only into a particular set of fields.

Kustomize vars are handled by the var transformer. Like all other builtin transformers, it has a builtin configuration, in its case defined in the file varreference.go. This file defines where $VAR can be placed in configuration yaml.

One can use the Configurations field in any kustomization file (see this test, this other test and these examples) to specify a file containing a custom set of field specs in the same format as varreference.go. This allows a user to add a $VAR to more string fields without changing kustomize code.

The var transformer is a singleton with global scope

Kustomize plugin transformers have no effect beyond the kustomization directory tree in whose root they are declared. Further, in one kustomization directory one may use many different instances of the same transformer - e.g. one can declare multiple label transformers that add different labels to different objects.

The var transformer, however, is special - it's a singleton with global scope. When var transformer configuration data is declared in a kustomization file, it's not immediately used, and is instead merged with any var transformer configuration data already read. The last step of a kustomize build is to take all accumulated var configuration, build a singleton var transformer, and run it over all objects in view.

The upshot is that one can put var definitions and var transformer configuration anywhere (in any kustomization.yaml file reachable from the kustomization file targetted by a kustomize build command) and get the same global effect.

This global behavior came from early requests to have var definitions propagate up, so that overlays could usefully decare $VARs in their patches.

Some issues

1782 Variable substitution only partly working on Jobs
1734 well-defined vars that were never replaced
1713 Kustomize vars not applied to Namespace resources
1592 Can't use kustomize vars in images
1585 var replacement doesn't occur on commonLabels
1553 Using overlay-provided secrets in a base
1540 Var reference does not work in volumes

I want to define vars that aren’t strings

It so happens that the current implementation of vars only allows strings, because only string fields (container command line args and mount paths) are allowed by default, and the yaml libraries in use make a distinction between fields of string, number, map, etc.

Relaxing this is straightforward (a bit more code, tests, and error handling) but pointless if vars are deprecated.

Some issues

1721 Variable is expected to be string

I want to use diamonds

Suppose one has a top level overlay, called all, that merges sibling levels dev, staging, prod (variants) by specifying them as resources - i.e. the file all/kustomization.yaml contains:

resources:
- ../prod
- ../staging
- ../dev

Suppose in turn that these variants modify a common base. A var defined in that common base will be defined three times at the all level, a currently disallowed behavior.

This is analogous to the compiler rejecting a construct like

type Bar struct {
  someVar int
}
type Foo struct {
  Bar
  Bar
}

@tkellen proposes a fix for this, #1620, allowing repeatedly defined var names as long as the source of the underlying value can be shown to be the same.

The solution works for particular use cases, but leaves us with a global vars that will sometimes work and sometimes not work. When they don't work - when the values don't match - what should the error encourage the user to do? Rename/re-arrange vars? Or do something else entirely?

If vars are retained, and get more complex scopes, then there should be a clear design associated with them. We don't actually need that design to know that it would be a firm step into configuration language territory, a non-goal of kustomize.

Meta question: why make a diamond structure?

Possibly to do one-shot apply operations. With the above kustomization file, one can deploy all environments with one command:

result=$(kustomize build all | kubectl apply -f -)

The price of this convenience is a completely useless result from apply, analogous to conflating the purchase of real estate, a car and a sandwich to one financial transaction, and accepting that it must all be undone because the sandwich had onions you didn't order.

A better way to do this as one command is make a scripted loop which can analyze the apply result for each environment:

for env in 'prod staging dev'; do
   result=$(kustomize build $env | kubectl apply -f -)
   handle(result)
done

Some issues

1600 don't cause variable conflicts when values are the

same
1248 Error when composing several identical bases that use the same var: "var ... already encountered

I just want a simple KV file

Here a user wants to get the value for the vars from some external KV file, rather than reflexively from some other part of the config.

One reason not to do this is that there's already a mechanism to convert KV files and other data representations into configmaps, and vars already know how to source configmaps, so let's try that first. The purpose of configmaps is to hold miscellaneous config data intended for container shell variables and command line arguments; they're aligned with the intent of kustomize vars. Having the configmaps in the cluster has other advantages, and no downside outside a questionable concern that configmaps will forever accumulate in storage.

Another reason not to do this is it would place kustomize firmly in the template business, which is a non-goal of kustomize. Kustomize users that also really want to use some templating can do so using some other tool - sed, jinja, erb, envsubst, kafka, helm, ksonnet, etc. - as the first stage in configuration rendering.

It's possible to use a kustomize generator plugin to drive such a tool (e.g. sed example, helm example) in this first stage, hopefully as a first step in coverting away from template use entirely.

Some issues

318 Enhancement: Literal vars
1737 Support for variable replacement in literals

Summary

Kustomize vars feel incomplete from from a template-user point of view. The existence of this feature inspires an urge to generalize vars to provide full templating and/or configuration language-like variable scoping.

Alternatives to vars

The kustomize approach is to eschew the temptation to generalize to yet another config language or templating tool, and solve specific kuberenetes configuration problems with a dedicated transformer or generator that, itself, is easy to configure and use.

Vars have been a sort of stopgap, used when one cannot frame the problem as solvable by an existing transformer or generator plugin.

What's a plugin?

Kustomize is just a finder and runner of transformer and generator plugins.

Transformers

Two common uses of template variables in kubernetes is to set the image name in a container spec, and set labels on objects.

In kustomize, these particular tasks are done with transformer plugins - respectively, the image transformer and the [label transformer].

A transformer plugin is a chunk of code written in any language that

is configurable by a kubernetes style YAML file,
can be found and run by kustomize when the path to its config file appears in the transformers: field of a kustomization file,
when run, accepts, modifies and emits YAML.

Transformers understand the structure of what they modify; they don't need template variable placeholders to do their job.

If a use case feels like adding a field or changing the value of a field, the kustomize way to do it is write a transformer plugin to perform the field addition or mutation

Generators

Since vars let one copy data from one field to another, it might be best to simply make the object or set of objects with these fields set to the same value from the outset.

In kustomize, the way to make objects (instead of reading them from storage) is to use a generator plugin. It's a chunk of code written in any language that

is configurable by a kubernetes-style YAML config file,
can be found and run by kustomize when the path to its config file appears in the generators: field in a kustomization file,
when run, emits kubernetes-style YAML.

A use case for generators is kubernetes Secret generation. There's a builtin secret generator plugin, and the documentation has an example of a plugin that produces Secrets from a database.

A plugin's config file holds values that would otherwise appear as part of a KV file. In the template world, it's possible to define keys as JSON paths (YAML being a superset of JSON) so that the KV file looks like a YAML object. But there's no object here in the traditional sense. The template user must rely on the branching and looping constructs of the template to do anything other than replace keys with these values.

A kustomize generator should have documentation and unit tests - it's a tangible, testable factory.

custom resources

While mentioning that KV files can be viewed as objects, it's appropriate to mention custom resources.

Given a set of related, live kubernetes objects - how can they be instantiated as one meaningful, monitorable thing with a life of its own? Custom resources are the answer to this question, and the general answer to extending the kubernetes API with more objects.

Kubebuilder is a tool to help write the controller that accompanies a custom resource definition in the kubernetes control plane. Like any other kubernetes API YAML, custom resource configuration can be generated or transformed by kustomize.

Specific alternatives to vars

Replacement Transformer

This is a general purpose var replacement, coded up as an example transformer in #1631.

The idea here is simple: eliminate embedding $VAR as the means of defining the target, replacing target specification with kind/field addresses in the kustomization file (as is already done for sources).

This keeps the raw material usable - no sprinkling of dollar variables throughout the config.

There’s a loss of functionality though. In the existing vars implementation, the $VAR can be embedded in a container command line string. I.e., one can replace part of a string field.

The atomic unit of replacement of this transformer is the whole field. That said, if your command line is a list of arguments (instead of one blank-delimitted string), the replacements transformer should be able to target list entries by index.

Path Transformer plugin

This doesn't exist yet, but someone could write it.

One use of vars is to serve as a placeholder in a string field that contains a file system or URL path, e.g. /top/config/$ENV/$SUBDIR/data. The var feature would replace $ENV and $SUBDIR with some values to resolve to a real path.

There's no need for a var here. Kustomize has a [prefix/suffix transformer] is dedicated to name field transformation. It could be copied and adapted to editting fields known to contain paths - prepending, postfixing or otherwise changing path declarations.

tkellen commented 4 years ago

Thank you for writing this up @monopole! My early vote is rip them out ASAP. I'll share a much more detailed justification with examples soon.

monopole commented 4 years ago

Thanks @tkellen , i'm hoping people will own some new transformers to allow that :)

tkellen commented 4 years ago

I'm just going to start sharing stuff incrementally or I'll never say anything.

Before we even talk about vars, I think we need to agree or disagree on the veracity of this point:

Kustomization files should be black boxes where we can do whatever we want. What comes out the other end must be raw yaml with no context about what happened inside.

If you agree, vars as implemented need to go away. Also, unless I am mistaken, agreeing also cleanly allows the "one shot" or "diamond" approach you've outlined as being a bad practice.

Here is a kustomization layout I have been using that supports every use case I have thrown at it for nearly a year (it relies on vars-type functionality in @jbrette's fork of kustomize but I believe it can work without it). As far as I can tell, if the features I'll write about later were added to kustomize we would have the base primitives needed to say k8s configuration management is a solved problem (speaking purely technically, this doesn't cover the sticky issues of trying to have humans collaborate together effectively that @pwittrock is trying to solve with https://github.com/kubernetes-sigs/kustomize/pull/1962 type things).



                                                                                                                                                                                                   one-shot dev                             +---------+  
                                                                                                                                                                                                  +-------------------+                     | dev     |  
     ENVIRONMENT: dev                                                                                                                              -----------------------------------------------| kustomization.yml |---------------------| k8s     |  
                                                                                                                                                  /contextless yaml                             / +-------------------+ contextless yaml    | cluster |  
          region.txt -----\                                                              BASE DEPLOY: dev                                        /                                             /                                            +---------+  
        env_name.txt ------\ +-------------------+                                       +-------------------+                                  /                                             /                                                          
        hostname.txt --------| kustomization.yml |---------------------------------------| kustomization.yml |---------------------------------/------------                                 /                                                           
        rds_host.txt ------/ +-------------------+ contextless yaml                    / +-------------------+\contextless yaml               /             \                               /                                                            
    project_name.txt -----/      configmap                                            /                        \                             /               \                             /                                                             
                                 generator                                           /                          \                           /                 \                           /                                                              
                                                                                    /                            \   service foo in dev    /                   \   service bar in dev    /                                                               
                                                                                   /                              \ +-------------------+ /                     \ +-------------------+ /                                                                
                STANDARD DEPLOY                                                   /                                -| kustomization.yml |/                       -| kustomization.yml |/                                                                 
                                                                                 /              APP: service-foo  / +-------------------+     APP: service-bar  / +-------------------+                                                                  
             serviceaccount.yml -----\                                          /               .                /                            .                /                                                                                         
                       role.yml ------\ +-------------------+                  /                ├── Makefile    /                             ├── Makefile    /                                                                                          
                rolebinding.yml --------| kustomization.yml |------------------                 ├── manifest.yml                              ├── manifest.yml                                                                                           
                 deployment.yml ------/ +-------------------+ contextless yaml \                └── src         \                             └── src         \                                                                                          
                    service.yml -----/                                          \                   └── main.go  \   service foo in prod          └── main.go  \   service bar in prod                                                                   
                                                                                 \                                \ +-------------------+                       \ +-------------------+                                                                  
                                                                                  \                                -| kustomization.yml |                        -| kustomization.yml |\                                                                 
                                                                                   \                              / +-------------------+\                      / +-------------------+ \                                                                
    ENVIRONMENT: prod                                                               \                            /                        \                    /                         \                                                               
                                                                                     \                          /                          \                  /                           \                                                              
          region.txt -----\                                                           \  BASE DEPLOY: prod     /                            \                /                             \                                                             
        env_name.txt ------\ +-------------------+                                     \ +-------------------+/                              \              /                               \                                                            
        hostname.txt --------| kustomization.yml |---------------------------------------| kustomization.yml |--------------------------------\-------------                                 \                                                           
        rds_host.txt ------/ +-------------------+ contextless yaml                      +-------------------+ contextless yaml                \                                              \                                                          
    project_name.txt -----/      configmap                                                                                                      \                                              \   one-shot prod                            +---------+  
                                 generator                                                                                                       \                                              \ +-------------------+                     | prod    |  
                                                                                                                                                  ------------------------------------------------| kustomization.yml |---------------------| k8s     |  
                                                                                                                                                   contextless yaml                               +-------------------+ contextless yaml    | cluster |  
                                                                                                                                                                                                                                            +---------+                                                                                                                                                                                                                                              ```

I am also mocking up a functional repository that shows this pattern here:
https://github.com/scaleoutllc/monorepo-template

...more as this evolves.

tkellen commented 4 years ago

Thanks @tkellen , i'm hoping people will own some new transformers to allow that :)

I understand and respect that hope. The lack of engagement between the maintainers of this project and would-be-contributors that I have watched over the last two years all but ensures I will not be contributing directly to this project unless I am made a maintainer or something drastic changes. At the moment I am hopeful the ideas I share will be powerful enough to evoke action.

monopole commented 4 years ago

I brought up diamonds above because their use surfaced problems with the global vars model. I noted that using a diamond solely as convenience to push to multiple environments as a one-shot unnecessarily complicates failures and retries - that's just a generality, i probably shouldn't have brought it up.

Diamonds without vars are allowed, and work:

monopole commented 4 years ago

@tkellen As for maintainers not being engaged enough, I agree. Sounds like you're on board with killing vars. Assuming someone else isn't passionate in the other direction, we need a deprecation plan - dates, new transformers, and doc about to convert use cases. Do you want to own that?

monopole commented 4 years ago

Sorry - doing other things. W/r to the black box comment:

kustomization files should be black boxes where we can do whatever we want. What comes out the other end must be raw yaml with no context about what happened inside.

There are two exceptions breaking this model by relying on global data

the var transformer (discussed ad nauseam above - no need to cover here)
name modifiers - currently the prefixsuffixtransformer and the hashtransformer.

Name modifiers break name references (e.g. a deployment referring to a configmap).

These references are currently repaired globally, not on a per directory basis. This allows a user to specify a patch to a resource in an overlay without having to know that the resource's name had been changed in a lower level kustomization.

If one allows this convenience to the user, then knowledge of the resource's current name (possibly changing as one goes up the stack) and original name must be retained to fix references.

Currently the original name is retained in global data, for a final pass to fix references. This could be changed - one could fix references at each level, so each level emits "correct" yaml. But the original names must be retained going up. If not in related kustomize data structures, they could be slapped into the yaml itself, either as a special comment on the name field, or as a newly created secondary name field - e.g. originalName (you'd have to be sure it didn't collide with an existing field). The name reference fixer would need to know where to find these original names.

All other builtin transformers are closed with respect to the kustomization directory root - i.e. the box is black for them.

le-duane commented 4 years ago

Could someone suggest how to accumulate variable values down the overlay tree and apply at the last or a particular overlay using plugins?

Seems like a lot of people have differing opinions about what Kustomize should do, but for me it's about reducing duplication via overlays. Not having some kind of cohesive way to use variables would mean separating things into separate kustomizations with hard-coded values.

monopole commented 4 years ago

@le-duane can you be really, really specific about the use case?

le-duane commented 4 years ago

Thanks for your interest @monopole

I'm dealing with stateful apps, so maybe different than what most are used to.

Still a WIP, but basic overlay tree looks something like this:

           base
             |
            env
             |
        kube-cluster
             |
         db-cluster
             |
      db-cluster-nodes

Example with 2 separate sandbox DB clusters:

          cool-db
             |
          sandbox
             |
         us-west-1
          /     \
       sxb1     sbx2
        |        |
     rand-az    az-a

Example with 1 prod DB cluster spread across regions/kube clusters:

          cool-db
             |
         production
           /     \ 
     us-east-1 us-east-2
         |         |
       prd1      prd1
         |       /  \
       az-a   az-a az-c

Two things I use Vars for.

There are services and endpoints which must be named exactly the same as the db cluster-name. The cluster-name can be changed at layer 4 to accommodate e.g. multiple sandbox clusters for testing. This is not compatible with suffixing, because there is nothing to suffix. Therefore, they must be named with a variable that points the cluster-name label.

apiVersion: v1
kind: Service
metadata:
  name: $(CLUSTER_NAME)

I want to be able to change labels for AZ/Cluster-Name/Region in an overlay and have that propagate to all of the related object properties.

e.g. For adding db nodes in availability zone B, the only thing that changes in the overlay is:

apiVersion: builtin
kind: LabelTransformer
metadata:
  name: "zoneLabeler"
labels:
  az: "b"
fieldSpecs:
# StatefulSet
- path: metadata/labels
  create: true
  kind: StatefulSet
- path: spec/selector/matchLabels
  create: true
  kind: StatefulSet
- path: spec/template/metadata/labels
  create: true
  kind: StatefulSet

and the rest is configured from a variable pointing to that label e.g:

apiVersion: builtin
kind: PrefixSuffixTransformer
metadata:
  name: "instance-namer"
suffix: -$(AZ)
fieldSpecs:
- path: metadata/name
  kind: StatefulSet
- path: metadata/name
  kind: ConfigMap
- path: metadata/name
  kind: Secret

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: failure-domain.beta.kubernetes.io/zone
                operator: In
                values:
                - $(REGION)$(AZ)

# not important to this discussion, but storage is tied to an AZ to help the cluster-autoscaler make decisions when we want replicas tied to an AZ
volumeClaimTemplates:
- metadata:
    name: cool-storage
spec:
    accessModes:
    - ReadWriteOnce
    resources:
    requests:
        storage: 100Gi
    storageClassName: standard-$(REGION)$(AZ)

These object definitions above are all in different overlays.

It is important for us to avoid having to re-template or manually modify the same thing in multiple places any time we want to add a new database cluster or set of nodes for an existing cluster.

I think I could use a transformer plugin to template at the last mile, but that requires duplicating Vars in every db-node overlay, which seems clunky and error prone because one has to pick which vars need to change between overlays and leave the rest unchanged. If I were to template at each overlay, so as to avoid duplicating top level vars, then I couldn't template object names because there would be a mismatch between the names at different layers. Also I couldn't override variables further down the chain.

I 100% agree other templating engines are always going to be better at that piece, but they have no knowledge of the kustomization hierarchy, which makes using them difficult from my point of view.

Finally, since diamonds were mentioned, I tried tying all resources for a kubernetes cluster together in a separate kustomization, but it blew up on the duplicate services/endpoints I mentioned earlier. If somehow duplicates could be squashed, that would be really helpful from a user and CI/CD standpoint.

edit: Forgot to add, if it weren't for the issues with diamonds, I think I could lose a layer of duplication.

           base                     base
             |                       |    
            env                  kube-cluster
             |                      /
         db-cluster                /
             |                    /
      db-cluster-nodes  <--------|

tkellen commented 4 years ago

@monopole can you help me understand what you mean precisely by:

Name modifiers break name references (e.g. a deployment referring to a configmap).

If a kustomization changes the name of something I think it is a fair expectation of the consumer of the yaml stream it produces to know what the "new" name is and use it. It is very confusing to me that I could run kustomize build in a folder, observe the output, and then in some other kustomization consume what I expect to be that output and then match on something that wasn't shown (to apply a patch, for example).

Perhaps I am unintentionally relying on this critical behavior in some fashion but based on my limited understanding of the implementation it seems to me that this, like vars, should be entirely removed, or, at the very least, the "hidden" context should appear in the output yaml as you describe.

tkellen commented 4 years ago

@le-duane I try to use kustomize in the same manner you do, to eliminate duplication through the use of overlays. As for how to propagate values through kustomize without vars, see the very first stage in my pipeline where I generate a configmap from flat files on disk? That configmap is present in every consumer and those values should be picked up and placed in other fields through replacements (https://github.com/kubernetes-sigs/kustomize/pull/1594). The primary missing functionality with replacements is the ability to interpolate multiple values and literal values.

tkellen commented 4 years ago

@le-duane, the removal of vars would mean you could no longer put $(whatever) in your source yaml. This is the basic antipattern the entire kustomize project is founded on resolving.

Here is how to accomplish one of the workflows you described without vars (this assumes you are using a layout like mine where you bring "variables" in as a configmap and pass it around wherever you need it):

This:

spec:
  template:
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: failure-domain.beta.kubernetes.io/zone
                operator: In
                values:
                - $(REGION)$(AZ)

...becomes this:

replacements:
  - from:
      value: 
        - configmap:environment.data.region
        - configmap:environment.data.az
      join: ""
    to:
      target:
        kind: Deployment
      fieldrefs:
        - spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[0].matchExpressions.values[0]

Importantly, the replacement syntax I've shown above is not yet implemented. It allows multiple values to be sourced with the assumption you're going to join them into a single string on a delimiter, in your case, no delimiter.

tkellen commented 4 years ago

@monopole Looks like I missed your edit of this during the initial posting:

@tkellen As for maintainers not being engaged enough, I agree. Sounds like you're on board with killing vars. Assuming someone else isn't passionate in the other direction, we need a deprecation plan - dates, new transformers, and doc about to convert use cases. Do you want to own that?

I'm interested in owning this, yes, though I cannot do it in my spare time. I will be available for contract work in the near term. Is there a more direct avenue to engage in a conversation about that?

bkuschel commented 4 years ago

Please don't remove vars until such time the replacement transformer is able to do partial string substitution effectively without imposing base resource requirements or requiring repetitive replacements.

Please see prefix example here: https://github.com/kubernetes-sigs/kustomize/pull/1738

Ripping it out would break us completely, my suggestions is to create a transformer out of it instead.

tkellen commented 4 years ago

Depending on the size of your team I would highly recommend switching to an envsubst or similar pattern ASAP. It seems clear that vars in their current form are not meant to be used in the manner you are using them and no matter what there will be breakages down the line (unless the codebase is drastically altered to respect configuration file versions).

For more context, I'm in the situation of relying on a fork to do more or less what you've done and I've taught an entire team of people how to use a feature that never landed, and, to some degree, to grok semantics that make no sense. Unwinding this is a nontrivial task.

bkuschel commented 4 years ago

I think we are using what it is intended for, at least according to @pwittrock 's description during the brainstorm session.
The general problem is that resources are more than k8s schemas, they contain strings, these string contain references to attributes in other kubernetes resources that change during the course of manifest generation and need to be replaced. Patches are not a good solution for replacing a substring in a random string coming from a base as the patch would need the entire string or arrays, not to mention the problems with SMPs with CRDs and arrays (like args). Prefix and namespaces would require an incredible amount of patches for 30+ micrososervices. With VARS (or a good replacement alternative) this is a no-brainer.

envsubst is something that would need to be done after a manifest generation so it's possible but problematic as kustomize is not generating a working manifest. Seems like an anti-pattern.

tkellen commented 4 years ago

Thanks for the clarification @bkuschel. I believe replacements can solve the issue you've raised RE: replacing substrings in a random string, but the configuration format and implementation most assuredly doesn't exist for it yet.

bkuschel commented 4 years ago

Right, this is why i'm against ripping out VARS at this time.

ricochet commented 4 years ago

While I agree that removing VARS is the right call longterm, I also want to echo @bkuschel in that removing VARS without a clear onboarding alternative (available at HEAD) is detrimental to consumers. I recommend starting with a deprecation warning and warnings throughout the doc to signal that a drop is incoming.

Converting a real OSS consumer like kubeflow will give the community far more legs than suggestions in issues. I reference kubeflow's KustomizeBestPractices for my own work.

Something else to consider is to add a similar implementation in the kubectl doc's app composition and deployment guide

                      +---------------+
                      |Deployment Base|
                      +---^--------^--+
                          |        |
                +---------+-+    +-+---------+
                |Golang Base|    |Spring Base|
                +^-------^--+    +---------^-+
                 |       |                 |
                 |       |                 |
                 |       |                 |
+----------------++   +--+--------------+  +-+-----------------+
|       (A)       |   |       (B)       |    |       (C)       |
|Component Variant|   |Component Variant|    |Component Variant|
|        +        |   |        +        |    |        +        |
|     Overlays    |   |     Overlays    |    |     Overlays    |
+-----------------+   +-----------------+    +-----------------+

           X n(variants) == off-the-shelf configuration

And a golang app with parameters of SERVICE_NAME=my-go-app and CONTEXT_PATH='/myApp'

+-----------------+
| Base Deployment |
+---------^-------+
          |
+---------+---------+              +------------+
| Golang Deployment |       +----->|Base Service|
+---------^---------+       |      +------------+
          |                 |
+---------+----------+      |   +-------------------+
| kustomization.yaml +--------->|Base VirtualService|
+--------------------+      |   +-------------------+
                            |
Resources a Deployment,     |      +------------+
Service, VirtualService,    +----->|Base Ingress|
and Ingress                        +------------+

tkellen commented 4 years ago

Thanks for sharing your workflow and thoughts @ricochet! I love the idea of a deprecation message as a resolution to https://github.com/kubernetes-sigs/kustomize/issues/2071 (and mentioned as much over there). I have limited free time to work on this but I do have plans to share a set of configuration formats for a replacements transformer that could accomplish everything vars are presently being used for (at least, every workflow mentioned in this issue).

benjamin-bergia commented 4 years ago

Hi, it's now one year that I have been managing our production and testing environments using exclusively kustomize. I also went through some of the patterns that have been posted, but have been able to refactor most of the cases to avoid variables. Currently I have a single use case where I haven't been able to remove the variables and it is for the host field of my ingresses.

I have a dozen or so ingresses per environment, all of them using a subdomain but sharing the same domain. The only solution I have found have been to have a "global" configmap containing my domain name and a variable referencing this entry in the configmap.

I can then populate all my ingresses using things like host: www.$(BASE_DOMAIN) or host: api.$(BASE_DOMAIN) I also pass this configmap in the environment of my web servers when they need to know their own url.

All this is to be able to:

use different domains on prod and testing without changing the subdomains
adding new ingresses without copy/pasting domains or patches all over the place
keep everything in sync across the whole environment

I also understand that this "global" configmap should be avoided but I have found that, unfortunately, we always end up with a handful of settings, mostly static, that have to be available environment wide and kept in sync. I find that keeping them in a separate configmap is need but unfortunately doesn't play well with kustomize configmap generators.

candlerb commented 4 years ago

Speaking as a newbie, I have one observation: kustomize's $(VAR) feature seems to overlap with kubernetes' own syntax to substitute environment variables in pod commands.

It's a potential point of confusion (for me anyway).

mattmceuen commented 4 years ago

Currently, the replacement transformer exists as a POC/ example plugin. My impression is that the next step is to reform it into a built-in plugin, is that correct? Are there any plans/timeline/critical path can be shared -- is it just a matter of time, or is there more evaluation that needs to happen first (or are we just waiting for someone to pick the work up)? Thanks!

rdubya16 commented 4 years ago

@benjamin-bergia This is the exact same thing we are trying to solve. There still seems to be no way to avoid a small set of variables, the other ones we struggle with is things like the cluster name (required for tagging and alb-ingress), names of AZs cluster is deployed in (needed in CRDs for prometheus), and region. There is really no source of truth for these.

I still feel like kustomize is lacking the tools to solve this use case without resorting to some kind of meta-templating on top of kustomize. We are trying manage multiple clusters with only a small set of differing variables which results in copying a bunch of boilerplate or resorting to meta-templating.

rehevkor5 commented 4 years ago

In the section, "I want to use diamonds":

top level overlay, called all, that merges sibling levels dev, staging, prod (variants) by specifying them as resources

and

With the above kustomization file, one can deploy all environments with one command

assume that the kustomizations being composed are variants and/or environments. However, that does not have to be the case in order to run into issues with different kustomizations declaring a variable with the same name, which breaks it ("var already encountered"). For example, after using kfctl to generate kustomizations, and using Kustomize itself to aggregate those and customize them for different environments, attempting to apply them with Kustomize fails due to the generated kustomizations declaring the same vars. The kustomizations are not environments or variants of the same thing: each one handles a different application or component of the overall system.

To me, as a total noobie to Kustomize (I prefer Helm's declarative style over Kustomize's imperative style), this is an issue of composability. If I have two kustomizations that work totally fine on their own, then I should be able to group both of them together into a 3rd kustomization, and the result should be identical either way. There shouldn't suddenly be leakages/side-effects happening between siblings.

mt-inside commented 4 years ago

A couple of comments here in the hope they help.

I can see the purpose and value of "vars", but maybe the name is just wrong? If they were called "cross-refs", would people stop asking for arbitrary, unstructured substitution?

I came here today googl'ing for arbitrary, unstructured substitution ;) My use-case is pretty simple: patchesStrategicMerge are great for "surgical" changes like "it's prod, increase replica count". But my DNS name / cluster name appears everywhere - most of the Istio CRDs, ConfigMaps, command-line args to a lot of containers, annotations (for logging), etc. Some things need just the zone name, some the FQDN.

Writing an overlay to set this in every place would be very long-winded and need rebasing constantly, plus every environment would need its own (unless I envsubst the patch).
patchesJson6902 would be a bit shorter, and also much easier to auto-generate, but again that's another tool and now I have a whole pipeline.
k/v file -> configMap -> (current) vars would work I guess, but feels nasty.
transformations would be hard to use because I need arbitrary text interpolation on some of those substitutions, eg passing the FQDN as --arbitrary arg names.

I get and love the spirit of Kustomize, and I'm using it to try to undo the mess I made with Helm... I may just have missed a way to do what I want? If not I hope this use case is enlightening.

chrisob commented 4 years ago

@monopole @Liujingfang1 Sorry for being "that guy" and polluting this (very important) issue with noise, but is there any plan of when the replacement transformer will be promoted to a builtin and make it into a new release?

fejta-bot commented 4 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta. /lifecycle stale

Shell32-Natsu commented 3 years ago

/remove-lifecycle stale

yanniszark commented 3 years ago

@Shell32-Natsu @monopole any updates on this? I have noticed that kpt is invested in setters and that the setter code lives in the kustomize repo (kyaml). Is the plan to use setters as a replacement for vars?

bburky commented 3 years ago

Re: I just want a simple KV file (plus environment variables please)

I was looking for a way to substitute Argo CD build environment variables into my Kustomize files and discovered this vars feature. With a scary chain of KV → ConfigMap → vars → Application I created a proof of concept: https://github.com/bburky/kustomize-transformer-environment-variables

This proof of concept feels really hacky, and leaves around an unneeded ConfigMap in the deployed environment. But I do like the general idea.

I would like it if the vars: feature accepted literals: and envs: like configMapGenerators. I think that would make the existing vars feature more useful. Or maybe this turns Kustomize into too much of a templating language.

hermanbanken commented 3 years ago

I want to add that we are now also using vars. Initially we didn't understand how to use it for our purpose, but it is a 100% fit. One example is our Ingress resource, which looks like this:

Example was moved to https://gist.github.com/hermanbanken/3d0f232ffd86236c9f1f198c9452aad9. Please comment there.

```yaml # file: base/ingress.yaml apiVersion: networking.k8s.io/v1beta1 kind: Ingress metadata: name: services annotations: kubernetes.io/ingress.global-static-ip-name: $(SERVICES_GLOBAL_STATIC_IP_NAME) kubernetes.io/ingress.allow-http: "false" ingress.gcp.kubernetes.io/pre-shared-cert: $(SERVICES_PRE_SHARED_CERT) kubernetes.io/ingress.class: "gce" spec: rules: - host: $(HOST_A) http: paths: - backend: serviceName: serviceA servicePort: 80 - host: $(HOST_B) http: paths: - backend: serviceName: serviceB servicePort: 80 - host: $(HOST_C) http: paths: - backend: serviceName: serviceC servicePort: 80 ``` Then our configMapGenerator / vars looks like this: ```yaml # file: base/kustomization.yaml bases: - ingress.yaml configMapGenerator: - name: ops-ingress-properties envs: [environment.properties] vars: - name: SERVICES_GLOBAL_STATIC_IP_NAME objref: { kind: ConfigMap, name: ops-ingress-properties, apiVersion: v1 } fieldref: { fieldpath: data.SERVICES_GLOBAL_STATIC_IP_NAME } - name: SERVICES_PRE_SHARED_CERT objref: { kind: ConfigMap, name: ops-ingress-properties, apiVersion: v1 } fieldref: { fieldpath: data.SERVICES_PRE_SHARED_CERT } - name: HOST_A objref: { kind: ConfigMap, name: ops-ingress-properties, apiVersion: v1 } fieldref: { fieldpath: data.HOST_A } - name: HOST_B objref: { kind: ConfigMap, name: ops-ingress-properties, apiVersion: v1 } fieldref: { fieldpath: data.HOST_B } - name: HOST_C objref: { kind: ConfigMap, name: ops-ingress-properties, apiVersion: v1 } fieldref: { fieldpath: data.HOST_C } ``` and the properties like this: ```properties # file: base/environment.properties # Ingress annotations SERVICES_GLOBAL_STATIC_IP_NAME=services SERVICES_PRE_SHARED_CERT=a-yyyymmdd,b-yyyymmdd,c-yyyymmdd # Hosts HOST_A=a.example.org HOST_B=b.example.org HOST_C=c.example.org ``` then in our overlays we redefine the `environment.properties` file and have this in Kustomization: ```yaml # file: overlay/staging/kustomization.yaml configMapGenerator: - name: ops-ingress-properties envs: [environment.properties] behavior: replace # <======= critical ``` which overwrites the values in the base like this: ```properties # file: overlay/staging/environment.properties # Ingress annotations SERVICES_GLOBAL_STATIC_IP_NAME=services-staging SERVICES_PRE_SHARED_CERT=a-staging-yyyymmdd,b-staging-yyyymmdd,c-staging-yyyymmdd # Hosts HOST_A=a-staging.example.org HOST_B=b-staging.example.org HOST_C=c-staging.example.org ```

This works ideal for us! A bit sad that it took us soo long to discover this feature. We really don't want it replaced/removed.

dan-slinky-ckpd commented 3 years ago

I would like to echo @hermanbanken comment. We also use and very much like this pattern.

wstrange commented 3 years ago

Hmm - I find the UX experience on that pattern to be less than optimal. It is verbose, and hard to follow. @dan-slinky-ckpd @hermanbanken - If you had kpt like setters, would that not solve the problem in a more elegant fashion?

hermanbanken commented 3 years ago

I only dislike the repetition in my vars statement. The rest I like because it is immediately clear from the Ingress that field are overridden somewhere.

Note that a JSON patch would also work, but has the downside of selecting host rules by index, which is very unstable. Only 1 host rule has to be prepended for all replacements to fail.

With kept, I'd need to have the possible replaces in my mental model. If someone is reading the Ingress for the first time he/she needs to know about any future kept setters, while vars are immediately clear, even if you don't know the vars-feature.

hermanbanken commented 3 years ago

@Shell32-Natsu I see references on this issue mention that custom Linux exec transformers would be an alternative to this, but I disagree.

The Kubernetes project won't adapt Kustomize with exec plugins enabled by default, right? We should really strive for Kustomize to be integrated in Kubernetes.

The vars feature implementation might be "less than ideal" (slow? not a nice UX?) but it is better than not having it. As said above, kpt is not really an alternative as it lacks the visibility.

Is the core team sure about wanting to remove vars?

srknc commented 3 years ago

Not sure whether it's scope of this conversation but, following @hermanbanken 's example, it does overwrites values at host section but not the ones under annotations with the error below; well-defined vars that were never replaced: SERVICES_GLOBAL_STATIC_IP_NAME,SERVICES_PRE_SHARED_CERT

Darren-wh commented 3 years ago

@hermanbanken I think it's the best way.Can you show the complete code? thanks. You can share your GitHub repository. or can you show your varReference configuration?

Darren-wh commented 3 years ago

@tkellen @ricochet How do you generate code architecture diagram？which tool do you use？can you share？

candlerb commented 3 years ago

Maybe this helps: http://asciiflow.com/ (although I don't think it does the diagonal lines)

Darren-wh commented 3 years ago

@candlerb great. Thanks

hermanbanken commented 3 years ago

Thanks for all the reactions on my sample. I've posted my example in a Gist now, because I think this issue is not the place to be discussing how to use vars. I was merely explaining why it is essential for the UX of kustomize. I did not intent to start a tutorial here.

Same for the ASCII flow: please keep this issue focussed about the kustomize vars feature!

You can discuss my example in the Gist comments: https://gist.github.com/hermanbanken/3d0f232ffd86236c9f1f198c9452aad9. Note that it is not a full fledged repository, nor can I guarantee that it is 100% correct, as I had to extract it from our non-open-source repository (which I can not share).

yanniszark commented 3 years ago

Now that you have the powerful kyaml library, can't we do something like the following:

Replace all $(VAR) notation with setter comment notation: # {"$kpt-set":"VAR"}
Anchor the VAR to an object attribute, like what the current vars field does.
Namespace the var to the firstkustomization file that declares the var.

@Shell32-Natsu is there any discussion / plan for a timeline on an alternative? This seems to be one of the biggest pain-points of using kustomize today, as it breaks the encapsulation of the kustomization and prevents the user from composing bigger kustomizations out of smaller ones.

monopole commented 3 years ago

Enhancing vars will inexorably lead to kustomize becoming yet another crappy, half-baked domain specific configuration language.

@yanniszark you can use setters for many things, they're fine to use in kustomize.

Hoping someone will dust off the replacement transformer, or build a new one, so people can try it (#3492).

wstrange commented 3 years ago

Kustomize setters still show as "alpha", and there is confusion on the positioning of kustomize vs. kpt.

Some guidance on the formal direction would be very helpful.

yanniszark commented 3 years ago

@monopole this is great! I have to ask though, does this solution (replacement transformer) support partial substitution? Partial substitution is one of the main uses of vars today. For example, to substitute a service name and namespace in a Certificate object: https://github.com/kubeflow/kfserving/blob/master/config/certmanager/certificate.yaml#L20

To elaborate some more, I believe there are two main use-cases that people use vars for today. @jlewi has very accurately described them in Kubeflow's kustomize best practices doc: https://github.com/kubeflow/manifests/blob/master/docs/KustomizeBestPractices.md#eschew-vars

Internal subsitution of fields Kustomize isn't aware of (e.g., https://github.com/kubeflow/kfserving/blob/master/config/certmanager/certificate.yaml#L20)
Global configuration (e.g., cluster domain in all VirtualService objects).

The replacement transformer could probably solve the first one, but it needs to be more powerful and allow string substitutions. The second one is probably doable as well, if one uses a single global ConfigMap for the options and then refers to it for every replacement (but incurs a lot of repetition of multiple lines). Or a transformer for the specific use-case. Or setters, but it's not clear what kustomize's line is on their use.

seh commented 3 years ago

Adding on to @wstrange's comment above (https://github.com/kubernetes-sigs/kustomize/issues/2052#issuecomment-763968859), I asked about this confusing situation here: https://github.com/kubernetes-sigs/kustomize/issues/3429#issuecomment-757585928.

jlewi commented 3 years ago

@yanniszark you can use setters for many things, they're fine to use in kustomize.

I believe the answer is yes. FWIW I use kpt+kustomize routinely. General pattern is

kpt cfg set ${KUSTOMIZEDIR}
kustomize build ${KUSTOMIZEDIR}

GuyPaddock commented 2 years ago

I really wish replacements had:

A string prefix and suffix feature.
The ability to reference the namespace that was set in the kustomization.yaml.

I am finding that I need to replace the namespace component of a nginx.ingress.kubernetes.io/auth-tls-secret annotation, since it's not getting updated automatically for a namespaced secret generated in each overlay.

I already have a ConfigMap that I am using as the source of values, and I have a namespace declaration in the kustomization.yaml file, but it appears that runs last (which kind of makes sense). So, I wound up having to manually declare the metadata.namespace within the ConfigMap for every overlay just so I could reference it.

Then, to replace the namespace part of the nginx.ingress.kubernetes.io/auth-tls-secret annotation, I ended up using the delimiter feature (with a delimiter of /) to target the zero-th component. That meant that in the base overlays I had to put a dummy namespace in that annotation.

kubernetes-sigs / kustomize