Support composition in kustomize

We (@arrikto) have been using kustomize lately in order to deploy our software on Kubernetes, using a single base and multiple overlays for each deployment site. This is the typical inheritance scenario that kustomize supports.

As the number of deployments grow, however, so do the things that are common between them. For example, consider the following deployments:

dev: Consists of configuration logic for Minikube, local storage, image pull secrets, app secrets.
test: Consists of configuration logic for GKE, local storage, app secrets.
staging: Consists of configuration logic for GKE, S3, app secrets.
prod: Consists of configuration logic for GKE, S3, app secrets.

This a composition scenario, where common configuration logic is put into components for reusability. If one wants to achieve it with kustomize, they would probably create intermediate overlays for each component, and multi-base overlays for each deployment, e.g.:

                    +--------------+
                    |              |
       +----------->+  Component1  +----------+
       |            |              |          |
+------+-------+    +--------------+      +---v----+
|              |                          |        |
|  Deployment  |                          |  Base  |
|              |                          |        |
+------+-------+    +--------------+      +---^----+
       |            |              |          |
       +----------->+  Component2  +----------+
                    |              |
                    +--------------+

Unfortunately, this does not work, since kustomize ends up duplicating the YAMLs of the base. We have an example topology here that showcases this error:

(NOTE: You need kustomize v2.1 to run the examples)

$ kustomize build github.com/ioandr/kustomize-example//diamond/overlays/aggregate
`Error: accumulating resources: recursed merging from path '../overlay2': may not add resource with an already registered id: extensions_v1beta1_Deployment|~X|my-deployment`

If we change our components to not point to the base:


                           +--------------+
                           |              |
              +----------->+  Component1  |
              |            |              |
              |            +--------------+
              |
       +------+-------+       +--------+
       |              |       |        |
       |  Deployment  +------>+  Base  |
       |              |       |        |
       +------+-------+       +--------+
              |
              |            +--------------+
              |            |              |
              +----------->+  Component2  |
                           |              |
                           +--------------+

This works for creating new resources, but fails for patches, since patches must refer to resources that are defined in one of their bases. We have an example topology here that showcases this error:

$ kustomize build github.com/ioandr/kustomize-example//mixin/overlays/aggregate
Error: accumulating resources: recursed accumulation of path '../overlay1': failed to find target for patch extensions_v1beta1_Deployment|my-deployment

It seems that currently there's no better way than duplicating code across overlays, which is not maintainable in the long run. Still, we strongly believe that some form of composition support would make kustomize much more usable. This is echoed in other issues too (#759, #942), as well as high-profile projects like Kubeflow, which needed to create a tool on top of kustomize to support composition.

Are there any plans to support composition in kustomize and, if not, does the core team have a better suggestion to tackle this configuration reusability issue?

/cc @monopole @Liujingfang1

I've sent a PR (#2438) with a user story, where it shows a situation that cannot be solved by overlays in a proper manner (i.e., without copy-pasting stuff), but can be solved with components. Hopefully it's simple enough so that newcomers can understand its value. If you feel there's an aspect of this issue that is not covered by this example, or anything else that can be improved, please chime in on #2438.

Update for those interested on this feature and who have not followed PR #2438; the main user story regarding composition has been ACKed by @pwittrock and @monopole. You can read more about what it entails in #2438 (rendered example here), and more formally on a KEP that will be sent shortly, and will be linked here as well. Any comments on the example or the KEP will help a lot.

We (me, @ioandr and @pgpx) have created a KEP for this issue: https://github.com/kubernetes/enhancements/pull/1803 (rendered document here). In this KEP, we try to explain roughly the following:

What is the model that Kustomize supports now? (inheritance model)
What are some examples that don't fit to this model?
How could we make these examples easier to write? (composition model)
Why can't we do so right now? (we explain the issue with Kustomize's handling of multiple sibling overlays)
What do we propose? (add a Component kustomization kind)

For those interested in this feature, please chime in on the KEP.

This remains an issue for non-namespaced objects such as CustomResourceDefinition (CRD)s referenced from multiple deployments.

It means that if a and b both depend on some CRD, this:

kustomize build a | kubectl apply -f
kustomize build b | kubectl apply -f

... works fine, but

kustomize build uses_a_and_b_as_based | kubectl apply -f

will fail.

To work around this, it's necessary to modify both a and b to omit the CRDs definitions, but then they do not define all the resources they depend on and cannot be deployed independently. Instead of kustomize build a you now need another layer.

Components do help, but for simple cases like CRDs, this is something Kustomize itself could greatly help the user out with. Especially since the CRDs may be embedded in upstream sources where the user cannot necessarily factor them out into Kustomize components.

For example, kube-prometheus bundles its CRDs in a large flattened blob of kube yaml configuration. It isn't feasible to extract it to accomodate the Kustomize component model.

The simplest option here would seem to be to ignore duplicate resources where the key matches and the contents are the same.

Sample error:

Error: accumulating resources: accumulation err='accumulating resources from 'base/kube-prometheus': '...../base/kube-prometheus' must resolve to a file': recursed merging from path '...../base/kube-prometheus': may not add resource with an already registered id: apiextensions.k8s.io_v1_CustomResourceDefinition|~X|alertmanagerconfigs.monitoring.coreos.com

Here's a demo.

TL;DR: You can't write a single Kustomization that can be used standalone or further composed because of this behaviour. Either your standalone base cannot be applied due to missing CRDs, with errors like:

Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post democrds.demo.example.com)

or if you include the CRDs in each base your combined deployment cannot be generated due to kustomize errors like

Error: accumulating resources: accumulation err='accumulating resources from '../crd_b': '/home/craig/.nosnapshot/craig_tmp/demo/crd_b' must resolve to a file': recursed merging from path '/home/craig/.nosnapshot/craig_tmp/demo/crd_b': may not add resource with an already registered id: apiextensions.k8s.io_v1_CustomResourceDefinition|~X|democrds.demo.example.com

It shows that it's not possible to define a single kustomization such that it can stand alone for deployment to a cluster and can be composed into a deployment containing another kustomization that depends on the same CRD. You have to instead split out the CRDs, and build another layer that composes "kustomizations without CRDs" and "just the CRDs" in various combinatorial ways depending on what you wish to deploy. This is painful for maintenance, but also breaks otherwise-logical compositions.

Crucially, generating and applying each piece of the Kustomization individually produces a sensible and correct result, even though Kustomize refuses to generate a kustomization that uses only those two as bases. And you can even concatenate the kustomize output of both parts, then kubectl apply that in a single pass, and get a correct result.

Kustomize should really be de-dup'ing identical objects here.

Components can help here, but only if you control all the definitions. Common real world deployments like kube-prometheus bundle their CRDs, behaving exactly like the a_crd and b_crd in the demo below.

The demo defines the following bases:

crd - contains the crd definition only
a - contains resource "a" using the crd only
b - contains resource "a" using the crd only

and composes them in a few ways:

crd_a uses crd and a as bases, so it can be deployed standalone to deploy "a"
crd_b uses crd and b as bases, so it can be deployed standalone to deploy "b"
crd_b_crd_a uses crd_a and crd_b as bases. It should be able to deploy both "a" and "b" at the same time but due to this issue it cannot be
crd_a_b uses crd, a, and b as bases, so it only includes the CRD definition once, and can be deployed.

What convinces me that Kustomize's behaviour is wrong here is that you can sequentially deploy crd_a and crd_b successfully:

$ kustomize build crd_a | kubectl apply -f -
namespace/a created
customresourcedefinition.apiextensions.k8s.io/democrds.demo.example.com created
democrd.demo.example.com/democrd-a created
$ kustomize build crd_b | kubectl apply -f -
namespace/b created
customresourcedefinition.apiextensions.k8s.io/democrds.demo.example.com unchanged
democrd.demo.example.com/democrd-b created

In fact, you can even concatenate the generated configurations and deploy them (the --validate=false is just because concatenating the output confuses kubectl a little):

$ cat <(kustomize build crd_a) <(kustomize build crd_b) | kubectl apply -f - --validate=false
namespace/a created
customresourcedefinition.apiextensions.k8s.io/democrds.demo.example.com created
namespace/b created
customresourcedefinition.apiextensions.k8s.io/democrds.demo.example.com unchanged
democrd.demo.example.com/democrd-b created

kustomize is fine with you generating and applying the CRD then both bases that use it:

$ kustomize build crd_a_b | kubectl apply -f -
namespace/a unchanged
namespace/b unchanged
customresourcedefinition.apiextensions.k8s.io/democrds.demo.example.com unchanged
democrd.demo.example.com/democrd-a unchanged
democrd.demo.example.com/democrd-b unchanged

but not composing two bases that are complete including their dependencies:

$ kustomize build crd_a_crd_b | kubectl apply -f -
Error: accumulating resources: accumulation err='accumulating resources from '../crd_b': '/home/craig/.nosnapshot/craig_tmp/demo/crd_b' must resolve to a file': recursed merging from path '/home/craig/.nosnapshot/craig_tmp/demo/crd_b': may not add resource with an already registered id: apiextensions.k8s.io_v1_CustomResourceDefinition|~X|democrds.demo.example.com
error: no objects passed to apply

However, the pieces that kustomize will let you compose successfully are invalid when deployed standalone:

$ kustomize build a | kubectl apply -f -
namespace/a created
Error from server (NotFound): error when creating "STDIN": the server could not find the requested resource (post democrds.demo.example.com)

so you can't write a single Kustomization that can be used standalone or further composed.

Generate the demo with the following

mkdir -p kustomization.yaml
cat > ./crd/kustomization.yaml <<'__END__'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - "crd.yaml"
__END__
mkdir -p crd.yaml
cat > ./crd/crd.yaml <<'__END__'
apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: democrds.demo.example.com
spec:
  group: demo.example.com
  versions:
    - name: v1
      served: true
      storage: true
      schema:
        openAPIV3Schema:
          type: object
          properties:
            spec:
              type: object
              properties:
                field:
                  type: string
  scope: Namespaced
  names:
    singular: democrd
    plural: democrds
    kind: DemoCrd

__END__
mkdir -p a_crd.yaml
cat > ./a/a_crd.yaml <<'__END__'
apiVersion: "demo.example.com/v1"
kind: DemoCrd
metadata:
  name: democrd-a
spec:
  field: "a"
__END__
mkdir -p kustomization.yaml
cat > ./a/kustomization.yaml <<'__END__'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: a
resources:
  - namespace.yaml
  - a_crd.yaml
__END__
mkdir -p namespace.yaml
cat > ./a/namespace.yaml <<'__END__'
apiVersion: v1
kind: Namespace
metadata:
  name: a
__END__
mkdir -p b_crd.yaml
cat > ./b/b_crd.yaml <<'__END__'
apiVersion: "demo.example.com/v1"
kind: DemoCrd
metadata:
  name: democrd-b
spec:
  field: "b"
__END__
mkdir -p kustomization.yaml
cat > ./b/kustomization.yaml <<'__END__'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: b
resources:
  - namespace.yaml
  - b_crd.yaml
__END__
mkdir -p namespace.yaml
cat > ./b/namespace.yaml <<'__END__'
apiVersion: v1
kind: Namespace
metadata:
  name: b
__END__
mkdir -p kustomization.yaml
cat > ./crd_a_b/kustomization.yaml <<'__END__'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../crd
  - ../a
  - ../b
__END__
mkdir -p kustomization.yaml
cat > ./crd_a/kustomization.yaml <<'__END__'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../crd
  - ../a
__END__
mkdir -p kustomization.yaml
cat > ./crd_b/kustomization.yaml <<'__END__'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
  - ../crd
  - ../b
__END__
mkdir -p kustomization.yaml
cat > ./crd_a_crd_b/kustomization.yaml <<'__END__'
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
bases:
        - ../crd_a
        - ../crd_b
__END__

TL;DR: You can't write a single Kustomization that can be used standalone or further composed because of this behaviour.

Yup, it's totally unusable for anything other than one deployment. Event doing prod + staging leads to full replication of everything, otherwise it's either unpatcheable or duplicated. I found one hack though: use GitOps (flux2) to deploy things in "stages" and thus they are not really dependent on each other outside of flux itself. But any new resource may need a full rearrangement or even creating a new separate "stage".

@pkit Exactly - and kustomize is already a wrapper tool, it's not something you really want to need a wrapper-wrapper for...

kubernetes-sigs / kustomize

Support composition in kustomize #1251