Nested Kustomizations and prune

[bozho]

Hi all,

I'm trying to understand prune behaviour when using nested kustomizations. I hit this problem when trying to create a test repository with composable components, following this.

I created a minimal sample trying to figure it out. My directories look like this:

├── clusters
|   └── k8s-small
|       ├── flux-system
|       ├── kustomization.yaml
|       └── podinfo.yaml
|
└── platform
    └── podinfo
        └── _base
            ├── kustomization.yaml
            └── podinfo.yaml

# clusters/k8s-small/kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml

# clusters/k8s-small/podinfo.yaml

---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 15m
  sourceRef:
    kind: GitRepository
    name: flux-system
  path: ./platform/podinfo/_base
  prune: false
  wait: true
  timeout: 5m

# platform/podinfo/_base/kustomization.yaml

---
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - podinfo.yaml

# platform/podinfo/_base/podinfo.yaml

---
apiVersion: source.toolkit.fluxcd.io/v1
kind: GitRepository
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 1m0s
  ref:
    branch: master
  url: https://github.com/stefanprodan/podinfo
---
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: podinfo
  namespace: flux-system
spec:
  interval: 30m0s
  path: ./kustomize
  prune: false
  retryInterval: 2m0s
  sourceRef:
    kind: GitRepository
    name: podinfo
  targetNamespace: default
  timeout: 3m0s
  wait: true

When prune is false in both Kustomizations, podinfo resources get created as expected.

When I set prune to true only in the "parent" Kustomization (clusters/k8s-small/podinfo.yaml), any running podinfo pods get deleted and then redeployed.

flux logs shows these lines:

2024-11-04T15:30:27.688Z info Kustomization/podinfo.flux-system - server-side apply completed
2024-11-04T15:30:27.825Z info Kustomization/podinfo.flux-system - garbage collection completed: HorizontalPodAutoscaler/default/podinfo deleted
Deployment/default/podinfo deleted
Service/default/podinfo deleted
2024-11-04T15:30:28.029Z info Kustomization/podinfo.flux-system - Reconciliation finished in 604.521568ms, next run in 15m0s
2024-11-04T15:30:28.327Z info Kustomization/podinfo.flux-system - server-side apply completed
2024-11-04T15:30:58.499Z info Kustomization/podinfo.flux-system - Reconciliation finished in 30.468408748s, next run in 30m0s

When I set prune to true only in the "child" Kustomization (platform/podinfo/_base/podinfo.yaml), any running podinfo pods get deleted.

When I use the simple podinfo Kustomization structure from flux tutorial, everything works as expected, with prune set to true.

[stefanprodan]

You can’t have a kustomization.yaml in the cluster dir, this would make Flux uninstall itself. The issue you are getting is from the fact that you have 2 Kustomizations fighting each other, rename the one under clusters to apps. See here an example of how to structure your repo: https://github.com/fluxcd/flux2-kustomize-helm-example

[bozho]

Hi,

I'm not sure I understand... I haven't seen Flux trying to uninstall itself or any of the components with my repo structure, only the podinfo resources (depending on prune values).

I've tried renaming the "parent" kustomization.yaml to apps.yaml and observed the same behaviour (I did have to add metadata.name to the file after renaming it - flux was complaining about it missing).

I've set up a minimal repo that replicates the issue I'm seeing.

I stumbled upon this problem while reading this blog about organising k8s/flux repos for scale, with reusable components. This is their example repo, and all their cluster instances have kustomization.yaml files inside <cluster>/platform directory, alongside flux-system directory.

I was following along, adding components to my repo - and everything works fine when the components are installed using Helm (as is the case in the blog). I've only seen the problem with the podinfo component, which I'm installing using Flux kustomization git repo (same as in my minimal example above).

[bozho]

Actually, if I bootstrap Flux without the parent kustomization.yaml file in my minimal example repo, I get the same behaviour with podinfo resources being deleted after provisioning.

Then, if I add the parent kustomization.yaml file, commit and push, Flux seems to try to uninstall itself.

OTOH, if I change the cluster folder structure (to the one used in the blog) and move "parent" YAML files to a platform directory, which is alongside flux-system directory:

├── clusters
|   └── k8s-small
|       ├── flux-system
|       └── platform
|           ├── kustomization.yaml
|           └── podinfo.yaml
|
└── platform
    └── podinfo
        └── _base
            ├── kustomization.yaml
            └── podinfo.yaml

I can start without the parent kustomization.yaml file. Flux bootstrap will deploy podinfo resources and immediately garbage-collect them. If I then add, commit and push the parent kustomization.yaml file, Flux will not attempt to uninstall itself, but the podinfo/prune behaviour is still there.

[stefanprodan]

You cannot have 2 Kustomizations with the same name in the same namespace, they will override each other. Same with the kustomization.yaml in the cluster dir. Your repo has both of these things, please follow the repo structure I posted above.

[bozho]

Oh my word... I understand now what you've been trying to explain :-). Both parent and child Kustomizations had the same name! What a silly mistake and a case of staring too hard into something and missing the obvious.

Thank you very much for your patience with silly me.