search

fluxcd / kustomize-controller

The GitOps Toolkit Kustomize reconciler
https://fluxcd.io
Apache License 2.0
253 stars 182 forks source link

A sharded kustomization with a dependsOn pointing to a Kustomization in another shard fails to find it #1175

Closed jordiclariana closed 1 month ago

jordiclariana commented 5 months ago

Hi,

So this is the scenario, I have 2 kustomizations, each one in different namespaces (although I don't think this actually matter), and each one in different shards (that we setup automatically with a mutatingwebhook when a KS is applied/updated in the cluster). These are the snippets of the kustomizations:

Kustomization #1:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: app
    kustomize.toolkit.fluxcd.io/namespace: application1-namespace
    sharding.fluxcd.io/key: shard-1
  name: application1-name
  namespace: application1-namespace
spec:
  dependsOn:
  - name: app
    namespace: application2-namespace
[...]

Kustomization #2:

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  labels:
    kustomize.toolkit.fluxcd.io/name: app
    kustomize.toolkit.fluxcd.io/namespace: application2-namespace
    sharding.fluxcd.io/key: shard-2
  name: app
  namespace: application2-namespace
spec:
[...]

Whenever Kustomization #1 tries to reconciliate, we get the error:

dependency 'application2-namespace/app' not found: Kustomization.kustomize.toolkit.fluxcd.io "app" not found

I was inspecting the code, and the Get for the checkDependencies function seems pretty normal, and does not seem like it is limiting what shard should it take the resources from. But then I noticed that in main.go the manager K8s Client is allow cache only for those resources within the watchSelector (in this case shards), and I have the feeling that that cache is used always and therefore there are no kustomizations outside its own shard, so can't find the dependency. I think this is fine for most of the cases, but in the case of a dependsOn it does not make sense to me. The dependsOn should not depend on which shard the dependency is in, should it?

This issue is between a bug and a feature request, tbh. I will let that up to you to decide, but to me is pretty limiting.

Here's my flux versions:

$ flux version
flux: v2.2.3
distribution: flux-
helm-controller: v0.37.4
image-automation-controller: v0.37.1
image-reflector-controller: v0.31.2
kustomize-controller: v1.2.2
notification-controller: v1.2.4
source-controller: v1.2.4
jusr3124 commented 2 months ago

this is an interesting use-case. I am also looking for this solution. Did you found something in the meanwhile?

jordiclariana commented 2 months ago

Hi. No, we just took another approach to our very particular use case (that could be solved without dependsOn). Is not as pretty, but works for us. I still think this should be addressed.