karmada-io / karmada

Open, Multi-Cloud, Multi-Cluster Kubernetes Orchestration
https://karmada.io
Apache License 2.0
4.37k stars 865 forks source link

Setting up declarative ResourceInterpreterCustomization #5238

Open DraganDjuric94 opened 1 month ago

DraganDjuric94 commented 1 month ago

We have set up Karmada using the karmada-operator Helm chart, version v1.10.3. Values used to release the Karmada instance:

apiVersion: operator.karmada.io/v1alpha1
kind: Karmada
metadata:
  name: karmada-demo
  namespace: platform
spec:
  components:
    etcd:
      local:
        replicas: 3
    karmadaAPIServer:
      imageRepository: registry.k8s.io/kube-apiserver
      replicas: 3
      certSANs:
        - "kubernetes.default.svc"
        - "my-domain.com"
    karmadaWebhook:
      extraArgs:
        v: "50"

We are trying to set up a declarative ResourceInterpreterCustomization, but have limited success. Some of the functions seem to work fine, some of them work only the first time the ResourceBinding is created, and some don't work at all.

For example, we use Gloo for accessing the applications. Gloo has a VirtualService kind whose statuses are generated on the member cluster side. For this purpose we have used the customization of the retention field and it is working as intended. However, when we try to change the aggregated status, no changes are made (only the default behavior is applied). We have tried using the healthInterpretation as well, but it works only the first time ResouceBinding is created (if we change to return false in the example below and apply the customization, health remains unchanged).

Used ResourceInterpreterCustomization:

apiVersion: config.karmada.io/v1alpha1
kind: ResourceInterpreterCustomization
metadata:
  name: vs-aggregated-status
spec:
  target:
    apiVersion: gateway.solo.io/v1
    kind: VirtualService
  customizations:
    retention: # <-- Works as intended
      luaScript: >
        function Retain(desiredObj, observedObj)
          desiredObj.status = observedObj.status
          return desiredObj
        end
    statusAggregation: # <-- Doesn't work at all (no "meow" field)
      luaScript: >
        function AggregateStatus(desiredObj, statusItems)
          if desiredObj.status == nil then
            desiredObj.status = {}
          end

          desiredObj.status.meow = "meow"
          return desiredObj
        end
    healthInterpretation: # <-- Works only the first time ResourceBinding is created
      luaScript: >
        function InterpretHealth(observedObj)
          return true
        end

When using the karmada cli interpret locally, everything works as intended (all three functions return correct results):

kubectl karmada interpret -f /path/to/resource-interpreter-customization.yaml --observed-file vs-observed.yaml --operation AggregateStatus

....
status:
    meow: meow
    statuses:
        gloo-system:
            reportedBy: gloo
            state: Accepted
            subresourceStatuses:
               ...

We have tried higher log verbosity on the Karmada webhook, but with no luck in debugging the issue.

Any info regarding the possible causes would be great :)

If you need more information, please let me know.

Environment:

XiShanYongYe-Chang commented 1 month ago

/cc @chaunceyjiang @chaosi-zju Can you help take a look?