search

fluxcd / kustomize-controller

The GitOps Toolkit Kustomize reconciler
https://fluxcd.io
Apache License 2.0
252 stars 179 forks source link

Constant config drift on webhook rules expansion #792

Open mbrancato opened 1 year ago

mbrancato commented 1 year ago

Every time I upgrade Knative, I need to manually expand the webhook rules because on apply, they are modified and no longer match the manifest.

e.g. https://github.com/knative/serving/blob/25de8121f169719c52e609308f701f45ddba5166/config/core/webhooks/domainmapping-validation.yaml https://github.com/knative/serving/blob/646d0b1a9626cf636faf3e9b3a58bdd8eeb90ee5/config/core/webhooks/defaulting.yaml

The rules start out like this:

rules:
  - apiGroups:
      - serving.knative.dev
    apiVersions:
      - "*"
    operations:
      - CREATE
      - UPDATE
    scope: "*"
    resources:
      - domainmappings
      - domainmappings/status

and are modified to this:

rules:
  - apiGroups:
      - serving.knative.dev
    apiVersions:
    - v1alpha1
    operations:
      - CREATE
      - UPDATE
    scope: "*"
    resources:
      - domainmappings
      - domainmappings/status
  - apiGroups:
      - serving.knative.dev
    apiVersions:
    - v1beta1
    operations:
      - CREATE
      - UPDATE
    scope: "*"
    resources:
      - domainmappings
      - domainmappings/status

I've previously opened issues in other projects without progress: https://github.com/knative/serving/issues/13449 https://github.com/kubernetes/kubernetes/issues/107318

mbrancato commented 1 year ago

I've created a repo to test this:

https://github.com/mbrancato/test-knative-serving

Some things to help:

% flux diff kustomization test-knative-serving --path "./knative-serving" -n default ``` ✓ Kustomization diffing... ► MutatingWebhookConfiguration/webhook.serving.knative.dev drifted metadata.generation ± value change - 8 + 9 webhooks.webhook.serving.knative.dev.rules - nine list entries removed: + one list entry added: - resources: - resources: │ - metrics │ - metrics │ - metrics/status │ - podautoscalers │ scope: "*" │ - certificates │ apiGroups: │ - ingresses │ - autoscaling.internal.knative.dev │ - serverlessservices │ apiVersions: │ - configurations │ - v1alpha1 │ - revisions │ operations: │ - routes │ - CREATE │ - services │ - UPDATE │ scope: "*" - resources: │ apiGroups: │ - podautoscalers │ - autoscaling.internal.knative.dev │ - podautoscalers/status │ - networking.internal.knative.dev │ scope: "*" │ - serving.knative.dev │ apiGroups: │ apiVersions: │ - autoscaling.internal.knative.dev │ - "*" │ apiVersions: │ operations: │ - v1alpha1 │ - CREATE │ operations: │ - UPDATE │ - CREATE │ - UPDATE - resources: │ - certificates │ - certificates/status │ scope: "*" │ apiGroups: │ - networking.internal.knative.dev │ apiVersions: │ - v1alpha1 │ operations: │ - CREATE │ - UPDATE - resources: │ - ingresses │ - ingresses/status │ scope: "*" │ apiGroups: │ - networking.internal.knative.dev │ apiVersions: │ - v1alpha1 │ operations: │ - CREATE │ - UPDATE - resources: │ - serverlessservices │ - serverlessservices/status │ scope: "*" │ apiGroups: │ - networking.internal.knative.dev │ apiVersions: │ - v1alpha1 │ operations: │ - CREATE │ - UPDATE - resources: │ - configurations │ - configurations/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE - resources: │ - revisions │ - revisions/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE - resources: │ - routes │ - routes/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE - resources: │ - services │ - services/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE ► MutatingWebhookConfiguration/webhook.domainmapping.serving.knative.dev drifted metadata.generation ± value change - 8 + 9 webhooks.webhook.domainmapping.serving.knative.dev.rules - two list entries removed: + one list entry added: - resources: - resources: │ - domainmappings │ - domainmappings │ - domainmappings/status │ - domainmappings/status │ scope: "*" │ scope: "*" │ apiGroups: │ apiGroups: │ - serving.knative.dev │ - serving.knative.dev │ apiVersions: │ apiVersions: │ - v1alpha1 │ - "*" │ operations: │ operations: │ - CREATE │ - CREATE │ - UPDATE │ - UPDATE - resources: │ - domainmappings │ - domainmappings/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1beta1 │ operations: │ - CREATE │ - UPDATE ► ValidatingWebhookConfiguration/validation.webhook.domainmapping.serving.knative.dev drifted metadata.generation ± value change - 8 + 9 webhooks.validation.webhook.domainmapping.serving.knative.dev.rules - two list entries removed: + one list entry added: - resources: - resources: │ - domainmappings │ - domainmappings │ - domainmappings/status │ - domainmappings/status │ scope: "*" │ scope: "*" │ apiGroups: │ apiGroups: │ - serving.knative.dev │ - serving.knative.dev │ apiVersions: │ apiVersions: │ - v1alpha1 │ - "*" │ operations: │ operations: │ - CREATE │ - CREATE │ - UPDATE │ - UPDATE │ - DELETE │ - DELETE - resources: │ - domainmappings │ - domainmappings/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1beta1 │ operations: │ - CREATE │ - UPDATE │ - DELETE ► ValidatingWebhookConfiguration/validation.webhook.serving.knative.dev drifted metadata.generation ± value change - 8 + 9 webhooks.validation.webhook.serving.knative.dev.rules - nine list entries removed: + one list entry added: - resources: - resources: │ - metrics │ - metrics │ - metrics/status │ - podautoscalers │ scope: "*" │ - certificates │ apiGroups: │ - ingresses │ - autoscaling.internal.knative.dev │ - serverlessservices │ apiVersions: │ - configurations │ - v1alpha1 │ - revisions │ operations: │ - routes │ - CREATE │ - services │ - UPDATE │ scope: "*" │ - DELETE │ apiGroups: - resources: │ - autoscaling.internal.knative.dev │ - podautoscalers │ - networking.internal.knative.dev │ - podautoscalers/status │ - serving.knative.dev │ scope: "*" │ apiVersions: │ apiGroups: │ - "*" │ - autoscaling.internal.knative.dev │ operations: │ apiVersions: │ - CREATE │ - v1alpha1 │ - UPDATE │ operations: │ - DELETE │ - CREATE │ - UPDATE │ - DELETE - resources: │ - certificates │ - certificates/status │ scope: "*" │ apiGroups: │ - networking.internal.knative.dev │ apiVersions: │ - v1alpha1 │ operations: │ - CREATE │ - UPDATE │ - DELETE - resources: │ - ingresses │ - ingresses/status │ scope: "*" │ apiGroups: │ - networking.internal.knative.dev │ apiVersions: │ - v1alpha1 │ operations: │ - CREATE │ - UPDATE │ - DELETE - resources: │ - serverlessservices │ - serverlessservices/status │ scope: "*" │ apiGroups: │ - networking.internal.knative.dev │ apiVersions: │ - v1alpha1 │ operations: │ - CREATE │ - UPDATE │ - DELETE - resources: │ - configurations │ - configurations/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE │ - DELETE - resources: │ - revisions │ - revisions/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE │ - DELETE - resources: │ - routes │ - routes/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE │ - DELETE - resources: │ - services │ - services/status │ scope: "*" │ apiGroups: │ - serving.knative.dev │ apiVersions: │ - v1 │ operations: │ - CREATE │ - UPDATE │ - DELETE ⚠️ identified at least one change, exiting with non-zero exit code ```
% flux -n default reconcile ks test-knative-serving ``` ► annotating Kustomization test-knative-serving in default namespace ✔ Kustomization annotated ◎ waiting for Kustomization reconciliation ✔ applied revision main/e2dfd732fe044af0fb79dd629a38c682fa7cefd8 ```
% kubectl describe ks test-knative-serving ``` Name: test-knative-serving Namespace: default Labels: Annotations: reconcile.fluxcd.io/requestedAt: 2023-02-09T09:39:53.049406-05:00 API Version: kustomize.toolkit.fluxcd.io/v1beta2 Kind: Kustomization Metadata: Creation Timestamp: 2023-02-09T14:24:44Z Finalizers: finalizers.fluxcd.io Generation: 1 Managed Fields: API Version: kustomize.toolkit.fluxcd.io/v1beta2 Fields Type: FieldsV1 fieldsV1: f:metadata: f:finalizers: .: v:"finalizers.fluxcd.io": Manager: gotk-kustomize-controller Operation: Update Time: 2023-02-09T14:24:44Z API Version: kustomize.toolkit.fluxcd.io/v1beta2 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: .: f:kubectl.kubernetes.io/last-applied-configuration: f:spec: .: f:force: f:interval: f:path: f:prune: f:sourceRef: .: f:kind: f:name: f:targetNamespace: Manager: kubectl-client-side-apply Operation: Update Time: 2023-02-09T14:24:44Z API Version: kustomize.toolkit.fluxcd.io/v1beta2 Fields Type: FieldsV1 fieldsV1: f:metadata: f:annotations: f:reconcile.fluxcd.io/requestedAt: Manager: flux Operation: Update Time: 2023-02-09T14:39:51Z API Version: kustomize.toolkit.fluxcd.io/v1beta2 Fields Type: FieldsV1 fieldsV1: f:status: f:conditions: f:inventory: .: f:entries: f:lastAppliedRevision: f:lastAttemptedRevision: f:lastHandledReconcileAt: f:observedGeneration: Manager: gotk-kustomize-controller Operation: Update Subresource: status Time: 2023-02-09T14:39:53Z Resource Version: 11313 UID: a7a5abc3-f670-4ca4-8357-7321b27dc29e Spec: Force: false Interval: 10m Path: ./knative-serving Prune: true Source Ref: Kind: GitRepository Name: test-knative-serving Target Namespace: default Status: Conditions: Last Transition Time: 2023-02-09T14:39:53Z Message: Applied revision: main/e2dfd732fe044af0fb79dd629a38c682fa7cefd8 Observed Generation: 1 Reason: ReconciliationSucceeded Status: True Type: Ready Inventory: Entries: Id: _certificates.networking.internal.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _clusterdomainclaims.networking.internal.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _configurations.serving.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _domainmappings.serving.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _images.caching.internal.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _ingresses.networking.internal.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _metrics.autoscaling.internal.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _podautoscalers.autoscaling.internal.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _revisions.serving.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _routes.serving.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _serverlessservices.networking.internal.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _services.serving.knative.dev_apiextensions.k8s.io_CustomResourceDefinition V: v1 Id: _default__Namespace V: v1 Id: default_controller__ServiceAccount V: v1 Id: _knative-serving-addressable-resolver_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-admin_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-aggregated-addressable-resolver_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-core_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-namespaced-admin_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-namespaced-edit_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-namespaced-view_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-podspecable-binding_rbac.authorization.k8s.io_ClusterRole V: v1 Id: _knative-serving-controller-addressable-resolver_rbac.authorization.k8s.io_ClusterRoleBinding V: v1 Id: _knative-serving-controller-admin_rbac.authorization.k8s.io_ClusterRoleBinding V: v1 Id: default_config-autoscaler__ConfigMap V: v1 Id: default_config-defaults__ConfigMap V: v1 Id: default_config-deployment__ConfigMap V: v1 Id: default_config-domain__ConfigMap V: v1 Id: default_config-features__ConfigMap V: v1 Id: default_config-gc__ConfigMap V: v1 Id: default_config-leader-election__ConfigMap V: v1 Id: default_config-logging__ConfigMap V: v1 Id: default_config-network__ConfigMap V: v1 Id: default_config-observability__ConfigMap V: v1 Id: default_config-tracing__ConfigMap V: v1 Id: default_domainmapping-webhook-certs__Secret V: v1 Id: default_knative-serving-certs__Secret V: v1 Id: default_serving-certs-ctrl-ca__Secret V: v1 Id: default_webhook-certs__Secret V: v1 Id: default_activator-service__Service V: v1 Id: default_autoscaler__Service V: v1 Id: default_controller__Service V: v1 Id: default_domainmapping-webhook__Service V: v1 Id: default_webhook__Service V: v1 Id: default_activator_apps_Deployment V: v1 Id: default_autoscaler_apps_Deployment V: v1 Id: default_controller_apps_Deployment V: v1 Id: default_domain-mapping_apps_Deployment V: v1 Id: default_domainmapping-webhook_apps_Deployment V: v1 Id: default_webhook_apps_Deployment V: v1 Id: default_activator-pdb_policy_PodDisruptionBudget V: v1 Id: default_webhook-pdb_policy_PodDisruptionBudget V: v1 Id: default_activator_autoscaling_HorizontalPodAutoscaler V: v2 Id: default_webhook_autoscaling_HorizontalPodAutoscaler V: v2 Id: default_queue-proxy_caching.internal.knative.dev_Image V: v1alpha1 Id: _webhook.domainmapping.serving.knative.dev_admissionregistration.k8s.io_MutatingWebhookConfiguration V: v1 Id: _webhook.serving.knative.dev_admissionregistration.k8s.io_MutatingWebhookConfiguration V: v1 Id: _config.webhook.serving.knative.dev_admissionregistration.k8s.io_ValidatingWebhookConfiguration V: v1 Id: _validation.webhook.domainmapping.serving.knative.dev_admissionregistration.k8s.io_ValidatingWebhookConfiguration V: v1 Id: _validation.webhook.serving.knative.dev_admissionregistration.k8s.io_ValidatingWebhookConfiguration V: v1 Last Applied Revision: main/e2dfd732fe044af0fb79dd629a38c682fa7cefd8 Last Attempted Revision: main/e2dfd732fe044af0fb79dd629a38c682fa7cefd8 Last Handled Reconcile At: 2023-02-09T09:39:53.049406-05:00 Observed Generation: 1 Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Progressing 15m kustomize-controller CustomResourceDefinition/certificates.networking.internal.knative.dev configured CustomResourceDefinition/clusterdomainclaims.networking.internal.knative.dev configured CustomResourceDefinition/configurations.serving.knative.dev configured CustomResourceDefinition/domainmappings.serving.knative.dev configured CustomResourceDefinition/images.caching.internal.knative.dev configured CustomResourceDefinition/ingresses.networking.internal.knative.dev configured CustomResourceDefinition/metrics.autoscaling.internal.knative.dev configured CustomResourceDefinition/podautoscalers.autoscaling.internal.knative.dev configured CustomResourceDefinition/revisions.serving.knative.dev configured CustomResourceDefinition/routes.serving.knative.dev configured CustomResourceDefinition/serverlessservices.networking.internal.knative.dev configured CustomResourceDefinition/services.serving.knative.dev configured Namespace/default configured ServiceAccount/default/controller created ClusterRole/knative-serving-addressable-resolver configured ClusterRole/knative-serving-admin configured ClusterRole/knative-serving-aggregated-addressable-resolver configured ClusterRole/knative-serving-core configured ClusterRole/knative-serving-namespaced-admin configured ClusterRole/knative-serving-namespaced-edit configured ClusterRole/knative-serving-namespaced-view configured ClusterRole/knative-serving-podspecable-binding configured ClusterRoleBinding/knative-serving-controller-addressable-resolver configured ClusterRoleBinding/knative-serving-controller-admin configured ConfigMap/default/config-autoscaler created ConfigMap/default/config-defaults created ConfigMap/default/config-deployment created ConfigMap/default/config-domain created ConfigMap/default/config-features created ConfigMap/default/config-gc created ConfigMap/default/config-leader-election created ConfigMap/default/config-logging created ConfigMap/default/config-network created ConfigMap/default/config-observability created ConfigMap/default/config-tracing created Secret/default/domainmapping-webhook-certs created Secret/default/knative-serving-certs created Secret/default/serving-certs-ctrl-ca created Secret/default/webhook-certs created Service/default/activator-service created Service/default/autoscaler created Service/default/controller created Service/default/domainmapping-webhook created Service/default/webhook created Deployment/default/activator created Deployment/default/autoscaler created Deployment/default/controller created Deployment/default/domain-mapping created Deployment/default/domainmapping-webhook created Deployment/default/webhook created PodDisruptionBudget/default/activator-pdb created PodDisruptionBudget/default/webhook-pdb created HorizontalPodAutoscaler/default/activator created HorizontalPodAutoscaler/default/webhook created Image/default/queue-proxy created MutatingWebhookConfiguration/webhook.domainmapping.serving.knative.dev configured MutatingWebhookConfiguration/webhook.serving.knative.dev configured ValidatingWebhookConfiguration/config.webhook.serving.knative.dev configured ValidatingWebhookConfiguration/validation.webhook.domainmapping.serving.knative.dev configured ValidatingWebhookConfiguration/validation.webhook.serving.knative.dev configured Normal ReconciliationSucceeded 15m kustomize-controller Reconciliation finished in 2.418812406s, next run in 10m0s Normal Progressing 13m kustomize-controller CustomResourceDefinition/certificates.networking.internal.knative.dev created CustomResourceDefinition/clusterdomainclaims.networking.internal.knative.dev created CustomResourceDefinition/configurations.serving.knative.dev created CustomResourceDefinition/domainmappings.serving.knative.dev created CustomResourceDefinition/images.caching.internal.knative.dev created CustomResourceDefinition/ingresses.networking.internal.knative.dev created CustomResourceDefinition/metrics.autoscaling.internal.knative.dev created CustomResourceDefinition/podautoscalers.autoscaling.internal.knative.dev created CustomResourceDefinition/revisions.serving.knative.dev created CustomResourceDefinition/routes.serving.knative.dev created CustomResourceDefinition/serverlessservices.networking.internal.knative.dev created CustomResourceDefinition/services.serving.knative.dev created ClusterRole/knative-serving-addressable-resolver created ClusterRole/knative-serving-admin created ClusterRole/knative-serving-aggregated-addressable-resolver created ClusterRole/knative-serving-core created ClusterRole/knative-serving-namespaced-admin created ClusterRole/knative-serving-namespaced-edit created ClusterRole/knative-serving-namespaced-view created ClusterRole/knative-serving-podspecable-binding created ClusterRoleBinding/knative-serving-controller-addressable-resolver created ClusterRoleBinding/knative-serving-controller-admin created Image/default/queue-proxy created MutatingWebhookConfiguration/webhook.domainmapping.serving.knative.dev created MutatingWebhookConfiguration/webhook.serving.knative.dev created ValidatingWebhookConfiguration/config.webhook.serving.knative.dev created ValidatingWebhookConfiguration/validation.webhook.domainmapping.serving.knative.dev created ValidatingWebhookConfiguration/validation.webhook.serving.knative.dev created Normal ReconciliationSucceeded 13m kustomize-controller Reconciliation finished in 3.297771017s, next run in 10m0s Normal ReconciliationSucceeded 11m kustomize-controller Reconciliation finished in 1.139306463s, next run in 10m0s Normal ReconciliationSucceeded 5m19s kustomize-controller Reconciliation finished in 1.103780242s, next run in 10m0s Normal ReconciliationSucceeded 69s kustomize-controller Reconciliation finished in 2.213234s, next run in 10m0s Normal Progressing 12s (x4 over 11m) kustomize-controller MutatingWebhookConfiguration/webhook.domainmapping.serving.knative.dev configured MutatingWebhookConfiguration/webhook.serving.knative.dev configured ValidatingWebhookConfiguration/validation.webhook.domainmapping.serving.knative.dev configured ValidatingWebhookConfiguration/validation.webhook.serving.knative.dev configured Normal ReconciliationSucceeded 12s kustomize-controller Reconciliation finished in 2.322141s, next run in 10m0s ```