Manifest CR is constantly in "Ready" state during deletion

[Tomasz-Smelcerz-SAP]

Description

During the deletion the Manifest CR is in Ready state, it never enters Deleting state. The only thing that indicates the deletion in progress is the deletionTimestamp in the object metadata. It as observed on PROD, KLM version: europe-docker.pkg.dev/kyma-project/prod/lifecycle-manager:1.0.0. Some other modules are in the Warning state, but this one is Ready When testing on STAGE (the same KLM version) with all modules Ready, the module being deleted correctly switched to Deleting state during deletion.

Steps to reproduce

• connect to the SKR using kubeconfig from Busola
• Add the 'keda' module in Busola, wait until it's ready
• Delete the 'keda' module using kubectl -n kyma-system edit kyma default

Environment Type

Managed

Environment Info

Kubernetes Version: x.y.z Docker Version: x.y.z Kyma CLI Version: x.y.z If used - k3d/k3s/minikube versions

Attachments

Dump from the command: kubectl -n kcp-system get --watch manifest 80ca5bc7-243d-4aae-a7c3-8f5792759cc2-keda-1095022605 -o yaml | tee watch.log

apiVersion: operator.kyma-project.io/v1beta2 kind: Manifest metadata: annotations: operator.kyma-project.io/fqdn: kyma-project.io/module/keda sync-oci-ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 creationTimestamp: "2024-07-09T09:05:56Z" finalizers:

• declarative.kyma-project.io/finalizer
• resource.kyma-project.io/finalizer generation: 1 labels: operator.kyma-project.io/channel: regular operator.kyma-project.io/controller-name: "" operator.kyma-project.io/kyma-name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 operator.kyma-project.io/managed-by: lifecycle-manager operator.kyma-project.io/module-name: keda name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2-keda-1095022605 namespace: kcp-system ownerReferences:
• apiVersion: operator.kyma-project.io/v1beta2 blockOwnerDeletion: true controller: true kind: Kyma name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 uid: b345e793-78c1-4dd6-8b82-7f2f04e0b437 resourceVersion: "6415641050" uid: 19bc3438-96b9-4d84-a117-912e2b0eec8e spec: install: name: raw-manifest source: name: kyma-project.io/module/keda ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 repo: europe-docker.pkg.dev/kyma-project/modules-internal/component-descriptors type: oci-ref remote: true resource: apiVersion: operator.kyma-project.io/v1alpha1 kind: Keda metadata: name: default namespace: kyma-system spec: logging: operator: level: info resources: metricServer: limits: cpu: 800m memory: 500Mi requests: cpu: 300m memory: 500Mi operator: limits: cpu: 800m memory: 200Mi requests: cpu: 300m memory: 150Mi version: 1.2.0 status: conditions:
• lastTransitionTime: "2024-07-09T09:09:48Z" message: resources are parsed and ready for use observedGeneration: 1 reason: ResourcesAvailable status: "True" type: Resources
• lastTransitionTime: "2024-07-09T09:12:36Z" message: installation is ready and resources can be used observedGeneration: 1 reason: Ready status: "True" type: Installation lastOperation: lastUpdateTime: "2024-07-09T09:12:36Z" operation: installation is ready and resources can be used state: Ready synced:
• group: apiextensions.k8s.io kind: CustomResourceDefinition name: kedas.operator.kyma-project.io namespace: "" version: v1
• group: "" kind: ServiceAccount name: keda-manager namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-leader-election-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-manager-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: RoleBinding name: keda-leader-election-rolebinding namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRoleBinding name: keda-manager-rolebinding namespace: "" version: v1
• group: "" kind: ConfigMap name: keda-kedas.operator.kyma-project.io namespace: kyma-system version: v1
• group: "" kind: ConfigMap name: keda-scaledobjects.keda.sh namespace: kyma-system version: v1
• group: scheduling.k8s.io kind: PriorityClass name: keda-priority-class namespace: "" version: v1

• group: apps kind: Deployment name: keda-manager namespace: kyma-system version: v1

  apiVersion: operator.kyma-project.io/v1beta2 kind: Manifest metadata: annotations: operator.kyma-project.io/fqdn: kyma-project.io/module/keda sync-oci-ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 creationTimestamp: "2024-07-09T09:05:56Z" finalizers:

• declarative.kyma-project.io/finalizer
• resource.kyma-project.io/finalizer generation: 1 labels: operator.kyma-project.io/channel: regular operator.kyma-project.io/controller-name: "" operator.kyma-project.io/kyma-name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 operator.kyma-project.io/managed-by: lifecycle-manager operator.kyma-project.io/module-name: keda name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2-keda-1095022605 namespace: kcp-system ownerReferences:
• apiVersion: operator.kyma-project.io/v1beta2 blockOwnerDeletion: true controller: true kind: Kyma name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 uid: b345e793-78c1-4dd6-8b82-7f2f04e0b437 resourceVersion: "6415651918" uid: 19bc3438-96b9-4d84-a117-912e2b0eec8e spec: install: name: raw-manifest source: name: kyma-project.io/module/keda ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 repo: europe-docker.pkg.dev/kyma-project/modules-internal/component-descriptors type: oci-ref remote: true resource: apiVersion: operator.kyma-project.io/v1alpha1 kind: Keda metadata: name: default namespace: kyma-system spec: logging: operator: level: info resources: metricServer: limits: cpu: 800m memory: 500Mi requests: cpu: 300m memory: 500Mi operator: limits: cpu: 800m memory: 200Mi requests: cpu: 300m memory: 150Mi version: 1.2.0 status: conditions:
• lastTransitionTime: "2024-07-09T09:09:48Z" message: resources are parsed and ready for use observedGeneration: 1 reason: ResourcesAvailable status: "True" type: Resources
• lastTransitionTime: "2024-07-09T09:12:36Z" message: installation is ready and resources can be used observedGeneration: 1 reason: Ready status: "True" type: Installation lastOperation: lastUpdateTime: "2024-07-09T09:12:36Z" operation: installation is ready and resources can be used state: Ready synced:
• group: apiextensions.k8s.io kind: CustomResourceDefinition name: kedas.operator.kyma-project.io namespace: "" version: v1
• group: "" kind: ServiceAccount name: keda-manager namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-leader-election-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-manager-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: RoleBinding name: keda-leader-election-rolebinding namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRoleBinding name: keda-manager-rolebinding namespace: "" version: v1
• group: "" kind: ConfigMap name: keda-kedas.operator.kyma-project.io namespace: kyma-system version: v1
• group: "" kind: ConfigMap name: keda-scaledobjects.keda.sh namespace: kyma-system version: v1
• group: scheduling.k8s.io kind: PriorityClass name: keda-priority-class namespace: "" version: v1

• group: apps kind: Deployment name: keda-manager namespace: kyma-system version: v1

  apiVersion: operator.kyma-project.io/v1beta2 kind: Manifest metadata: annotations: operator.kyma-project.io/fqdn: kyma-project.io/module/keda sync-oci-ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 creationTimestamp: "2024-07-09T09:05:56Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2024-07-09T09:40:02Z" finalizers:

• declarative.kyma-project.io/finalizer
• resource.kyma-project.io/finalizer generation: 2 labels: operator.kyma-project.io/channel: regular operator.kyma-project.io/controller-name: "" operator.kyma-project.io/kyma-name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 operator.kyma-project.io/managed-by: lifecycle-manager operator.kyma-project.io/module-name: keda name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2-keda-1095022605 namespace: kcp-system ownerReferences:
• apiVersion: operator.kyma-project.io/v1beta2 blockOwnerDeletion: true controller: true kind: Kyma name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 uid: b345e793-78c1-4dd6-8b82-7f2f04e0b437 resourceVersion: "6415657824" uid: 19bc3438-96b9-4d84-a117-912e2b0eec8e spec: install: name: raw-manifest source: name: kyma-project.io/module/keda ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 repo: europe-docker.pkg.dev/kyma-project/modules-internal/component-descriptors type: oci-ref remote: true resource: apiVersion: operator.kyma-project.io/v1alpha1 kind: Keda metadata: name: default namespace: kyma-system spec: logging: operator: level: info resources: metricServer: limits: cpu: 800m memory: 500Mi requests: cpu: 300m memory: 500Mi operator: limits: cpu: 800m memory: 200Mi requests: cpu: 300m memory: 150Mi version: 1.2.0 status: conditions:
• lastTransitionTime: "2024-07-09T09:09:48Z" message: resources are parsed and ready for use observedGeneration: 1 reason: ResourcesAvailable status: "True" type: Resources
• lastTransitionTime: "2024-07-09T09:12:36Z" message: installation is ready and resources can be used observedGeneration: 1 reason: Ready status: "True" type: Installation lastOperation: lastUpdateTime: "2024-07-09T09:12:36Z" operation: installation is ready and resources can be used state: Ready synced:
• group: apiextensions.k8s.io kind: CustomResourceDefinition name: kedas.operator.kyma-project.io namespace: "" version: v1
• group: "" kind: ServiceAccount name: keda-manager namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-leader-election-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-manager-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: RoleBinding name: keda-leader-election-rolebinding namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRoleBinding name: keda-manager-rolebinding namespace: "" version: v1
• group: "" kind: ConfigMap name: keda-kedas.operator.kyma-project.io namespace: kyma-system version: v1
• group: "" kind: ConfigMap name: keda-scaledobjects.keda.sh namespace: kyma-system version: v1
• group: scheduling.k8s.io kind: PriorityClass name: keda-priority-class namespace: "" version: v1

• group: apps kind: Deployment name: keda-manager namespace: kyma-system version: v1

  apiVersion: operator.kyma-project.io/v1beta2 kind: Manifest metadata: annotations: operator.kyma-project.io/fqdn: kyma-project.io/module/keda sync-oci-ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 creationTimestamp: "2024-07-09T09:05:56Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2024-07-09T09:40:02Z" finalizers:

• resource.kyma-project.io/finalizer generation: 2 labels: operator.kyma-project.io/channel: regular operator.kyma-project.io/controller-name: "" operator.kyma-project.io/kyma-name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 operator.kyma-project.io/managed-by: lifecycle-manager operator.kyma-project.io/module-name: keda name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2-keda-1095022605 namespace: kcp-system ownerReferences:
• apiVersion: operator.kyma-project.io/v1beta2 blockOwnerDeletion: true controller: true kind: Kyma name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 uid: b345e793-78c1-4dd6-8b82-7f2f04e0b437 resourceVersion: "6415661906" uid: 19bc3438-96b9-4d84-a117-912e2b0eec8e spec: install: name: raw-manifest source: name: kyma-project.io/module/keda ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 repo: europe-docker.pkg.dev/kyma-project/modules-internal/component-descriptors type: oci-ref remote: true resource: apiVersion: operator.kyma-project.io/v1alpha1 kind: Keda metadata: name: default namespace: kyma-system spec: logging: operator: level: info resources: metricServer: limits: cpu: 800m memory: 500Mi requests: cpu: 300m memory: 500Mi operator: limits: cpu: 800m memory: 200Mi requests: cpu: 300m memory: 150Mi version: 1.2.0 status: conditions:
• lastTransitionTime: "2024-07-09T09:09:48Z" message: resources are parsed and ready for use observedGeneration: 1 reason: ResourcesAvailable status: "True" type: Resources
• lastTransitionTime: "2024-07-09T09:12:36Z" message: installation is ready and resources can be used observedGeneration: 1 reason: Ready status: "True" type: Installation lastOperation: lastUpdateTime: "2024-07-09T09:12:36Z" operation: installation is ready and resources can be used state: Ready synced:
• group: apiextensions.k8s.io kind: CustomResourceDefinition name: kedas.operator.kyma-project.io namespace: "" version: v1
• group: "" kind: ServiceAccount name: keda-manager namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-leader-election-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-manager-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: RoleBinding name: keda-leader-election-rolebinding namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRoleBinding name: keda-manager-rolebinding namespace: "" version: v1
• group: "" kind: ConfigMap name: keda-kedas.operator.kyma-project.io namespace: kyma-system version: v1
• group: "" kind: ConfigMap name: keda-scaledobjects.keda.sh namespace: kyma-system version: v1
• group: scheduling.k8s.io kind: PriorityClass name: keda-priority-class namespace: "" version: v1

• group: apps kind: Deployment name: keda-manager namespace: kyma-system version: v1

  apiVersion: operator.kyma-project.io/v1beta2 kind: Manifest metadata: annotations: operator.kyma-project.io/fqdn: kyma-project.io/module/keda sync-oci-ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 creationTimestamp: "2024-07-09T09:05:56Z" deletionGracePeriodSeconds: 0 deletionTimestamp: "2024-07-09T09:40:02Z" finalizers:

• resource.kyma-project.io/finalizer generation: 2 labels: operator.kyma-project.io/channel: regular operator.kyma-project.io/controller-name: "" operator.kyma-project.io/kyma-name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 operator.kyma-project.io/managed-by: lifecycle-manager operator.kyma-project.io/module-name: keda name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2-keda-1095022605 namespace: kcp-system ownerReferences:
• apiVersion: operator.kyma-project.io/v1beta2 blockOwnerDeletion: true controller: true kind: Kyma name: 80ca5bc7-243d-4aae-a7c3-8f5792759cc2 uid: b345e793-78c1-4dd6-8b82-7f2f04e0b437 resourceVersion: "6415697429" uid: 19bc3438-96b9-4d84-a117-912e2b0eec8e spec: install: name: raw-manifest source: name: kyma-project.io/module/keda ref: sha256:5ce335c6c3a21d76ab32941cf771fcdc67ff149abe6a19b0ec9dbd0e2f7fe546 repo: europe-docker.pkg.dev/kyma-project/modules-internal/component-descriptors type: oci-ref remote: true resource: apiVersion: operator.kyma-project.io/v1alpha1 kind: Keda metadata: name: default namespace: kyma-system spec: logging: operator: level: info resources: metricServer: limits: cpu: 800m memory: 500Mi requests: cpu: 300m memory: 500Mi operator: limits: cpu: 800m memory: 200Mi requests: cpu: 300m memory: 150Mi version: 1.2.0 status: conditions:
• lastTransitionTime: "2024-07-09T09:09:48Z" message: resources are parsed and ready for use observedGeneration: 1 reason: ResourcesAvailable status: "True" type: Resources
• lastTransitionTime: "2024-07-09T09:12:36Z" message: installation is ready and resources can be used observedGeneration: 1 reason: Ready status: "True" type: Installation lastOperation: lastUpdateTime: "2024-07-09T09:12:36Z" operation: installation is ready and resources can be used state: Ready synced:
• group: apiextensions.k8s.io kind: CustomResourceDefinition name: kedas.operator.kyma-project.io namespace: "" version: v1
• group: "" kind: ServiceAccount name: keda-manager namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-leader-election-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: ClusterRole name: keda-manager-role namespace: "" version: v1
• group: rbac.authorization.k8s.io kind: RoleBinding name: keda-leader-election-rolebinding namespace: kyma-system version: v1
• group: rbac.authorization.k8s.io kind: ClusterRoleBinding name: keda-manager-rolebinding namespace: "" version: v1
• group: "" kind: ConfigMap name: keda-kedas.operator.kyma-project.io namespace: kyma-system version: v1
• group: "" kind: ConfigMap name: keda-scaledobjects.keda.sh namespace: kyma-system version: v1
• group: scheduling.k8s.io kind: PriorityClass name: keda-priority-class namespace: "" version: v1
• group: apps kind: Deployment name: keda-manager namespace: kyma-system version: v1

[Tomasz-Smelcerz-SAP]

I have verified that when all the modules are in Ready state, the module under deletion has Deleting state correctly set. It looks like if there are modules with Warning state in the Kyma installation, then deleting a Ready module succeeds, but the status of the module is never set to Deleting - it's just Ready until it's finally removed.

[janmedrek]

With the new 1.1.0 release this issue cannot be reproduced. We will reopen the ticket if that thing pops up again.