search

IBM / cloud-operators

Provision and bind IBM Cloud services to your Kubernetes cluster in a Kubernetes-native way
Apache License 2.0
42 stars 33 forks source link

Deletion of CRs (service and binding) ends with an unstable state for the CRs and the IBM Cloud Operator #265

Closed thomassuedbroecker closed 2 years ago

thomassuedbroecker commented 2 years ago

Hi, today we figured out following:

Our objective was

We wanted to delete instances of the "bindings and service" by removing the CR instances inside a project/namespace, because we would like to delete the services on IBM Cloud and setup new configurations for our application in the cluster.

Steps to reproduce:

Question:

How to avoid this or even better how to fix that unstable status?

Steps to get a stable status again:

We tried to following steps to get a stable status again, but we had no success at the end.

  1. Deletion of the operator development project with the CRs failed (status terminating)
  2. Deletion of IBM Cloud Services has no impact to the IBM Cloud Operator
  3. We noticed that the IBM Cloud Operator restarts itself again and again
  4. Deinstall the IBM Cloud Operator has no impact to the deletion or CRs and the srojects
  5. Restart the master node of the OpenShift cluster
  6. Restart the work nodes of the OpenShift cluster

Conclusion at the moment

Don't delete CRDs for the IBM Cloud Operator and don't delete projects that contains CRDs of the IBM Cloud Operator! The deletion will break the IBM Cloud Operator and you need to install it again and maybe ends in an unstable state for projects at the cluster.

thomassuedbroecker commented 2 years ago

We fixed it with more memory for our cluster and a reinstallation of the Operator once more and a clean-up in IBM Cloud.

JohnStarich commented 2 years ago

Hey @thomassuedbroecker, thanks for posting your solution. Was the only issue memory pressure?

Deleting CR's should behave correctly, of course. 🙏

thomassuedbroecker commented 2 years ago

Hi @JohnStarich, we noticed: you should don't delete too fast too many IBM Cloud Operator CRDs , maybe that results in an unstable state and can be related to Writing a Kubernetes Operator: the Hard Parts - Sebastien Guilloux, Elastic .