Incorrect KamajiControlPlaneReady condition

jds9090 commented 10 months ago

Related to : #66

For the first time, I missed the important fact.

cluster controller also watches external object : https://github.com/kubernetes-sigs/cluster-api/blob/14efefeb46dbe8d0cd0f5b7d1718e00ec58fc079/internal/controllers/cluster/cluster_controller_phases.go#L96

I could check that cluster and kamaji control plane are inheriting the readiness from Kamaji.

But I think the kamajiControlPlane condition is not correct. It is always true during my test.

Steps to reproduce

deploy cluster by CAPI kamaji

delete deployment(control plane)

ubuntu@kamaji:~$ k delete deploy kamajicontrolplane-27
deployment.apps "kamajicontrolplane-27" deleted

check tenant control plane's status : Ready -> NotReady

kubectl get tcp -o=jsonpath={.items[0].status.kubernetesResources.version.status}

check kamaji control plane's status : true -> false

kubectl get kamajicontrolplane -o=jsonpath={.items[0].status.ready}

check kamaji control plane condition's KamajiControlPlaneReady : true -> true (not correct)

kubectl get kamajicontrolplane -o=jsonpath='{.items[0].status.conditions[4].type} {.items[0].status.conditions[4].status}'

check cluster's status : true -> false

kubectl get cluster -o=jsonpath={.items[0].status.controlPlaneReady}

about 50 seconds later

deployment(control plane) : Running

check tenant control plane's status : NotReady -> Ready

kubectl get tcp -o=jsonpath={.items[0].status.kubernetesResources.version.status}

check kamaji control plane's status : false -> true

kubectl get kamajicontrolplane -o=jsonpath={.items[0].status.ready}

check kamaji control plane condition's KamajiControlPlaneReady : true -> true (not correct)

kubectl get kamajicontrolplane -o=jsonpath='{.items[0].status.conditions[4].type} {.items[0].status.conditions[4].status}'

check cluster's status : false -> true

kubectl get cluster -o=jsonpath={.items[0].status.controlPlaneReady}

Additionally,

What still makes me confused is as below(@prometherion shared this).

# REDACTED
status:
  conditions:
  ...
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: TenantControlPlane in Provisioning status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False" #  KamajiControlPlaneIsReady condition is not ready
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: true # KamajiControlPlane's status is ready
  readyReplicas: 2
  replicas: 2
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 2
  version: v1.23.10

When KamajiControlPlane's status is ready, KamajiControlPlaneIsReady condition is not ready.

When can this situation happen?

Please, describe this situation :) @prometherion

prometherion commented 10 months ago

I tried to replicate your issue @jds9090 but with no luck.

By deleting the Deployment, the KamajiControlPlane is marked as non ready:

$: kubectl delete deployment capi-quickstart-kubevirt
deployment.apps "capi-quickstart-kubevirt" deleted

$: kubectl get kamajicontrolplane capi-quickstart-kubevirt -o='jsonpath={.status.ready}'
false

Also the condition seems reporting the correct status along with the error.

  - lastTransitionTime: "2023-12-28T16:13:30Z"
    message: TenantControlPlane in NotReady status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KamajiControlPlaneIsReady

Once the Deployment is created back, everything's ok.

$: kubectl get capi-quickstart-kubevirt
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
capi-quickstart-kubevirt   2/2     2            2           113s

$: kubectl get kamajicontrolplane capi-quickstart-kubevirt -o='jsonpath={.status.ready}'
true

  - lastTransitionTime: "2023-12-28T16:14:56Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsReady

jds9090 commented 10 months ago

I tried to replicate your issue @jds9090 but with no luck.

By deleting the Deployment, the KamajiControlPlane is marked as non ready:

$: kubectl delete deployment capi-quickstart-kubevirt
deployment.apps "capi-quickstart-kubevirt" deleted

$: kubectl get kamajicontrolplane capi-quickstart-kubevirt -o='jsonpath={.status.ready}'
false

Also the condition seems reporting the correct status along with the error.

  - lastTransitionTime: "2023-12-28T16:13:30Z"
    message: TenantControlPlane in NotReady status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KamajiControlPlaneIsReady

Once the Deployment is created back, everything's ok.

$: kubectl get capi-quickstart-kubevirt
NAME                       READY   UP-TO-DATE   AVAILABLE   AGE
capi-quickstart-kubevirt   2/2     2            2           113s

$: kubectl get kamajicontrolplane capi-quickstart-kubevirt -o='jsonpath={.status.ready}'
true

  - lastTransitionTime: "2023-12-28T16:14:56Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsReady

Which version of KamajiControlPlane were you using?

I used

cluster-api:v1.5.2
cluster-api-control-plane-provider-kamaji:v0.4.1
kamaji:v0.3.5

By deleting the Deployment, I could replicate the issue again.

ubuntu@kamaji:~/capi-watcher$ k delete deploy kamajicontrolplane-27 
deployment.apps "kamajicontrolplane-27" deleted

ubuntu@kamaji:~/capi-watcher$ k get kamajicontrolplane -o custom-columns="name":.metadata.name,"initialized":.status.initialized,"ready":.status.ready,"version":.status.version,"conditions":.status.conditions -w
name                    initialized   ready   version   conditions
kamajicontrolplane-27   true          true    v1.27.2   [map[lastTransitionTime:2023-12-26T06:40:22Z message: observedGeneration:1 reason:Succeeded status:True type:TenantControlPlaneCreated] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:TenantcontrolPlaneAddressReady] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:InfrastructureClusterPatched] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsInitialized] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsReady] map[lastTransitionTime:2023-12-26T06:41:42Z message: observedGeneration:1 reason:Succeeded status:True type:KubeadmResourcesCreated]]
kamajicontrolplane-27   true          false   v1.27.2   [map[lastTransitionTime:2023-12-26T06:40:22Z message: observedGeneration:1 reason:Succeeded status:True type:TenantControlPlaneCreated] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:TenantcontrolPlaneAddressReady] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:InfrastructureClusterPatched] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsInitialized] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsReady] map[lastTransitionTime:2023-12-26T06:41:42Z message: observedGeneration:1 reason:Succeeded status:True type:KubeadmResourcesCreated]]
kamajicontrolplane-27   true          false   v1.27.2   [map[lastTransitionTime:2023-12-26T06:40:22Z message: observedGeneration:1 reason:Succeeded status:True type:TenantControlPlaneCreated] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:TenantcontrolPlaneAddressReady] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:InfrastructureClusterPatched] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsInitialized] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsReady] map[lastTransitionTime:2023-12-26T06:41:42Z message: observedGeneration:1 reason:Succeeded status:True type:KubeadmResourcesCreated]]
kamajicontrolplane-27   true          false   v1.27.2   [map[lastTransitionTime:2023-12-26T06:40:22Z message: observedGeneration:1 reason:Succeeded status:True type:TenantControlPlaneCreated] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:TenantcontrolPlaneAddressReady] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:InfrastructureClusterPatched] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsInitialized] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsReady] map[lastTransitionTime:2023-12-26T06:41:42Z message: observedGeneration:1 reason:Succeeded status:True type:KubeadmResourcesCreated]]
kamajicontrolplane-27   true          false   v1.27.2   [map[lastTransitionTime:2023-12-26T06:40:22Z message: observedGeneration:1 reason:Succeeded status:True type:TenantControlPlaneCreated] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:TenantcontrolPlaneAddressReady] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:InfrastructureClusterPatched] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsInitialized] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsReady] map[lastTransitionTime:2023-12-26T06:41:42Z message: observedGeneration:1 reason:Succeeded status:True type:KubeadmResourcesCreated]]
kamajicontrolplane-27   true          true    v1.27.2   [map[lastTransitionTime:2023-12-26T06:40:22Z message: observedGeneration:1 reason:Succeeded status:True type:TenantControlPlaneCreated] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:TenantcontrolPlaneAddressReady] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:InfrastructureClusterPatched] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsInitialized] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsReady] map[lastTransitionTime:2023-12-26T06:41:42Z message: observedGeneration:1 reason:Succeeded status:True type:KubeadmResourcesCreated]]
kamajicontrolplane-27   true          true    v1.27.2   [map[lastTransitionTime:2023-12-26T06:40:22Z message: observedGeneration:1 reason:Succeeded status:True type:TenantControlPlaneCreated] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:TenantcontrolPlaneAddressReady] map[lastTransitionTime:2023-12-26T06:40:23Z message: observedGeneration:1 reason:Succeeded status:True type:InfrastructureClusterPatched] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsInitialized] map[lastTransitionTime:2023-12-26T06:40:32Z message: observedGeneration:1 reason:Succeeded status:True type:KamajiControlPlaneIsReady] map[lastTransitionTime:2023-12-26T06:41:42Z message: observedGeneration:1 reason:Succeeded status:True type:KubeadmResourcesCreated]]

ubuntu@kamaji:~/capi-watcher$ k get cluster -o custom-columns="name":.metadata.name,"controlPlaneReady":.status.controlPlaneReady,"infrastructureReady":.status.infrastructureReady,"phase":.status.phase -w
name             controlPlaneReady   infrastructureReady   phase
capi-kamaji-27   true                true                  Provisioned
capi-kamaji-27   true                true                  Provisioned
capi-kamaji-27   false               true                  Provisioned
capi-kamaji-27   false               true                  Provisioned
capi-kamaji-27   true                true                  Provisioned

prometherion commented 10 months ago

Which version of KamajiControlPlane were you using?

I was running from master branch. Now that you let me think about it, I think aed438a is doing the trick. If you agree, I would cut out a new version with the improvements and hotfixes collected, if the problem persists we could open the issue back.

Deal?

jds9090 commented 10 months ago

Which version of KamajiControlPlane were you using?

I was running from master branch. Now that you let me think about it, I think aed438a is doing the trick. If you agree, I would cut out a new version with the improvements and hotfixes collected, if the problem persists we could open the issue back.

Deal?

I think that would be a good idea.

prometherion commented 10 months ago

v0.4.2 has been released, please, give it a try and feel free to open this issue back if the problem persists.

clastix / cluster-api-control-plane-provider-kamaji

Incorrect KamajiControlPlaneReady condition #71