search

clastix / cluster-api-control-plane-provider-kamaji

The Kamaji Control Plane provider implementation of the Cluster Management API
Apache License 2.0
74 stars 25 forks source link

Refactoring KamajiControlPlane Ready status report #69

Closed prometherion closed 9 months ago

prometherion commented 9 months ago

Closes #66.

@jds9090 I tried to summarize the state transitions in the following YAML, let me know what do you think.

apiVersion: controlplane.cluster.x-k8s.io/v1alpha1
kind: KamajiControlPlane
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"controlplane.cluster.x-k8s.io/v1alpha1","kind":"KamajiControlPlane","metadata":{"annotations":{},"creationTimestamp":"2023-10-12T10:16:00Z","generation":2,"labels":{"cluster.x-k8s.io/cluster-name":"capi-quickstart"},"name":"capi-quickstart-kubevirt","namespace":"default","ownerReferences":[{"apiVersion":"cluster.x-k8s.io/v1beta1","blockOwnerDeletion":true,"controller":true,"kind":"Cluster","name":"capi-quickstart","uid":"913044ea-5135-4a61-a230-f07132d9ea04"}],"resourceVersion":"4317317","uid":"c8943729-c308-4a55-9630-6e5830a2b57a"},"spec":{"addons":{"coreDNS":{"dnsServiceIPs":["10.96.0.10"]},"kubeProxy":{}},"dataStoreName":"default","kubelet":{"cgroupfs":"systemd","preferredAddressTypes":["InternalIP","ExternalIP"]},"network":{"serviceType":"LoadBalancer"},"registry":"registry.k8s.io","replicas":2,"version":"1.23.10"},"status":{"externalManagedControlPlane":true,"initialized":true,"ready":true,"readyReplicas":2,"replicas":2,"selector":"cluster.x-k8s.io/cluster-name=capi-quickstart","unavailableReplicas":0,"updatedReplicas":2,"version":"v1.23.10"}}
  creationTimestamp: "2023-12-11T16:59:14Z"
  generation: 1
  labels:
    cluster.x-k8s.io/cluster-name: capi-quickstart
  name: capi-quickstart-kubevirt
  namespace: default
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: capi-quickstart
    uid: 913044ea-5135-4a61-a230-f07132d9ea04
  resourceVersion: "4334700"
  uid: d1765689-f8c4-42ff-9e50-4b4f59671211
spec:
  addons:
    coreDNS:
      dnsServiceIPs:
      - 10.96.0.10
    kubeProxy: {}
  dataStoreName: default
  kubelet:
    cgroupfs: systemd
    preferredAddressTypes:
    - InternalIP
    - ExternalIP
  network:
    serviceType: LoadBalancer
  registry: registry.k8s.io
  replicas: 2
  version: 1.23.10
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: Control Plane Endpoint is not yet available since unprocessed by Kamaji
    observedGeneration: 1
    reason: Pending
    status: "False"
    type: TenantcontrolPlaneAddressReady
  externalManagedControlPlane: true
  initialized: false
  ready: false
  readyReplicas: 0
  replicas: 0
  selector: ""
  unavailableReplicas: 0
  updatedReplicas: 0
  version: ""
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: Control Plane Endpoint is not yet available since unprocessed by Kamaji
    observedGeneration: 1
    reason: Pending
    status: "False"
    type: TenantcontrolPlaneAddressReady
  externalManagedControlPlane: true
  initialized: true
  ready: false
  readyReplicas: 0
  replicas: 0
  selector: ""
  unavailableReplicas: 0
  updatedReplicas: 0
  version: ""
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: Control Plane Endpoint is not yet available since unprocessed by Kamaji
    observedGeneration: 1
    reason: Pending
    status: "False"
    type: TenantcontrolPlaneAddressReady
  externalManagedControlPlane: true
  initialized: true
  ready: false
  readyReplicas: 0
  replicas: 0
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 0
  version: ""
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: admin kubeconfig still unprocessed by Kamaji, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KubeadmResourcesCreated
  externalManagedControlPlane: true
  initialized: true
  ready: false
  readyReplicas: 0
  replicas: 0
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 0
  version: ""
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KubeadmResourcesCreated
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: TenantControlPlane in Provisioning status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: false
  readyReplicas: 0
  replicas: 0
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 0
  version: ""
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KubeadmResourcesCreated
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: TenantControlPlane in Provisioning status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: false
  readyReplicas: 0
  replicas: 0
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 2
  updatedReplicas: 0
  version: ""
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KubeadmResourcesCreated
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: TenantControlPlane in Provisioning status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: false
  readyReplicas: 0
  replicas: 2
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 2
  updatedReplicas: 2
  version: ""
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KubeadmResourcesCreated
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: TenantControlPlane in Provisioning status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: false
  readyReplicas: 2
  replicas: 2
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 2
  version: v1.23.10
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KubeadmResourcesCreated
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: TenantControlPlane in Provisioning status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False"
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: true
  readyReplicas: 2
  replicas: 2
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 2
  version: v1.23.10
---
# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KubeadmResourcesCreated
  - lastTransitionTime: "2023-12-11T16:59:28Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: true
  readyReplicas: 2
  replicas: 2
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 2
  version: v1.23.10
jds9090 commented 9 months ago

Why don't you let the defer function update not only the condition but also the status of kamaji control plane(KCP)?

I think both ready Condition and ready Status should be updated at the same time.

Is there a reason for calling the updateKamajiControlPlaneStatus receiver inside the TrackConditionType function instead of letting the defer function update the status of the kamaji control plane(KCP)?

# REDACTED
status:
  conditions:
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantControlPlaneCreated
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: TenantcontrolPlaneAddressReady
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: InfrastructureClusterPatched
  - lastTransitionTime: "2023-12-11T16:59:14Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KamajiControlPlaneIsInitialized
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: ""
    observedGeneration: 1
    reason: Succeeded
    status: "True"
    type: KubeadmResourcesCreated
  - lastTransitionTime: "2023-12-11T16:59:16Z"
    message: TenantControlPlane in Provisioning status, enqueue back
    observedGeneration: 1
    reason: Failed
    status: "False" #  ready Condition and ready Status should be updated at the same time. 
    type: KamajiControlPlaneIsReady
  externalManagedControlPlane: true
  initialized: true
  ready: true #  ready Condition and ready Status should be updated at the same time. 
  readyReplicas: 2
  replicas: 2
  selector: cluster.x-k8s.io/cluster-name=capi-quickstart
  unavailableReplicas: 0
  updatedReplicas: 2
  version: v1.23.10
prometherion commented 9 months ago

Ready is unbounded from the required resources such as the CA and the Kubeconfig, I could have the secret resources created but the TCP not yet ready.

We're tricked by the fact Kamaji is extremely fast, in other circumstances it could tale time and we wait for the API Servers from being up and ready.

jds9090 commented 9 months ago

Ready is unbounded from the required resources such as the CA and the Kubeconfig, I could have the secret resources created but the TCP not yet ready.

We're tricked by the fact Kamaji is extremely fast, in other circumstances it could tale time and we wait for the API Servers from being up and ready.

Wouldn’t this comment be confusing?

type KamajiControlPlaneStatus struct {
    // The TenantControlPlane API Server is ready to receive requests.
    Ready bool `json:"ready"`
prometherion commented 9 months ago

@jds9090 that ready field is reported by Kamaji itself. From a Kamaji standpoint, if the Pods are up and running, it means the API Server is ready to receive requests, as well as the Control Plane from the CAPI domain.

Of course, if the network is not well configured, it would be out of the scope since we cannot check variables outside of our control, such as a not yet announced IP, NetworkPolicies, etc.

jds9090 commented 9 months ago

@jds9090 that ready field is reported by Kamaji itself. From a Kamaji standpoint, if the Pods are up and running, it means the API Server is ready to receive requests, as well as the Control Plane from the CAPI domain.

Of course, if the network is not well configured, it would be out of the scope since we cannot check variables outside of our control, such as a not yet announced IP, NetworkPolicies, etc.

Even when KCP(Kamaji Control Plane) is in ready state, Tenant control plane can be not ready to communicate.

That's why I think the comment is confusing.

My suggestion is as below //Kamaji control plane is ready to link CAPI with tenant control plane.

prometherion commented 9 months ago

I accept the suggestion, thanks! 🤗