[BUG] Rancher deletes Cluster API Cluster resources that it did not create

[tmmorin]

Rancher Server Setup

• Rancher version: 2.6.9
• Installation option: Helm chart on a kubeadm cluster
• Proxy/Cert Details:

Information about the Cluster

• Kubernetes version:
• Cluster Type (Local/Downstream): local

User Information

Admin User

Describe the bug

When one deteles a Cluster.provisioning.cattle.io resource, if there is a Cluster.cluster.x-k8s.io resource (that a user may have created in the context of a use of ClusterAPI) with the same name, Rancher will delete that one as well, merely based on name matching only and despite the fact that Rancher has no relationship with this resource.

To Reproduce

• create a Cluster.cluster.x-k8s.io resource with name foo in a namespace
• create a Cluster.provisioning.cattle.io resource with name foo in a namespace
• delete this Cluster.provisioning.cattle.io resource

Result

The Cluster.cluster.x-k8s.io resource is deleted.

Expected Result

The Cluster.cluster.x-k8s.io resource should not be deleted.

[tmmorin]

ping @richardcase, as a follow up to our discussions

[richardcase]

In the remove handler for Cluster.provisioning.cattle.io if a CAPI Cluster (Cluster.cluster.x-k8s.io) exists with the same name & namespace then it is deleted without checking ownership reference, labels/annotations:

https://github.com/rancher/rancher/blob/release/v2.7/pkg/controllers/provisioningv2/cluster/remove.go#L95:L106

[richardcase]

@tmmorin - also to confirm you use embedded-cluster-api=false to disable the embedded CAPI controllers? (because you are using the standard CAPI controllers/providers)

[tmmorin]

@tmmorin - also to confirm you use embedded-cluster-api=false to disable the embedded CAPI controllers? (because you are using the standard CAPI controllers/providers)

yes, indeed

we install rancher with the official Helm chart, with this in the values:

features: embedded-cluster-api=false,provisioningv2=true

[richardcase]

Internal reference: SURE-5768

[kkaempf]

https://github.com/rancher/rancher/pull/40714

[sowmyav27]

moving it to Review state since the linked PR ^ is not merged.

[richardcase]

@cpinjani - these are the instructions i used.

• Check if you have a docker network called kind

docker network ls

• If its not listed (i.e. you haven't used kind on the machine before) run the following:

docker network create \
      --driver=bridge \
      --subnet=172.19.0.0/16 \
      --gateway=172.19.0.1 \
      --opt "com.docker.network.bridge.enable_ip_masquerade"="true" \
      --opt "com.docker.network.driver.mtu"="1500" \
      kind

• create a file called k3d-capd.yaml to configure k3d for CAPD

apiVersion: k3d.io/v1alpha5
kind: Simple 
metadata:
    name: rancher-test
servers: 1
agents: 0
image: rancher/k3s:v1.25.9-k3s1
network: kind
volumes:
    - volume: /var/run/docker.sock:/var/run/docker.sock
      nodeFilters:
        - server:*

• Create the k3d management cluster

k3d cluster create --config k3d-capd.yaml

• Install cert manager

kubectl apply -f https://github.com/jetstack/cert-manager/releases/download/v1.5.1/cert-manager.crds.yaml
helm install cert-manager jetstack/cert-manager --namespace cert-manager --create-namespace --version v1.5.1

• Install CAPI with the docker provider (**If this is done after Rancher, then Rancher fails to start)

export EXP_CLUSTER_RESOURCE_SET=true
clusterctl init -i docker

• Install Rancher with the embedded capi feature disabled

kubectl create namespace cattle-system
helm install rancher rancher-stable/rancher --namespace cattle-system --set bootstrapPassword=admin --set replicas=1  --set hostname=tunnel.fruitcase.dev --set 'extraEnv[0].name=CATTLE_FEATURES' --set 'extraEnv[0].value=embedded-cluster-api=false' --set global.cattle.psp.enabled=false --version 2.7.3

• Do whatever you need to do to expose your cluster (i.e. ngrok, inlets)
• Create the capi cluster definition (note the name is test1)

export CONTROL_PLANE_MACHINE_COUNT=1
export WORKER_MACHINE_COUNT=1
export KUBERNETES_VERSION=v1.26.4

clusterctl generate cluster test1 --target-namespace fleet-default --from https://github.com/capi-samples/opensuse-23/blob/main/templates/kubeadm-docker.yaml > cluster.yaml

• Apply the cluster to capi/rancher cluster:

kubectl apply -f cluster.yaml

• Wait for the control plane machine to be ready

kubectl get machines -A -w

• Import the CAPI cluster into Rancher
  • In Rancher Manager go to Cluster Management
  • Import Cluster with type Generic and set the name to test1
• Delete the cluster from Rancher:
  • In Rancher Manager go to Cluster Management
  • Tick the box on the left to select test1 cluster
  • Click Delete button
• Observe if the test1 CAPI cluster has been deleted or not:

kubectl get clusters.cluster.x-k8s.io -A

• If the fix works it should not delete the CAPI cluster

[cpinjani]

@richardcase Able to reproduce with above steps on reported version 2.6.9, will proceed with validation on fixed version.

Imported CAPI cluster into rancher:

$ kubectl get clusters.cluster.x-k8s.io -A
NAMESPACE       NAME    PHASE         AGE   VERSION
fleet-default   test1   Provisioned   14m   

$ kubectl get Cluster.provisioning.cattle.io -A
NAMESPACE       NAME    READY   KUBECONFIG
fleet-local     local   true    local-kubeconfig
fleet-default   test1   true    test1-kubeconfig

After deleting imported cluster in Rancher:

$ kubectl get Cluster.provisioning.cattle.io -A
NAMESPACE     NAME    READY   KUBECONFIG
fleet-local   local   true    local-kubeconfig

$ kubectl get clusters.cluster.x-k8s.io -A
No resources found

[cpinjani]

As the fix is reverted, moving back to "In-progress" cc: @richardcase

[kkaempf]

too late for 2.7.5, moving to Q3

[richardcase]

We fixed the original issue but due to a change elsewhere we were unable to verify it. This was due to another issue with the 2 resources named the same related to the kubeconfig secret.

We need to take into account both aspects into account when fixing this.

[cpinjani]

Validation results on v2.8-ef234b38637b8ab5c73f81e2104124e47ae71bd2-head Unable to reproduce this issue. @alexander-demicev Please review the issue for closure.

Cluster.provisioning.cattle.io resource name and display names are now different. For eg. Display name on Rancher UI: cluster1 Resource name:

$ kubectl get Cluster.provisioning.cattle.io -A
NAMESPACE       NAME      READY   KUBECONFIG
fleet-default   c-wm4qd   true    c-wm4qd-kubeconfig
fleet-local     local     true    local-kubeconfig

Steps

• Create a Cluster.provisioning.cattle.io resource with name foo in a namespace
• Create a Cluster.cluster.x-k8s.io resource with name foo in a namespace
• Delete this Cluster.provisioning.cattle.io resource

Results

• Create a Cluster.cluster.x-k8s.io resource with name same as Cluster.provisioning.cattle.io resource name

  
  $ kubectl get Cluster.provisioning.cattle.io -A
  NAMESPACE       NAME      READY   KUBECONFIG
  fleet-default   c-67xdp   true    c-67xdp-kubeconfig
  fleet-local     local     true    local-kubeconfig

$ kubectl get clusters.cluster.x-k8s.io NAME PHASE AGE VERSION c-67xdp Provisioned 3m46s

- Import CAPI cluster into Rancher, importing using resource name not allowed
![image](https://github.com/rancher/rancher/assets/73099870/b898d60b-de20-4cb1-bb46-db2dec67ebdb)

- Import using display name

$ kubectl get Cluster.provisioning.cattle.io -A NAMESPACE NAME READY KUBECONFIG fleet-default c-67xdp true c-67xdp-kubeconfig fleet-default cluster1 true cluster1-kubeconfig fleet-local local true local-kubeconfig

- Delete `Cluster.provisioning.cattle.io` resource
CAPI cluster is intact:

$ kubectl get clusters.cluster.x-k8s.io NAME PHASE AGE VERSION c-67xdp Provisioned 8m43s

[salasberryfin]

As of today, we can no longer reproduce this and consider it solved.