Issue while deploying 2nd + n cluster in the same VPC

[Wolfslicht]

Description When deploying new clusters in the Open Telekom Cloud Kontainer Engine via Rancher Manager in the eu-ch2 region, the initial cluster is created successfully. However, any new cluster creation attempts with the same base configuration (with variations in Container CIDR and node count) result in failure, even after the complete deletion of the first deployment's resources. This issue occurs despite each deployment being initiated as a new creation rather than as an extension or modification of the existing cluster.

Expected Behavior Each new cluster creation attempt should be treated independently, allowing for multiple clusters to be deployed using similar configurations without interference or conflict from previous deployments.

Current Behavior The first cluster is deployed successfully with no issues. All subsequent new cluster creation attempts fail, even with minor configuration changes and after ensuring no resources from previous deployments remain.

Steps to Reproduce Successfully deploy a first cluster in the OTC Kontainer Engine using Rancher Manager with a specific configuration. Delete all resources associated with the first cluster upon successful deployment. Attempt to create a new cluster with the same base configuration but varying the Container CIDR and node count. Observe the failure in the deployment process for the new cluster.

Error Message The deployments fail with a 409 Conflict HTTP response code, indicating a duplicate resource or conflict, despite the previous resources being deleted and the new deployment having unique parameters.

Failed to create cluster: error creating OpenTelekomCloud cluster: Expected HTTP response code [201] when accessing [POST https://cce.eu-ch2.sc.otc.t-systems.com/api/v3/projects/16fcfca67044442cb171c44e5f7d2700/clusters], but got 409 instead {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","code":409,"errorCode":"CCE.01409001","errorMessage":"The resource already exists.","error_code":"CCE_CM.0202","error_msg":"Resource has exist","message":"Duplicate cluster name","reason":"Conflict"}

Environment Region: eu-ch2 Rancher Version: v2.8.2 Dashboard Version: v2.8.0 Helm Version: v2.16.8-rancher2 Machine Version: v0.15.0-rancher106 OTC Kontainer Engine Version: 1.1.1 OTC Kontainer Engine UI Version: 1.20

Additional Context The issue seems to suggest a deeper systemic problem, possibly related to how the OTC Kontainer Engine or Rancher Manager manages the lifecycle or tracking of cluster deployments. This problem persists even when deploying entirely new clusters with distinct configurations and after ensuring that no remnants of previous deployments exist.

Possible Cause This could indicate an issue with the internal handling or tracking of deployments by the OTC Kontainer Engine or Rancher Manager, which might not be fully resetting or acknowledging the deletion of previous deployments.

Requested Information Insights into handling or tracking deployments and any potential caching or state preservation that could interfere with new cluster creations. Any guidelines or best practices for ensuring a clean state for new deployments in the OTC Kontainer Engine, especially after deleting previous clusters.

Screenshots: UI of CCE VPC 1 This is informational that this also happens in other Tenants/Projects, even when the original first one is deleted

UI of CCE VPC 1 Operations Record: This is informational that this also happens in other Tenants/Projects, even when the original first one is deleted

UI of CCE VPC new + Operations Record: You find the API View in the lower section, the Project, VPC and Subnet were completely new created

UI of Rancher Manager:

API yaml First Cluster "new VPC": In Rancher UI named: cce-test-new-project-test1 in the OTC UI named: c-tpv5z

{
        "id": "fleet-default/**c-tpv5z**",
        "type": "provisioning.cattle.io.cluster",
        "links": {
        "remove": "…/v1/provisioning.cattle.io.clusters/fleet-default/c-tpv5z",
        "self": "…/v1/provisioning.cattle.io.clusters/fleet-default/c-tpv5z",
        "update": "…/v1/provisioning.cattle.io.clusters/fleet-default/c-tpv5z",
        "view": "…/apis/provisioning.cattle.io/v1/namespaces/fleet-default/clusters/c-tpv5z"
        },
        "apiVersion": "provisioning.cattle.io/v1",
        "kind": "Cluster",
        "metadata": {
        "annotations": {
        "field.cattle.io/creatorId": "system:serviceaccount:cattle-system:rancher",
        "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4yQTU7DMBCFr4JmnZRAlKS1xAIhxIItF5jYk9Zgjy17EiSq3B25dFEJ8bO09L3nb94RPAkaFAR1BGQOgmID5/IM4ytpySSbZMNGo4ijjQ3X1oCCmMJisw1seV9rN2ehVOtEKATVj9HwzpTq/fIGCjwy7skTywWwtNXVs2Vz9/BV+WcXoydQoGuJS/fxLzxH1CUDawUnYxv4xXrKgj6C4tm5ChyO5H6d4YD5AAq27aR3SDvsdN8PzTjQLU7ddmj6cTtQ1920Zje0TV9++yZ7qTM5IqkNTTg7KXSOpIuACxrdeY/7WQ6PbGKwLKCOa8EEZT6bZkoLmSdiSqezQDXr+hkAAP//Pd2rAuUBAAA",
        "objectset.rio.cattle.io/id": "provisioning-cluster-create",
        "objectset.rio.cattle.io/owner-gvk": "management.cattle.io/v3, Kind=Cluster",
        "objectset.rio.cattle.io/owner-name": "c-tpv5z",
        "objectset.rio.cattle.io/owner-namespace": ""
        },
        "creationTimestamp": "2024-02-23T08:26:21Z",
        "fields": [ 3 items
        "c-tpv5z",
        "true",
        "c-tpv5z-kubeconfig"
        ],
        "finalizers": [ 3 items
        "wrangler.cattle.io/cloud-config-secret-remover",
        "wrangler.cattle.io/provisioning-cluster-remove",
        "wrangler.cattle.io/rke-cluster-remove"
        ],
        "generation": 1,
        "labels": {
        "objectset.rio.cattle.io/hash": "83fc9ae9a5c6670b7e2af58706b87e5513d97306"
        },
        "managedFields": [ 3 items
        {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
        "f:metadata": {
        "f:annotations": {
        ".": { },
        "f:objectset.rio.cattle.io/applied": { },
        "f:objectset.rio.cattle.io/id": { },
        "f:objectset.rio.cattle.io/owner-gvk": { },
        "f:objectset.rio.cattle.io/owner-name": { },
        "f:objectset.rio.cattle.io/owner-namespace": { }
        },
        "f:finalizers": {
        "v:\"wrangler.cattle.io/provisioning-cluster-remove\"": { },
        "v:\"wrangler.cattle.io/rke-cluster-remove\"": { }
        },
        "f:labels": {
        ".": { },
        "f:objectset.rio.cattle.io/hash": { }
        }
        },
        "f:spec": {
        ".": { },
        "f:localClusterAuthEndpoint": { }
        }
        },
        "manager": "rancher",
        "operation": "Update",
        "time": "2024-02-23T08:26:21Z"
        },
        {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
        "f:metadata": {
        "f:finalizers": {
        ".": { },
        "v:\"wrangler.cattle.io/cloud-config-secret-remover\"": { }
        }
        }
        },
        "manager": "rancher-v2.8.2-secret-migrator",
        "operation": "Update",
        "time": "2024-02-23T08:26:21Z"
        },
        {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
        "f:status": {
        ".": { },
        "f:agentDeployed": { },
        "f:clientSecretName": { },
        "f:clusterName": { },
        "f:conditions": { },
        "f:observedGeneration": { },
        "f:ready": { }
        }
        },
        "manager": "rancher",
        "operation": "Update",
        "subresource": "status",
        "time": "2024-02-23T08:27:33Z"
        }
        ],
        "name": "c-tpv5z",
        "namespace": "fleet-default",
        "relationships": [ 12 items
        {
        "fromId": "c-tpv5z",
        "fromType": "management.cattle.io.cluster",
        "rel": "applies",
        "state": "active",
        "message": "Resource is Ready"
        },
        {
        "toId": "fleet-default/crt-c-tpv5z-nodes-view",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/c-tpv5z-kubeconfig",
        "toType": "secret",
        "rel": "owner",
        "state": "active",
        "message": "Resource is always ready"
        },
        {
        "toId": "fleet-default/c-tpv5z",
        "toType": "fleet.cattle.io.cluster",
        "rel": "applies",
        "state": "active",
        "message": "Resource is Ready"
        },
        {
        "toId": "fleet-default/r-cluster-c-tpv5z-view-p-msbfw-creator-project-owner-tdal5glrwq",
        "toType": "rbac.authorization.k8s.io.rolebinding",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/r-cluster-c-tpv5z-view-p-hp26q-creator-project-owner-tdal5glrwq",
        "toType": "rbac.authorization.k8s.io.rolebinding",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "c-tpv5z/c-tpv5z-fleet-default-owner",
        "toType": "management.cattle.io.clusterroletemplatebinding",
        "rel": "applies",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-tpv5z-cluster-owner",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/r-cluster-c-tpv5z-view",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-tpv5z-nodes-manage",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-tpv5z-cluster-admin",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-tpv5z-cluster-member",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        }
        ],
        "resourceVersion": "3440280",
        "state": {
        "error": false,
        "message": "Resource is Ready",
        "name": "active",
        "transitioning": false
        },
        "uid": "c6722407-db63-4aec-a19b-8bcd47e1f30f"
        },
        "spec": {
        "localClusterAuthEndpoint": { }
        },
        "status": {
        "agentDeployed": true,
        "clientSecretName": "c-tpv5z-kubeconfig",
        "clusterName": "c-tpv5z",
        "conditions": [ 23 items
        {
        "error": false,
        "lastUpdateTime": "",
        "status": "True",
        "transitioning": false,
        "type": "Pending"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:27:32Z",
        "status": "True",
        "transitioning": false,
        "type": "Waiting"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:15:26Z",
        "status": "True",
        "transitioning": false,
        "type": "BackingNamespaceCreated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:15:26Z",
        "status": "True",
        "transitioning": false,
        "type": "DefaultProjectCreated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:15:26Z",
        "status": "True",
        "transitioning": false,
        "type": "SystemProjectCreated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:15:26Z",
        "status": "True",
        "transitioning": false,
        "type": "InitialRolesPopulated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:15:32Z",
        "status": "True",
        "transitioning": false,
        "type": "CreatorMadeOwner"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:27:24Z",
        "status": "True",
        "transitioning": false,
        "type": "Connected"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:27:32Z",
        "status": "True",
        "transitioning": false,
        "type": "Ready"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:25:50Z",
        "status": "True",
        "transitioning": false,
        "type": "ServiceAccountMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:25:55Z",
        "status": "True",
        "transitioning": false,
        "type": "NoDiskPressure"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:25:55Z",
        "status": "True",
        "transitioning": false,
        "type": "NoMemoryPressure"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:00Z",
        "status": "True",
        "transitioning": false,
        "type": "SecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:00Z",
        "status": "True",
        "transitioning": false,
        "type": "ServiceAccountSecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:00Z",
        "status": "True",
        "transitioning": false,
        "type": "RKESecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:00Z",
        "status": "True",
        "transitioning": false,
        "type": "ACISecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:05Z",
        "status": "True",
        "transitioning": false,
        "type": "SystemAccountCreated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:07Z",
        "status": "True",
        "transitioning": false,
        "type": "AgentDeployed"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:17Z",
        "status": "True",
        "transitioning": false,
        "type": "GlobalAdminsSynced"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:27:33Z",
        "status": "False",
        "transitioning": false,
        "type": "Reconciling"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:22Z",
        "status": "False",
        "transitioning": false,
        "type": "Stalled"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:27:33Z",
        "status": "True",
        "transitioning": false,
        "type": "Created"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T08:26:22Z",
        "status": "True",
        "transitioning": false,
        "type": "RKECluster"
        }
        ],
        "observedGeneration": 1,
        "ready": true
        }
}

API yaml Second Cluster "new VPC": In Rancher UI named: cce-test-new-project-test2 in the OTC UI named: c-b75k5

{
        "id": "fleet-default/**c-b75k5**",
        "type": "provisioning.cattle.io.cluster",
        "links": {
        "remove": "…/v1/provisioning.cattle.io.clusters/fleet-default/c-b75k5",
        "self": "…/v1/provisioning.cattle.io.clusters/fleet-default/c-b75k5",
        "update": "…/v1/provisioning.cattle.io.clusters/fleet-default/c-b75k5",
        "view": "…/apis/provisioning.cattle.io/v1/namespaces/fleet-default/clusters/c-b75k5"
        },
        "apiVersion": "provisioning.cattle.io/v1",
        "kind": "Cluster",
        "metadata": {
        "annotations": {
        "field.cattle.io/creatorId": "system:serviceaccount:cattle-system:rancher",
        "objectset.rio.cattle.io/applied": "H4sIAAAAAAAA/4yQTU7DMBCFr4JmnZSENDS1xAIhxIItFxjbk9bUGUf2JCyi3B25dFEJ8bO09L3nb94CAwlaFAS1ADIHQXGBU34G/U5GEskmurAxKOJp48Kts6BgjGF2yQV2fCiNn5JQLE0kFILix2j4YIrlYT6BggEZDzQQyxUwN8XNq2P78PRV+WcX40CgwJR6157af+FpRJMzsBZwNnaB39xASXAYQfHkfQEeNflfZzhiOoKCpupIb3d1p6tt3WLXk26bHrt6f9fcm9Z2ut9hXe3zb99kr3V6TySlpR4nL5lOI5ks4INBf9njcZLjM9sxOBZQy5oxQZkuponiTPaFmOL5LFDVun4GAAD//+dKS13lAQAA",
        "objectset.rio.cattle.io/id": "provisioning-cluster-create",
        "objectset.rio.cattle.io/owner-gvk": "management.cattle.io/v3, Kind=Cluster",
        "objectset.rio.cattle.io/owner-name": "c-b75k5",
        "objectset.rio.cattle.io/owner-namespace": ""
        },
        "creationTimestamp": "2024-02-23T11:45:12Z",
        "fields": [ 3 items
        "c-b75k5",
        null,
        null
        ],
        "finalizers": [ 3 items
        "wrangler.cattle.io/cloud-config-secret-remover",
        "wrangler.cattle.io/provisioning-cluster-remove",
        "wrangler.cattle.io/rke-cluster-remove"
        ],
        "generation": 1,
        "labels": {
        "objectset.rio.cattle.io/hash": "308eb4718b0415a8feb53fa819236c5d8bf7a109"
        },
        "managedFields": [ 3 items
        {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
        "f:metadata": {
        "f:annotations": {
        ".": { },
        "f:objectset.rio.cattle.io/applied": { },
        "f:objectset.rio.cattle.io/id": { },
        "f:objectset.rio.cattle.io/owner-gvk": { },
        "f:objectset.rio.cattle.io/owner-name": { },
        "f:objectset.rio.cattle.io/owner-namespace": { }
        },
        "f:finalizers": {
        "v:\"wrangler.cattle.io/provisioning-cluster-remove\"": { },
        "v:\"wrangler.cattle.io/rke-cluster-remove\"": { }
        },
        "f:labels": {
        ".": { },
        "f:objectset.rio.cattle.io/hash": { }
        }
        },
        "f:spec": {
        ".": { },
        "f:localClusterAuthEndpoint": { }
        }
        },
        "manager": "rancher",
        "operation": "Update",
        "time": "2024-02-23T11:45:12Z"
        },
        {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
        "f:status": {
        ".": { },
        "f:clusterName": { },
        "f:conditions": { },
        "f:observedGeneration": { }
        }
        },
        "manager": "rancher",
        "operation": "Update",
        "subresource": "status",
        "time": "2024-02-23T11:45:12Z"
        },
        {
        "apiVersion": "provisioning.cattle.io/v1",
        "fieldsType": "FieldsV1",
        "fieldsV1": {
        "f:metadata": {
        "f:finalizers": {
        ".": { },
        "v:\"wrangler.cattle.io/cloud-config-secret-remover\"": { }
        }
        }
        },
        "manager": "rancher-v2.8.2-secret-migrator",
        "operation": "Update",
        "time": "2024-02-23T11:45:12Z"
        }
        ],
        "name": "c-b75k5",
        "namespace": "fleet-default",
        "relationships": [ 9 items
        {
        "fromId": "c-b75k5",
        "fromType": "management.cattle.io.cluster",
        "rel": "applies",
        "state": "provisioning",
        "message": "failed to create cluster: error creating OpenTelekomCloud cluster: Expected HTTP response code [201] when accessing [POST https://cce.eu-ch2.sc.otc.t-systems.com/api/v3/projects/4e59a102ff5849b4a27f8327d58d4f20/clusters], but got 409 instead\n{\"kind\":\"Status\",\"apiVersion\":\"v1\",\"metadata\":{},\"status\":\"Failure\",\"code\":409,\"errorCode\":\"CCE.01409001\",\"errorMessage\":\"The resource already exists.\",\"error_code\":\"CCE_CM.0202\",\"error_msg\":\"Resource has exist\",\"message\":\"Duplicate cluster name\",\"reason\":\"Conflict\"}",
        "error": true
        },
        {
        "toId": "fleet-default/r-cluster-c-b75k5-view",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-b75k5-nodes-manage",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-b75k5-cluster-admin",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-b75k5-nodes-view",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-b75k5-cluster-member",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/r-cluster-c-b75k5-view-p-2qxvj-creator-project-owner-tdal5glrwq",
        "toType": "rbac.authorization.k8s.io.rolebinding",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/r-cluster-c-b75k5-view-p-h7kcx-creator-project-owner-tdal5glrwq",
        "toType": "rbac.authorization.k8s.io.rolebinding",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        },
        {
        "toId": "fleet-default/crt-c-b75k5-cluster-owner",
        "toType": "rbac.authorization.k8s.io.role",
        "rel": "owner",
        "state": "active",
        "message": "Resource is current"
        }
        ],
        "resourceVersion": "3564857",
        "state": {
        "error": false,
        "message": "",
        "name": "waiting",
        "transitioning": true
        },
        "uid": "cfe598e9-16a0-4aca-9075-8c8aa5302a57"
        },
        "spec": {
        "localClusterAuthEndpoint": { }
        },
        "status": {
        "clusterName": "c-b75k5",
        "conditions": [ 18 items
        {
        "error": false,
        "lastUpdateTime": "",
        "status": "True",
        "transitioning": false,
        "type": "Pending"
        },
        {
        "error": false,
        "lastUpdateTime": "",
        "status": "Unknown",
        "transitioning": true,
        "type": "Waiting"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:37:18Z",
        "status": "True",
        "transitioning": false,
        "type": "BackingNamespaceCreated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:37:18Z",
        "status": "True",
        "transitioning": false,
        "type": "DefaultProjectCreated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:37:18Z",
        "status": "True",
        "transitioning": false,
        "type": "SystemProjectCreated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:37:18Z",
        "status": "True",
        "transitioning": false,
        "type": "InitialRolesPopulated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:37:23Z",
        "status": "True",
        "transitioning": false,
        "type": "CreatorMadeOwner"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:37:24Z",
        "status": "False",
        "transitioning": false,
        "type": "Connected"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:44:47Z",
        "status": "True",
        "transitioning": false,
        "type": "NoDiskPressure"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:44:47Z",
        "status": "True",
        "transitioning": false,
        "type": "NoMemoryPressure"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:44:53Z",
        "status": "True",
        "transitioning": false,
        "type": "SecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:44:53Z",
        "status": "True",
        "transitioning": false,
        "type": "ServiceAccountSecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:44:53Z",
        "status": "True",
        "transitioning": false,
        "type": "RKESecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:44:53Z",
        "status": "True",
        "transitioning": false,
        "type": "ACISecretsMigrated"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:45:13Z",
        "reason": "Reconciling",
        "status": "True",
        "transitioning": true,
        "type": "Reconciling"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:45:13Z",
        "status": "False",
        "transitioning": false,
        "type": "Stalled"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:45:13Z",
        "status": "True",
        "transitioning": false,
        "type": "Created"
        },
        {
        "error": false,
        "lastUpdateTime": "2024-02-23T11:45:13Z",
        "status": "True",
        "transitioning": false,
        "type": "RKECluster"
        }
        ],
        "observedGeneration": 1
        }
}

@Nils-Magnus as bespoken.

[artem-lifshits]

Hello @Wolfslicht have you provided SATA disk option to cluster config by any chance? That's the only way I was able to reproduce the issue.

In short SATA (common) disks are not available on Swiss so please try to change your node disk type to SAS or SSD.

While documentation states that SATA disks are available in reality the API response for SATA cluster nodes creation results to 400 error with following response:

{
    "kind": "Status",
    "apiVersion": "v1",
    "metadata": {},
    "status": "Failure",
    "code": 400,
    "errorCode": "CCE.01400013",
    "errorMessage": "Insufficient volume quota.",
    "error_code": "CCE_CM.0307",
    "error_msg": "Volume quota is not enough",
    "message": "Volume quota checking failed as abnormal volume quota response for volumes_SATA",
    "reason": "QuotaAbnormal"
}

Which leads to node creation failure and rancher driver tries to recreate cluster and related resources in a loop.

[anton-sidelnikov]

Hi @Wolfslicht please try with latest UI https://otc-rancher.obs.eu-de.otc.t-systems.com/cluster/ui/latest/component.js