search

rancher / rancher

Complete container management platform
http://rancher.com
Apache License 2.0
23.36k stars 2.97k forks source link

[BUG] Rancher fails to start with Feature-flag set to embedded-cluster-api=false #41724

Open cpinjani opened 1 year ago

cpinjani commented 1 year ago

Rancher Server Setup

Describe the bug

Rancher fails to start with Feature-flag set to embedded-cluster-api=false

To Reproduce

docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher:v2.7.3 --features=embedded-cluster-api=false

Expected Result

Rancher should start with Feature-flag set to embedded-cluster-api=false

Additional context

It is working on 2.6.9

Logs

2023/06/02 06:59:17 [INFO] Rancher version v2.7.3 (bbba80e2a) is starting
2023/06/02 06:59:17 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:embedded-cluster-api=false ClusterRegistry:}
2023/06/02 06:59:17 [INFO] Listening on /tmp/log.sock
2023/06/02 06:59:17 [INFO] Waiting for k3s to start
2023/06/02 06:59:18 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2023/06/02 06:59:20 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2023/06/02 06:59:27 [INFO] Running in single server mode, will not peer connections
2023/06/02 06:59:27 [INFO] Applying CRD features.management.cattle.io
2023/06/02 06:59:27 [INFO] Waiting for CRD features.management.cattle.io to become available
2023/06/02 06:59:28 [INFO] Done waiting for CRD features.management.cattle.io to become available
2023/06/02 06:59:30 [INFO] Applying CRD navlinks.ui.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusters.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD apiservices.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD settings.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD preferences.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD features.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD operations.catalog.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD apps.catalog.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD bundles.fleet.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusters.fleet.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD managedcharts.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusters.provisioning.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD clusters.provisioning.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkeclusters.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD custommachines.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
2023/06/02 06:59:31 [INFO] Waiting for CRD etcdsnapshots.rke.cattle.io to become available
2023/06/02 06:59:32 [INFO] Done waiting for CRD etcdsnapshots.rke.cattle.io to become available
2023/06/02 06:59:32 [INFO] Creating CRD authconfigs.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD groupmembers.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD groups.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD tokens.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD userattributes.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD users.management.cattle.io
2023/06/02 06:59:32 [INFO] Waiting for CRD groups.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Done waiting for CRD groups.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Waiting for CRD tokens.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Done waiting for CRD tokens.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Waiting for CRD userattributes.management.cattle.io to become available
2023/06/02 06:59:34 [INFO] Done waiting for CRD userattributes.management.cattle.io to become available
2023/06/02 06:59:34 [INFO] Waiting for CRD users.management.cattle.io to become available
2023/06/02 06:59:34 [INFO] Done waiting for CRD users.management.cattle.io to become available
2023/06/02 06:59:35 [INFO] Creating CRD catalogs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD apps.project.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD catalogtemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD catalogtemplateversions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD apprevisions.project.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusteralerts.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusterroletemplatebindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusteralertgroups.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clustercatalogs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD dynamicschemas.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusteralertrules.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD etcdbackups.management.cattle.io
2023/06/02 06:59:35 [INFO] Waiting for CRD apps.project.cattle.io to become available
2023/06/02 06:59:35 [INFO] Creating CRD globalrolebindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clustermonitorgraphs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD globalroles.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD composeconfigs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD multiclusterapps.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD kontainerdrivers.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodedrivers.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD multiclusterapprevisions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodepools.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD monitormetrics.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD notifiers.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodetemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectalerts.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodes.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectalertgroups.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD podsecuritypolicytemplateprojectbindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD podsecuritypolicytemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectcatalogs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectalertrules.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectnetworkpolicies.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectroletemplatebindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectmonitorgraphs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD templates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projects.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD templateversions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rkek8ssystemimages.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD templatecontents.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD globaldnses.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD globaldnsproviders.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rkek8sserviceoptions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rancherusernotifications.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rkeaddons.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD roletemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD samltokens.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clustertemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Waiting for CRD clustermonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:35 [INFO] Done waiting for CRD apps.project.cattle.io to become available
2023/06/02 06:59:35 [INFO] Waiting for CRD apprevisions.project.cattle.io to become available
2023/06/02 06:59:35 [INFO] Creating CRD clustertemplaterevisions.management.cattle.io
2023/06/02 06:59:35 [INFO] Waiting for CRD nodedrivers.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD clustermonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD multiclusterapprevisions.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD apprevisions.project.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD nodedrivers.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD rkeaddons.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD multiclusterapprevisions.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD notifiers.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD rkeaddons.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD samltokens.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD notifiers.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD projectcatalogs.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD samltokens.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD clustertemplates.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD projectcatalogs.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD templates.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD clustertemplates.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD templates.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD templatecontents.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD projectroletemplatebindings.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD templatecontents.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD projectalertgroups.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD projectroletemplatebindings.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD projects.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD projectalertgroups.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD projectalertrules.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD projects.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD rkek8ssystemimages.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD projectalertrules.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD templateversions.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD rkek8ssystemimages.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD nodepools.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD templateversions.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD globaldnsproviders.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD nodepools.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD nodetemplates.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD globaldnsproviders.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD rancherusernotifications.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD nodetemplates.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD nodes.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD rancherusernotifications.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD multiclusterapps.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD nodes.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD rkek8sserviceoptions.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD multiclusterapps.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD projectalerts.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD rkek8sserviceoptions.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD roletemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD projectalerts.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD globaldnses.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD roletemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD podsecuritypolicytemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD globaldnses.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD monitormetrics.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD podsecuritypolicytemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD projectnetworkpolicies.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Done waiting for CRD monitormetrics.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Waiting for CRD projectmonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Done waiting for CRD projectnetworkpolicies.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Waiting for CRD podsecuritypolicytemplateprojectbindings.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Done waiting for CRD projectmonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Waiting for CRD composeconfigs.management.cattle.io to become available
2023/06/02 06:59:44 [INFO] Done waiting for CRD podsecuritypolicytemplateprojectbindings.management.cattle.io to become available
2023/06/02 06:59:44 [INFO] Done waiting for CRD composeconfigs.management.cattle.io to become available
W0602 06:59:48.498258      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:48.498944      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 06:59:50.069058      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:50.069098      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 06:59:52.817423      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:52.817520      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 06:59:56.515287      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:56.515529      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 07:00:04.282081      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 07:00:04.282123      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
salasberryfin commented 1 year ago

As mentioned in rancher-sandbox/rancher-turtles#207, this may be solved by replacing --set features=embedded-cluster-api=false with --set CATTLE_FEATURES=embedded-cluster-api=false. Hopefully this fixes the issue.

cpinjani commented 12 months ago

Still hitting this issue on Rancher version 2.7.8 when below installation commands are used: Docker:

docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 -e CATTLE_BOOTSTRAP_PASSWORD="rancherpassword" rancher/rancher:v2.7.8 --features=embedded-cluster-api=false

Helm:

helm install rancher rancher-stable/rancher --namespace cattle-system --create-namespace --set bootstrapPassword=rancheradmin --set replicas=1 --set hostname=<hostname> --set 'extraEnv[0].name=CATTLE_FEATURES' --set 'extraEnv[0].value=embedded-cluster-api=false' --set global.cattle.psp.enabled=false --version v2.7.8

Workaround is to install CAPI cluster with the docker provider before Rancher installation as per https://github.com/rancher/rancher/issues/39924#issuecomment-1568661964

richardcase commented 12 months ago

This will fail unless Rancher Turtles (or directly CAPI) is installed first.

salasberryfin commented 12 months ago

This is strange because I've seen it failing multiple times but this E2E scenario worked for me and is following the same order of installation: Rancher first with CAPI disabled, then Rancher Turtles, after moving to using this syntax.

cpinjani commented 6 months ago

Issue is reproducible on v2.8.2 by using below install command:

helm install rancher rancher-latest/rancher --version 2.8.2 --namespace cattle-system --set hostname=<hostname>  --set 'extraEnv[0].name=CATTLE_FEATURES' --set 'extraEnv[0].value=embedded-cluster-api=false' --set replicas=1 --set global.cattle.psp.enabled=false --set bootstrapPassword=rancherpassword --set rancherImageTag=v2.8.2 --wait

Logs:

kubectl logs -n cattle-system rancher-7659f497db-ddmst -f2024/03/27 07:13:45 [INFO] Rancher version v2.8.2 (2f7113dc3) is starting
2024/03/27 07:13:45 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:embedded-cluster-api=false ClusterRegistry:}
2024/03/27 07:13:45 [INFO] Listening on /tmp/log.sock
2024/03/27 07:13:45 [INFO] Running in clustered mode with ID 10.42.0.13, monitoring endpoint cattle-system/rancher
2024/03/27 07:13:45 [INFO] Applying CRD features.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD clusterroletemplatebindings.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD globalroles.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD globalrolebindings.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD projects.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD projectroletemplatebindings.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD roletemplates.management.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD navlinks.ui.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD clusters.management.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD apiservices.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD settings.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD preferences.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD features.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD operations.catalog.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD apps.catalog.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD managedcharts.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusters.provisioning.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusters.provisioning.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkeclusters.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD custommachines.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
W0327 07:13:52.064694      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:13:52.064743      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:13:53.547738      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:13:53.547775      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:13:56.678475      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:13:56.678508      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:14:02.836781      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:14:02.836810      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:14:12.433680      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:14:12.433716      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:14:36.325709      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:14:36.325749      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:15:13.225011      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:15:13.225041      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2024/03/27 07:15:44 [INFO] Starting project.cattle.io/v3, Kind=App controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=Node workers
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding workers
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2024/03/27 07:15:44 [INFO] Shutting down project.cattle.io/v3, Kind=App workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=Group workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterTemplateRevision workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=ProjectCatalog workers
2024/03/27 07:15:44 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2024/03/27 07:15:44 [INFO] Shutting down provisioning.cattle.io/v1, Kind=Cluster workers
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=User controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=NodePool workers
2024/03/27 07:15:44 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=User workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=RoleTemplate workers
2024/03/27 07:15:44 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=GlobalRoleBinding workers
2024/03/27 07:15:44 [FATAL] failed to wait for caches to sync
richardcase commented 6 months ago

I think initially we will require Rancher to be started without that flag set. But long term we need to find a solution to this.