search

rancher / rancher

Complete container management platform
http://rancher.com
Apache License 2.0
23.52k stars 2.98k forks source link

[BUG] Rancher fails to start with Feature-flag set to embedded-cluster-api=false #41724

Open cpinjani opened 1 year ago

cpinjani commented 1 year ago

Rancher Server Setup

Describe the bug

Rancher fails to start with Feature-flag set to embedded-cluster-api=false

To Reproduce

docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 rancher/rancher:v2.7.3 --features=embedded-cluster-api=false

Expected Result

Rancher should start with Feature-flag set to embedded-cluster-api=false

Additional context

It is working on 2.6.9

Logs

2023/06/02 06:59:17 [INFO] Rancher version v2.7.3 (bbba80e2a) is starting
2023/06/02 06:59:17 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:embedded-cluster-api=false ClusterRegistry:}
2023/06/02 06:59:17 [INFO] Listening on /tmp/log.sock
2023/06/02 06:59:17 [INFO] Waiting for k3s to start
2023/06/02 06:59:18 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2023/06/02 06:59:20 [INFO] Waiting for server to become available: the server is currently unable to handle the request
2023/06/02 06:59:27 [INFO] Running in single server mode, will not peer connections
2023/06/02 06:59:27 [INFO] Applying CRD features.management.cattle.io
2023/06/02 06:59:27 [INFO] Waiting for CRD features.management.cattle.io to become available
2023/06/02 06:59:28 [INFO] Done waiting for CRD features.management.cattle.io to become available
2023/06/02 06:59:30 [INFO] Applying CRD navlinks.ui.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusters.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD apiservices.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD settings.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD preferences.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD features.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD operations.catalog.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD apps.catalog.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD bundles.fleet.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusters.fleet.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD managedcharts.management.cattle.io
2023/06/02 06:59:30 [INFO] Applying CRD clusters.provisioning.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD clusters.provisioning.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkeclusters.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD custommachines.rke.cattle.io
2023/06/02 06:59:31 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
2023/06/02 06:59:31 [INFO] Waiting for CRD etcdsnapshots.rke.cattle.io to become available
2023/06/02 06:59:32 [INFO] Done waiting for CRD etcdsnapshots.rke.cattle.io to become available
2023/06/02 06:59:32 [INFO] Creating CRD authconfigs.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD groupmembers.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD groups.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD tokens.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD userattributes.management.cattle.io
2023/06/02 06:59:32 [INFO] Creating CRD users.management.cattle.io
2023/06/02 06:59:32 [INFO] Waiting for CRD groups.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Done waiting for CRD groups.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Waiting for CRD tokens.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Done waiting for CRD tokens.management.cattle.io to become available
2023/06/02 06:59:33 [INFO] Waiting for CRD userattributes.management.cattle.io to become available
2023/06/02 06:59:34 [INFO] Done waiting for CRD userattributes.management.cattle.io to become available
2023/06/02 06:59:34 [INFO] Waiting for CRD users.management.cattle.io to become available
2023/06/02 06:59:34 [INFO] Done waiting for CRD users.management.cattle.io to become available
2023/06/02 06:59:35 [INFO] Creating CRD catalogs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD apps.project.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD catalogtemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD catalogtemplateversions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD apprevisions.project.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusteralerts.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusterroletemplatebindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusteralertgroups.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clustercatalogs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD dynamicschemas.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clusteralertrules.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD etcdbackups.management.cattle.io
2023/06/02 06:59:35 [INFO] Waiting for CRD apps.project.cattle.io to become available
2023/06/02 06:59:35 [INFO] Creating CRD globalrolebindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clustermonitorgraphs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD globalroles.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD composeconfigs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD multiclusterapps.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD kontainerdrivers.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodedrivers.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD multiclusterapprevisions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodepools.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD monitormetrics.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD notifiers.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodetemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectalerts.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD nodes.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectalertgroups.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD podsecuritypolicytemplateprojectbindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD podsecuritypolicytemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectcatalogs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectalertrules.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectnetworkpolicies.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectroletemplatebindings.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projectmonitorgraphs.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD templates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD projects.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD templateversions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rkek8ssystemimages.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD templatecontents.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD globaldnses.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD globaldnsproviders.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rkek8sserviceoptions.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rancherusernotifications.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD rkeaddons.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD roletemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD samltokens.management.cattle.io
2023/06/02 06:59:35 [INFO] Creating CRD clustertemplates.management.cattle.io
2023/06/02 06:59:35 [INFO] Waiting for CRD clustermonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:35 [INFO] Done waiting for CRD apps.project.cattle.io to become available
2023/06/02 06:59:35 [INFO] Waiting for CRD apprevisions.project.cattle.io to become available
2023/06/02 06:59:35 [INFO] Creating CRD clustertemplaterevisions.management.cattle.io
2023/06/02 06:59:35 [INFO] Waiting for CRD nodedrivers.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD clustermonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD multiclusterapprevisions.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD apprevisions.project.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD nodedrivers.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD rkeaddons.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD multiclusterapprevisions.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD notifiers.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Done waiting for CRD rkeaddons.management.cattle.io to become available
2023/06/02 06:59:36 [INFO] Waiting for CRD samltokens.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD notifiers.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD projectcatalogs.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD samltokens.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD clustertemplates.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD projectcatalogs.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD templates.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Done waiting for CRD clustertemplates.management.cattle.io to become available
2023/06/02 06:59:37 [INFO] Waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD templates.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD templatecontents.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD clustertemplaterevisions.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD projectroletemplatebindings.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD templatecontents.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD projectalertgroups.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Done waiting for CRD projectroletemplatebindings.management.cattle.io to become available
2023/06/02 06:59:38 [INFO] Waiting for CRD projects.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD projectalertgroups.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD projectalertrules.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD projects.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD rkek8ssystemimages.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD projectalertrules.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD templateversions.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Done waiting for CRD rkek8ssystemimages.management.cattle.io to become available
2023/06/02 06:59:39 [INFO] Waiting for CRD nodepools.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD templateversions.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD globaldnsproviders.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD nodepools.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD nodetemplates.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD globaldnsproviders.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD rancherusernotifications.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Done waiting for CRD nodetemplates.management.cattle.io to become available
2023/06/02 06:59:40 [INFO] Waiting for CRD nodes.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD rancherusernotifications.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD multiclusterapps.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD nodes.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD rkek8sserviceoptions.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD multiclusterapps.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD projectalerts.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Done waiting for CRD rkek8sserviceoptions.management.cattle.io to become available
2023/06/02 06:59:41 [INFO] Waiting for CRD roletemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD projectalerts.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD globaldnses.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD roletemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD podsecuritypolicytemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD globaldnses.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD monitormetrics.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Done waiting for CRD podsecuritypolicytemplates.management.cattle.io to become available
2023/06/02 06:59:42 [INFO] Waiting for CRD projectnetworkpolicies.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Done waiting for CRD monitormetrics.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Waiting for CRD projectmonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Done waiting for CRD projectnetworkpolicies.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Waiting for CRD podsecuritypolicytemplateprojectbindings.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Done waiting for CRD projectmonitorgraphs.management.cattle.io to become available
2023/06/02 06:59:43 [INFO] Waiting for CRD composeconfigs.management.cattle.io to become available
2023/06/02 06:59:44 [INFO] Done waiting for CRD podsecuritypolicytemplateprojectbindings.management.cattle.io to become available
2023/06/02 06:59:44 [INFO] Done waiting for CRD composeconfigs.management.cattle.io to become available
W0602 06:59:48.498258      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:48.498944      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 06:59:50.069058      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:50.069098      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 06:59:52.817423      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:52.817520      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 06:59:56.515287      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 06:59:56.515529      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0602 07:00:04.282081      33 reflector.go:425] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0602 07:00:04.282123      33 reflector.go:141] pkg/mod/github.com/rancher/client-go@v1.25.4-rancher1/tools/cache/reflector.go:170: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
salasberryfin commented 1 year ago

As mentioned in rancher-sandbox/rancher-turtles#207, this may be solved by replacing --set features=embedded-cluster-api=false with --set CATTLE_FEATURES=embedded-cluster-api=false. Hopefully this fixes the issue.

cpinjani commented 1 year ago

Still hitting this issue on Rancher version 2.7.8 when below installation commands are used: Docker:

docker run --privileged -d --restart=unless-stopped -p 80:80 -p 443:443 -e CATTLE_BOOTSTRAP_PASSWORD="rancherpassword" rancher/rancher:v2.7.8 --features=embedded-cluster-api=false

Helm:

helm install rancher rancher-stable/rancher --namespace cattle-system --create-namespace --set bootstrapPassword=rancheradmin --set replicas=1 --set hostname=<hostname> --set 'extraEnv[0].name=CATTLE_FEATURES' --set 'extraEnv[0].value=embedded-cluster-api=false' --set global.cattle.psp.enabled=false --version v2.7.8

Workaround is to install CAPI cluster with the docker provider before Rancher installation as per https://github.com/rancher/rancher/issues/39924#issuecomment-1568661964

richardcase commented 1 year ago

This will fail unless Rancher Turtles (or directly CAPI) is installed first.

salasberryfin commented 1 year ago

This is strange because I've seen it failing multiple times but this E2E scenario worked for me and is following the same order of installation: Rancher first with CAPI disabled, then Rancher Turtles, after moving to using this syntax.

cpinjani commented 8 months ago

Issue is reproducible on v2.8.2 by using below install command:

helm install rancher rancher-latest/rancher --version 2.8.2 --namespace cattle-system --set hostname=<hostname>  --set 'extraEnv[0].name=CATTLE_FEATURES' --set 'extraEnv[0].value=embedded-cluster-api=false' --set replicas=1 --set global.cattle.psp.enabled=false --set bootstrapPassword=rancherpassword --set rancherImageTag=v2.8.2 --wait

Logs:

kubectl logs -n cattle-system rancher-7659f497db-ddmst -f2024/03/27 07:13:45 [INFO] Rancher version v2.8.2 (2f7113dc3) is starting
2024/03/27 07:13:45 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:embedded-cluster-api=false ClusterRegistry:}
2024/03/27 07:13:45 [INFO] Listening on /tmp/log.sock
2024/03/27 07:13:45 [INFO] Running in clustered mode with ID 10.42.0.13, monitoring endpoint cattle-system/rancher
2024/03/27 07:13:45 [INFO] Applying CRD features.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD clusterroletemplatebindings.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD globalroles.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD globalrolebindings.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD projects.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD projectroletemplatebindings.management.cattle.io
2024/03/27 07:13:45 [INFO] Updating embedded CRD roletemplates.management.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD navlinks.ui.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD clusters.management.cattle.io
2024/03/27 07:13:45 [INFO] Applying CRD apiservices.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD settings.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD preferences.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD features.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD operations.catalog.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD apps.catalog.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD managedcharts.management.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusters.provisioning.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD clusters.provisioning.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkeclusters.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD custommachines.rke.cattle.io
2024/03/27 07:13:46 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
W0327 07:13:52.064694      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:13:52.064743      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:13:53.547738      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:13:53.547775      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:13:56.678475      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:13:56.678508      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:14:02.836781      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:14:02.836810      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:14:12.433680      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:14:12.433716      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:14:36.325709      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:14:36.325749      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W0327 07:15:13.225011      33 reflector.go:533] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E0327 07:15:13.225041      33 reflector.go:148] pkg/mod/github.com/rancher/client-go@v1.27.4-rancher1/tools/cache/reflector.go:231: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=ProjectCatalog controller
2024/03/27 07:15:44 [INFO] Starting project.cattle.io/v3, Kind=App controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=Node controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=Node workers
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=ClusterTemplateRevision controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=PodSecurityPolicyTemplateProjectBinding workers
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=Group controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=RoleTemplate controller
2024/03/27 07:15:44 [INFO] Shutting down project.cattle.io/v3, Kind=App workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=Group workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=ClusterTemplateRevision workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=ProjectCatalog workers
2024/03/27 07:15:44 [INFO] Starting provisioning.cattle.io/v1, Kind=Cluster controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=NodePool controller
2024/03/27 07:15:44 [INFO] Shutting down provisioning.cattle.io/v1, Kind=Cluster workers
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=User controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=NodePool workers
2024/03/27 07:15:44 [INFO] Starting apiregistration.k8s.io/v1, Kind=APIService controller
2024/03/27 07:15:44 [INFO] Starting management.cattle.io/v3, Kind=GlobalRoleBinding controller
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=User workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=RoleTemplate workers
2024/03/27 07:15:44 [INFO] Shutting down apiregistration.k8s.io/v1, Kind=APIService workers
2024/03/27 07:15:44 [INFO] Shutting down management.cattle.io/v3, Kind=GlobalRoleBinding workers
2024/03/27 07:15:44 [FATAL] failed to wait for caches to sync
richardcase commented 8 months ago

I think initially we will require Rancher to be started without that flag set. But long term we need to find a solution to this.

cpinjani commented 2 weeks ago

On Rancher v2.9.3, tried helm install with flags:

$ kubectl get app -n cattle-provisioning-capi-system
NAME                        CHART                       VERSION           RELEASE NAME                RELEASE VERSION   STATUS
rancher-provisioning-capi   rancher-provisioning-capi   104.0.0+up0.3.0   rancher-provisioning-capi   1                 deployed

Logs

2024/11/14 08:03:48 [INFO] Rancher version v2.9.3 (261bb3c70eeba7b6367a9924c9c2d0a3433f27c2) is starting
2024/11/14 08:03:48 [INFO] Rancher arguments {ACMEDomains:[] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features:embedded-cluster-api=false ClusterRegistry:}
2024/11/14 08:03:48 [INFO] Listening on /tmp/log.sock
2024/11/14 08:03:48 [INFO] Running in clustered mode with ID 10.42.0.13, monitoring endpoint cattle-system/rancher
2024/11/14 08:03:49 [INFO] Applying CRD features.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD clusterroletemplatebindings.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD globalroles.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD globalrolebindings.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD projects.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD projectroletemplatebindings.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD roletemplates.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD clusterproxyconfigs.management.cattle.io
2024/11/14 08:03:49 [INFO] Updating embedded CRD uiplugins.catalog.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD navlinks.ui.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD clusters.management.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD apiservices.management.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD settings.management.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD preferences.management.cattle.io
2024/11/14 08:03:49 [INFO] Applying CRD features.management.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD clusterrepos.catalog.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD operations.catalog.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD apps.catalog.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD fleetworkspaces.management.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD managedcharts.management.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD clusters.provisioning.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD clusters.provisioning.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD rkeclusters.rke.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD rkebootstraps.rke.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD custommachines.rke.cattle.io
2024/11/14 08:03:50 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
W1114 08:03:56.991419      39 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E1114 08:03:56.991469      39 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W1114 08:03:58.061156      39 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E1114 08:03:58.061190      39 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W1114 08:04:01.015687      39 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E1114 08:04:01.015720      39 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W1114 08:04:06.350859      39 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E1114 08:04:06.350891      39 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W1114 08:04:15.088479      39 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E1114 08:04:15.088512      39 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
W1114 08:04:30.593750      39 reflector.go:547] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)
E1114 08:04:30.593778      39 reflector.go:150] pkg/mod/k8s.io/client-go@v0.30.1/tools/cache/reflector.go:232: Failed to watch *v1beta1.Machine: failed to list *v1beta1.Machine: the server could not find the requested resource (get machines.meta.k8s.io)