Open tobiasmuehl opened 6 months ago
Tried this again with a fresh docker image. It applies CRDs and then gets stuck in reboot loop
rancher-1 | 2024/02/07 05:27:13 [INFO] Rancher version v2.8.1 (fa9e804ff) is starting
rancher-1 | 2024/02/07 05:27:13 [INFO] Rancher arguments {ACMEDomains:[censored.com] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
rancher-1 | 2024/02/07 05:27:13 [INFO] Listening on /tmp/log.sock
rancher-1 | 2024/02/07 05:27:13 [INFO] Waiting for k3s to start
rancher-1 | 2024/02/07 05:27:14 [INFO] Waiting for server to become available: the server is currently unable to handle the request
rancher-1 | 2024/02/07 05:27:16 [INFO] Waiting for server to become available: the server is currently unable to handle the request
rancher-1 | 2024/02/07 05:27:18 [INFO] Waiting for server to become available: the server is currently unable to handle the request
rancher-1 | 2024/02/07 05:27:20 [INFO] Running in single server mode, will not peer connections
rancher-1 | 2024/02/07 05:27:20 [INFO] Applying CRD features.management.cattle.io
rancher-1 | 2024/02/07 05:27:20 [INFO] Waiting for CRD features.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:20 [INFO] Done waiting for CRD features.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:22 [INFO] Creating embedded CRD clusterroletemplatebindings.management.cattle.io
rancher-1 | 2024/02/07 05:27:22 [INFO] Creating embedded CRD globalroles.management.cattle.io
rancher-1 | 2024/02/07 05:27:22 [INFO] Creating embedded CRD globalrolebindings.management.cattle.io
rancher-1 | 2024/02/07 05:27:22 [INFO] Creating embedded CRD projects.management.cattle.io
rancher-1 | 2024/02/07 05:27:22 [INFO] Creating embedded CRD projectroletemplatebindings.management.cattle.io
rancher-1 | 2024/02/07 05:27:22 [INFO] Creating embedded CRD roletemplates.management.cattle.io
rancher-1 | 2024/02/07 05:27:22 [INFO] Waiting for CRD projects.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:22 [INFO] Done waiting for CRD projects.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:22 [INFO] Waiting for CRD projectroletemplatebindings.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:22 [INFO] Done waiting for CRD projectroletemplatebindings.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:22 [INFO] Waiting for CRD roletemplates.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:23 [INFO] Done waiting for CRD roletemplates.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD navlinks.ui.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD clusters.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD apiservices.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD settings.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD preferences.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD features.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD clusterrepos.catalog.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD operations.catalog.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD apps.catalog.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD fleetworkspaces.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD bundles.fleet.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD clusters.fleet.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD clustergroups.fleet.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD managedcharts.management.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD clusters.provisioning.cattle.io
rancher-1 | 2024/02/07 05:27:23 [INFO] Applying CRD clusters.provisioning.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD rkeclusters.rke.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD rkebootstraps.rke.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD custommachines.rke.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD clusters.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD machines.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Applying CRD machinesets.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:27:24 [INFO] Waiting for CRD machinesets.cluster.x-k8s.io to become available
rancher-1 | 2024/02/07 05:27:25 [INFO] Done waiting for CRD machinesets.cluster.x-k8s.io to become available
rancher-1 | 2024/02/07 05:27:25 [INFO] Creating CRD authconfigs.management.cattle.io
rancher-1 | 2024/02/07 05:27:25 [INFO] Creating CRD groupmembers.management.cattle.io
rancher-1 | 2024/02/07 05:27:25 [INFO] Creating CRD groups.management.cattle.io
rancher-1 | 2024/02/07 05:27:25 [INFO] Creating CRD tokens.management.cattle.io
rancher-1 | 2024/02/07 05:27:25 [INFO] Creating CRD userattributes.management.cattle.io
rancher-1 | 2024/02/07 05:27:25 [INFO] Creating CRD users.management.cattle.io
rancher-1 | 2024/02/07 05:27:25 [INFO] Waiting for CRD tokens.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:26 [INFO] Done waiting for CRD tokens.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:26 [INFO] Waiting for CRD userattributes.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:26 [INFO] Done waiting for CRD userattributes.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:26 [INFO] Waiting for CRD users.management.cattle.io to become available
rancher-1 | 2024/02/07 05:27:27 [FATAL] k3s exited with: exit status 1
rancher-1 | INFO: Running k3s server --cluster-init --cluster-reset
rancher-1 | 2024/02/07 05:27:45 [INFO] Rancher version v2.8.1 (fa9e804ff) is starting
rancher-1 | 2024/02/07 05:27:45 [INFO] Rancher arguments {ACMEDomains:[censored.com] AddLocal:true Embedded:false BindHost: HTTPListenPort:80 HTTPSListenPort:443 K8sMode:auto Debug:false Trace:false NoCACerts:false AuditLogPath:/var/log/auditlog/rancher-api-audit.log AuditLogMaxage:10 AuditLogMaxsize:100 AuditLogMaxbackup:10 AuditLevel:0 Features: ClusterRegistry:}
rancher-1 | 2024/02/07 05:27:45 [INFO] Listening on /tmp/log.sock
rancher-1 | 2024/02/07 05:27:45 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:27:47 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:27:49 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:27:51 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:27:53 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:27:55 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:27:57 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:27:59 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:28:01 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:28:03 [INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
rancher-1 | 2024/02/07 05:28:05 [INFO] Running in single server mode, will not peer connections
rancher-1 | 2024/02/07 05:28:05 [INFO] Applying CRD features.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Updating embedded CRD clusterroletemplatebindings.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Updating embedded CRD globalroles.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Updating embedded CRD globalrolebindings.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Updating embedded CRD projects.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Updating embedded CRD projectroletemplatebindings.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Updating embedded CRD roletemplates.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Applying CRD navlinks.ui.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Applying CRD podsecurityadmissionconfigurationtemplates.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Applying CRD clusters.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Applying CRD apiservices.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Applying CRD clusterregistrationtokens.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Applying CRD settings.management.cattle.io
rancher-1 | 2024/02/07 05:28:06 [INFO] Applying CRD preferences.management.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD features.management.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD clusterrepos.catalog.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD operations.catalog.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD apps.catalog.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD fleetworkspaces.management.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD managedcharts.management.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD clusters.provisioning.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD clusters.provisioning.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD rkeclusters.rke.cattle.io
rancher-1 | 2024/02/07 05:28:07 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD rkebootstraps.rke.cattle.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD rkebootstraptemplates.rke.cattle.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD rkecontrolplanes.rke.cattle.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD custommachines.rke.cattle.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD etcdsnapshots.rke.cattle.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD clusters.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD machinedeployments.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD machinehealthchecks.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD machines.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:28:08 [INFO] Applying CRD machinesets.cluster.x-k8s.io
rancher-1 | 2024/02/07 05:28:09 [FATAL] k3s exited with: exit status 1
rancher-1 | INFO: Running k3s server --cluster-init --cluster-reset
Exact same setup works when running on Ubuntu 22.04. I will not be attempting to install it on RockyLinux then, but leaving this open in case anyone wants to investigate
I encountered the same problem when using Anolis OS. Use docker cp 34103a0fc4e0:/var/lib/rancher/k3s.log .
to view the k3s log, and you can see such an error:
time="2024-03-13T05:40:49Z" level=info msg="Starting the netpol controller version , built on , go1.19.8"
I0313 05:40:49.389109 51 network_policy_controller.go:163] Starting network policy controller
F0313 05:40:49.389689 51 network_policy_controller.go:404] failed to run iptables command to create KUBE-ROUTER-INPUT chain due to running [/usr/bin/iptables -t filter -S KUBE-ROUTER-INPUT 1 --wait]: exit status 3: iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
panic: F0313 05:40:49.389689 51 network_policy_controller.go:404] failed to run iptables command to create KUBE-ROUTER-INPUT chain due to running [/usr/bin/iptables -t filter -S KUBE-ROUTER-INPUT 1 --wait]: exit status 3: iptables v1.8.8 (legacy): can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.
goroutine 41838 [running]:
k8s.io/klog/v2.(*loggingT).output(0x82f93c0, 0x3, 0x0, 0xc014b1af50, 0x1, {0x67beecd?, 0x2?}, 0xc000100800?, 0x0)
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.80.1-k3s1/klog.go:926 +0x6bd
k8s.io/klog/v2.(*loggingT).printfDepth(0x82f93c0, 0x11?, 0x0, {0x0, 0x0}, 0x19c?, {0x4ec51ee, 0x3b}, {0xc0044faec0, 0x2, ...})
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.80.1-k3s1/klog.go:736 +0x1f3
k8s.io/klog/v2.(*loggingT).printf(...)
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.80.1-k3s1/klog.go:718
k8s.io/klog/v2.Fatalf(...)
/go/pkg/mod/github.com/k3s-io/klog/v2@v2.80.1-k3s1/klog.go:1598
github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).ensureTopLevelChains(0xc005f99320)
/go/pkg/mod/github.com/k3s-io/kube-router/v2@v2.0.1-0.20230411195838-cced939a8ba1/pkg/controllers/netpol/network_policy_controller.go:404 +0x166f
github.com/cloudnativelabs/kube-router/v2/pkg/controllers/netpol.(*NetworkPolicyController).Run(0xc005f99320, 0xc00bcc2420, 0xc000190c00, 0xc016f9a710)
/go/pkg/mod/github.com/k3s-io/kube-router/v2@v2.0.1-0.20230411195838-cced939a8ba1/pkg/controllers/netpol/network_policy_controller.go:167 +0x175
created by github.com/k3s-io/k3s/pkg/agent/netpol.Run
/go/src/github.com/k3s-io/k3s/pkg/agent/netpol/netpol.go:141 +0xd85
So I checked iptables and executing lsmod | grep iptable
returned nothing. Then I executed the following command to manually load the iptables related modules. Docker run again to deploy rancher, and it will work normally.
sudo modprobe iptable_filter
sudo modprobe iptable_nat
sudo modprobe iptable_mangle
had the same problem on RHEL 8.4, activating the kernel modules like @rootwuj did, worked!
Can confirm the below works for RHEL 9
sudo modprobe iptable_filter sudo modprobe iptable_nat sudo modprobe iptable_mangle
thanks @rootwuj
Following "Deploy Rancher" from the official website
Hiya - the method you've to launch rancher is intended as a "preview" mechanism and not a supported production method to deploy rancher. These are quick directions provided on the marketing website as a brief way to "dip your toes in the water". On the technical documents website (linked via the "Getting Started Guide" from the quick start) you will find a more comprehensive quick start guide. That page has a more clear indicator that these directions are meant for test purposes:
The intent of these guides is to quickly launch a sandbox that you can use to evaluate Rancher. These guides are not intended for production environments. For comprehensive setup instructions, see Installation.
When you are using rancher in production it should be on a supported k8s distribution, not as a standalone container. In this preview mode Rancher needs to provide a k8s runtime inside the container for it to work, and that will come with certain limitations. These limitations of preview mode are fine for most light use preview use-cases; however anything more extensive would be better of being tested on a true deployment.
If you're already on a system running Docker, then perhaps opting to deploy RKE1 to that system and then testing Rancher will be easiest. Alternatively, I would suggest that k3s is quite easy to setup as a single node and deploy rancher onto that. Some helpful resources:
Rancher Server Setup
Information about the Cluster
Describe the bug
See docker compose logs
To Reproduce
Following "Deploy Rancher" from the official website
Buy Hetzner Cloud server size CX41 with RockyLinux 9
Install ntp and docker:
Add
docker-compose.yml
docker compose up -d && docker compose logs -f
Result Calls to
[INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
happen rapidly a few times, then Rancher gives up waiting for the k3s cluster and crashes the whole docker container. This results is a reboot loop.Expected Result
Calls to
[INFO] Waiting for server to become available: Get "https://127.0.0.1:6444/version?timeout=15m0s": dial tcp 127.0.0.1:6444: connect: connection refused
actually apply the 15 minute timeout and eventually succeed