kurborg
commented
7 months ago ● rke2-agent.service - Rancher Kubernetes Engine v2 (agent)
Loaded: loaded (/usr/local/lib/systemd/system/rke2-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2024-03-28 13:39:14 EDT; 2min 30s ago
Docs: https://github.com/rancher/rke2#readme
Process: 861436 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
Process: 861434 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
Process: 861431 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
Main PID: 861437 (rke2)
Tasks: 86
Memory: 4.5G
CGroup: /system.slice/rke2-agent.service
├─861437 /usr/local/bin/rke2 agent
├─861460 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd
├─861513 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --allowed-unsafe-sysctls=net.ipv4.ip_forward,net.ipv6.conf.all.forwarding --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke2/agent/client-ca.crt --cloud-provider=external --cluster-dns=10.43.0.10 --cluster-domain=cluster.local --container-runtime-endpoint=unix:///run/k3s/containerd/containerd.sock --containerd=/run/k3s/containerd/containerd.sock --eviction-hard=imagefs.available<5%,nodefs.available<5% --eviction-minimum-reclaim=imagefs.available=10%,nodefs.available=10% --fail-swap-on=false --healthz-bind-address=127.0.0.1 --hostname-override=silverbox2 --kubeconfig=/var/lib/rancher/rke2/agent/kubelet.kubeconfig --node-ip=203.0.113.254 --node-labels= --pod-infra-container-image=index.docker.io/rancher/pause:3.6 --pod-manifest-path=/var/lib/rancher/rke2/agent/pod-manifests --read-only-port=0 --resolv-conf=/var/lib/rancher/rke2/agent/etc/resolv.conf --serialize-image-pulls=false --tls-cert-file=/var/lib/rancher/rke2/agent/serving-kubelet.crt --tls-private-key-file=/var/lib/rancher/rke2/agent/serving-kubelet.key
└─861626 /var/lib/rancher/rke2/data/v1.26.14-rke2r1-b13f824cc417/bin/containerd-shim-runc-v2 -namespace k8s.io -id e796b7d2594362be074497cc3c5366a6ebaa4594e019778241eeca3427275f1a -address /run/k3s/containerd/containerd.sock
Mar 28 13:41:34 silverbox2 rke2[861437]: time="2024-03-28T13:41:34-04:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 203.0.113.254:9345: connect: connection refused"
Mar 28 13:41:34 silverbox2 rke2[861437]: time="2024-03-28T13:41:34-04:00" level=error msg="Remotedialer proxy error" error="dial tcp 203.0.113.254:9345: connect: connection refused"
Mar 28 13:41:39 silverbox2 rke2[861437]: time="2024-03-28T13:41:39-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": EOF"
Mar 28 13:41:39 silverbox2 rke2[861437]: time="2024-03-28T13:41:39-04:00" level=info msg="Connecting to proxy" url="wss://203.0.113.254:9345/v1-rke2/connect"
Mar 28 13:41:39 silverbox2 rke2[861437]: time="2024-03-28T13:41:39-04:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 203.0.113.254:9345: connect: connection refused"
Mar 28 13:41:39 silverbox2 rke2[861437]: time="2024-03-28T13:41:39-04:00" level=error msg="Remotedialer proxy error" error="dial tcp 203.0.113.254:9345: connect: connection refused"
Mar 28 13:41:44 silverbox2 rke2[861437]: time="2024-03-28T13:41:44-04:00" level=info msg="Waiting to retrieve kube-proxy configuration; server is not ready: failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\": read tcp 127.0.0.1:41722->127.0.0.1:6444: read: connection reset by peer"
Mar 28 13:41:44 silverbox2 rke2[861437]: time="2024-03-28T13:41:44-04:00" level=info msg="Connecting to proxy" url="wss://203.0.113.254:9345/v1-rke2/connect"
Mar 28 13:41:44 silverbox2 rke2[861437]: time="2024-03-28T13:41:44-04:00" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 203.0.113.254:9345: connect: connection refused"
Mar 28 13:41:44 silverbox2 rke2[861437]: time="2024-03-28T13:41:44-04:00" level=error msg="Remotedialer proxy error" error="dial tcp 203.0.113.254:9345: connect: connection refused"Per the instructions I set up a dummy route doing
sudo ip link add dummy0 type dummy sudo ip link set dummy0 up sudo ip addr add 203.0.113.254/31 dev dummy0 sudo ip route add default via 203.0.113.255 dev dummy0 metric 1000
But without this I get an error for ip route without this apiserver exited: unable to find suitable network address.error='no default routes found in "/proc/net/route" or "/proc/net/ipv6_route"
brandond
Environmental Info: RKE2 Version: v1.26.15+rke2r11)
Node(s) CPU architecture, OS, and Version: Rocky 8.6
Cluster Configuration: 1 server, 3 agents. Airgapped network. Do I need to setup certs if on an airgapped installation?
Describe the bug: Trying to run rke2 on an airgapped webserver container 4 nodes. The rke2-server.service is running fine, but when trying to run rke2-agent.service on the other nodes and using the node-token from the primary node into the /etc/rancher/rke2/config.yaml but I'm getting an error for
failed to get CA certs: Get \"https://127.0.0.1:6444/cacerts\
I'm able to curl -vks https://SERVERIP:9345/ping with the expected output. rke2-server is running fine and showing when I run kubectl get nodes It's running on separate servers/nodes not on the same server