Closed jankohout95 closed 3 months ago
axgkl
commented
3 months ago Hi, can you post
./hetzner-k3s --version To me the first run failure looks like a firewall problem - could you reach 23.88.105.19 on port 6443?
At the second run the problem is not clear - did it time out?
PS: No Stress but would be nice to get those long logs formatted, so that a line start is at a line start, plus put into a <details><summary>syslog</summary>..... </details> section :-)
jankohout95
commented
3 months ago Thanks for such a fast reply! @axgkl
Just an important note to mention, everything above the version of k3s v1.27.9+k3s1 seems not working.
hetzner-k3s version: 1.1.5
jankohout95
commented
3 months ago @axgkl
Here are all the logs in readable form:hetzner_k3s_bug_resolving
Btw,
Error 1: (curl was performed from worker node)
curl --insecure https://23.88.105.19:6443 { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 }
Error 2: I left it running for a couple of minutes with no timeout, so it seems like there is no default value for this type of connection.
axgkl
commented
3 months ago Hi, looking at your error 1 syslog, this here puzzles me:
Hi could never successfully connect to the master via the internal network - over 4 Minutes, leading to an exit and attempted restart of the k3s-agent.
I got a hetzner status error yesterday about issues with their private ip networks, maybe it's that?
To crosscheck: Can you attempt now to connect to 10.0.0.2 port 6443, from the worker at 10.0.0.3? Is that port reachable now?
And IF that port is reachable:
Whats the status of systemctl status k3s-agent on your worker now?
Also try restarting it and look at the journal.
Regarding version differences I can say nothing, not an expert on different k3s versions. I personally tend to think those connect issues are independent of those versions.
jankohout95
commented
3 months ago Hi @axgkl,
I was trying the same process multiple times with v1.27.9+k3s1 (the last working version), and it was working without any issues. E.g., creating and deleting clusters 5 times in a row. That was working with this particular version, so I am pretty sure it is version-related.
The status of the agent was "enabled", but it has not started. I did not try to restart it, because I hoped it would self-heal.
axgkl
commented
3 months ago Hi @jankohout95
and with 1.30 it failed at the time where 1.27 worked? That would be really crazy. Asking because of those hetzner problems with private ip networks, we rely on.
I'm rather new in here but from the source code I don't see any different behaviour regarding what hetzner-k3s does, for different k3s versions.
Plus, at least with our new release v2, I use 1.30 all the time w/o probs (except when there had been those hetzner probs).
jankohout95
commented
3 months ago Hi @axgkl,
still the same scenario, I doubt the problem with Hetzner networking.
jrudolph
commented
3 months ago I also cannot set up new clusters with newer k8s versions. Upgrading old clusters to 1.30 works but not setting up new ones. k8s 1.28.5 still works but 1.28.11 is already broken.
I'm gathering more info about what might be wrong and will report back.
jrudolph
commented
3 months ago Adding new nodes directly with 1.30+ also works, so I assume that something in the setup process of the master that starts to break in later versions.
My config is quite straight-forward:
cluster_name: test
kubeconfig_path: "./kubeconfig"
k3s_version: v1.30.2+k3s2
public_ssh_key_path: "~/.ssh/id_rsa.pub"
private_ssh_key_path: "~/.ssh/id_rsa"
use_ssh_agent: true
ssh_allowed_networks:
- 0.0.0.0/0
api_allowed_networks:
- 0.0.0.0/0
schedule_workloads_on_masters: false
masters_pool:
instance_type: cax11
instance_count: 1
location: fsn1
worker_node_pools:
- name: x86
image: ubuntu-24.04
instance_type: cax11
instance_count: 1
location: fsn1
additional_packages:
- nfs-common
kubelet_args:
- "eviction-hard=memory.available\\<200Mi"I wonder if the problem is network related, the master logs show this:
level=warning msg="no subnet found for key: FLANNEL_NETWORK in file: /run/flannel/subnet.env"
level=warning msg="no subnet found for key: FLANNEL_SUBNET in file: /run/flannel/subnet.env"
level=warning msg="no subnet found for key: FLANNEL_IPV6_NETWORK in file: /run/flannel/subnet.env"
level=warning msg="no subnet found for key: FLANNEL_IPV6_SUBNET in file: /run/flannel/subnet.env"and later an error like this is shown multiple times:
level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint"I think it's this change in k3s-1.28.6 (https://docs.k3s.io/release-notes/v1.28.X#changes-since-v1285k3s1):
Wait for taint to be gone in the node before starting the netpol controller (#9175)
Maybe the result is a chicken-egg situation. If I understand correctly the node.cloudprovider.kubernetes.io/uninitialized taint will only be removed when the CCM is running. However, hetzner-k3s is deploying the CCM only after the workers. The worker systemd services however never finish activating completely because of the taint and so hetzner-k3s never gets to install the CCM...
A simple workaround is to set the number of worker to 0 for the initial install and then run again with the final number.
axgkl
commented
3 months ago @vitobotta See the @jrudolph's comment above - just to make sure this does not escape your attention, crazy good find, imho. Should we deploy any static workers only after the CCM in general?
vitobotta
commented
3 months ago I think I addressed this in the WIP v2 already, as the software is installed before provisioning the worker nodes. Can you try and see if you can reproduce the issue using v2?
jrudolph
commented
3 months ago I think I addressed this in the WIP v2 already, as the software is installed before provisioning the worker nodes. Can you try and see if you can reproduce the issue using v2?
I have only quickly tried v2 (because I try to not change everything at once and I'm already in the process of a largish infrastructure overhaul..), and it failed with some non-descriptive error I didn't want to debug yet, but I can have another try and report in the v2 discussion thread.
vitobotta
commented
3 months ago I think I addressed this in the WIP v2 already, as the software is installed before provisioning the worker nodes. Can you try and see if you can reproduce the issue using v2?
I have only quickly tried v2 (because I try to not change everything at once and I'm already in the process of a largish infrastructure overhaul..), and it failed with some non-descriptive error I didn't want to debug yet, but I can have another try and report in the v2 discussion thread.
I just replied there. I'd open a separate issue if you are having what seems a problem with the config file when using v2.
vitobotta
commented
3 months ago I'm closing this, please open a separate issue for v2 if you still have problems with that version, since I will release it soon.
Deployment of k3s cluster fails with following multiple errors:
Version v1.27.10-rc1+k3s1:
`hetzner-k3s create --config vito_botta_test_cluster_config.yaml Validating configuration......configuration seems valid.
=== Creating infrastructure resources === Creating network...done. Creating firewall...done. Creating SSH key...done. Creating placement group test-masters...done. Creating placement group test-small-static-1...done. Creating server test-cx22-pool-small-static-worker1... Creating server test-cx22-master1... ...server test-cx22-master1 created. ...server test-cx22-pool-small-static-worker1 created. Server test-cx22-master1 already exists, skipping. Waiting for successful ssh connectivity with server test-cx22-master1... Server test-cx22-pool-small-static-worker1 already exists, skipping. Waiting for successful ssh connectivity with server test-cx22-pool-small-static-worker1... ...server test-cx22-master1 is now up. ...server test-cx22-pool-small-static-worker1 is now up.
=== Setting up Kubernetes === Deploying k3s to first master test-cx22-master1... [INFO] Using v1.27.10+k3s2 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.10+k3s2/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.10+k3s2/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit [INFO] systemd: Starting k3s Waiting for the control plane to be ready... Saving the kubeconfig file to test_project/kubeconfig... ...k3s has been deployed to first master test-cx22-master1 and the control plane is up. Deploying k3s to worker test-cx22-pool-small-static-worker1... [INFO] Using v1.27.10+k3s2 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.27.10+k3s2/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.27.10+k3s2/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service [INFO] systemd: Enabling k3s-agent unit [INFO] systemd: Starting k3s-agent ...k3s has been deployed to worker test-cx22-pool-small-static-worker1.
=== Deploying Hetzner drivers ===
Creating secret for Hetzner Cloud token... error: error validating "STDIN": error validating data: failed to download openapi: Get "https://23.88.105.19:6443/openapi/v2?timeout=32s": dial tcp 23.88.105.19:6443: connect: connection refused; if you choose to ignore these errors, turn validation off with --validate=false Failed to create Hetzner Cloud secret:
`
Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.594357 2966 kube.go:489] Starting kube subnet manager Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.595211 2966 server.go:657] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.595751 2966 container_manager_linux.go:265] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.595858 2966 container_manager_linux.go:270] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] CPUManagerPolicy:none CPUManagerPolicyOptions:map[] TopologyManagerScope:container CPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] PodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms TopologyManagerPolicy:none ExperimentalTopologyManagerPolicyOptions:map[]} Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.595906 2966 topology_manager.go:136] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.595939 2966 container_manager_linux.go:301] "Creating device plugin manager" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.596025 2966 state_mem.go:36] "Initialized new in-memory state store" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.599325 2966 kubelet.go:405] "Attempting to sync node with API server" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.599571 2966 kubelet.go:298] "Adding static pod path" path="/var/lib/rancher/k3s/agent/pod-manifests" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.599809 2966 kubelet.go:309] "Adding apiserver pod source" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.600240 2966 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.603922 2966 kuberuntime_manager.go:257] "Container runtime initialized" containerRuntime="containerd" version="v1.7.11-k3s2.27" apiVersion="v1" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.604357 2966 kube.go:510] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.0.0/24] Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.604539 2966 kube.go:510] Creating the node lease for IPv4. This is the n.Spec.PodCIDRs: [10.244.1.0/24] Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.604786 2966 server.go:1163] "Started kubelet" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:23:15.606408 2966 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/rancher/k3s/agent/containerd/io.containerd.snapshotter.v1.overlayfs" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:23:15.606627 2966 kubelet.go:1400] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.606998 2966 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.611634 2966 server.go:162] "Starting to listen" address="0.0.0.0" port=10250 Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.612809 2966 server.go:461] "Adding debug handlers to kubelet server" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.614159 2966 ratelimit.go:65] "Setting rate limiting for podresources endpoint" qps=100 burstTokens=10 Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.615083 2966 volume_manager.go:284] "Starting Kubelet Volume Manager" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.617832 2966 desired_state_of_world_populator.go:145] "Desired state populator starts to run" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.621169 2966 kubelet_network_linux.go:63] "Initialized iptables rules." protocol=IPv4 Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.622573 2966 kubelet_network_linux.go:63] "Initialized iptables rules." protocol=IPv6 Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.622599 2966 status_manager.go:207] "Starting to sync pod status with apiserver" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.622623 2966 kubelet.go:2257] "Starting kubelet main sync loop" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:23:15.622697 2966 kubelet.go:2281] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.644143 2966 cpu_manager.go:214] "Starting CPU manager" policy="none" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.644424 2966 cpu_manager.go:215] "Reconciling" reconcilePeriod="10s" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.644557 2966 state_mem.go:36] "Initialized new in-memory state store" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.644912 2966 state_mem.go:88] "Updated default CPUSet" cpuSet="" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.645082 2966 state_mem.go:96] "Updated CPUSet assignments" assignments=map[] Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.645210 2966 policy_none.go:49] "None policy: Start" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.646007 2966 memory_manager.go:169] "Starting memorymanager" policy="None" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.646159 2966 state_mem.go:35] "Initializing new in-memory state store" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.646491 2966 state_mem.go:75] "Updated machine memory state" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.655291 2966 manager.go:471] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.655872 2966 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.717998 2966 kubelet_node_status.go:70] "Attempting to register node" node="test-cx22-pool-small-static-worker1" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.746684 2966 kubelet_node_status.go:108] "Node was previously registered" node="test-cx22-pool-small-static-worker1" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.746940 2966 kubelet_node_status.go:73] "Successfully registered node" node="test-cx22-pool-small-static-worker1" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.749536 2966 kuberuntime_manager.go:1460] "Updating runtime config through cri with podcidr" CIDR="10.244.1.0/24" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.750910 2966 kubelet_network.go:61] "Updating Pod CIDR" originalPodCIDR="" newPodCIDR="10.244.1.0/24" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:23:15Z" level=info msg="Running kube-proxy --cluster-cidr=10.244.0.0/16 --conntrack-max-per-core=0 --conntrack-tcp-timeout-close-wait=0s --conntrack-tcp-timeout-established=0s --healthz-bind-address=127.0.0.1 --hostname-override=test-cx22-pool-small-static-worker1 --kubeconfig=/var/lib/rancher/k3s/agent/kubeproxy.kubeconfig --proxy-mode=iptables" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.872410 2966 server.go:226] "Warning, all flags other than --config, --write-config-to, and --cleanup are deprecated, please begin using a config file ASAP" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.896843 2966 node.go:141] Successfully retrieved node IP: 10.0.0.3 Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.896892 2966 server_others.go:110] "Detected node IP" address="10.0.0.3" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.907886 2966 server_others.go:192] "Using iptables Proxier" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.907956 2966 server_others.go:199] "kube-proxy running in dual-stack mode" ipFamily=IPv4 Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.907988 2966 server_others.go:200] "Creating dualStackProxier for iptables" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.908035 2966 server_others.go:484] "Detect-local-mode set to ClusterCIDR, but no IPv6 cluster CIDR defined, defaulting to no-op detect-local for IPv6" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.908110 2966 proxier.go:253] "Setting route_localnet=1 to allow node-ports on localhost; to change this either disable iptables.localhostNodePorts (--iptables-localhost-nodeports) or set nodePortAddresses (--nodeport-addresses) to filter loopback addresses" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.909672 2966 server.go:658] "Version info" version="v1.27.10+k3s2" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.909725 2966 server.go:660] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.911794 2966 config.go:188] "Starting service config controller" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.911844 2966 shared_informer.go:311] Waiting for caches to sync for service config Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.911886 2966 config.go:97] "Starting endpoint slice config controller" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.911899 2966 shared_informer.go:311] Waiting for caches to sync for endpoint slice config Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.912955 2966 config.go:315] "Starting node config controller" Aug 1 19:23:15 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:15.912997 2966 shared_informer.go:311] Waiting for caches to sync for node config Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.011969 2966 shared_informer.go:318] Caches are synced for endpoint slice config Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.012000 2966 shared_informer.go:318] Caches are synced for service config Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.013121 2966 shared_informer.go:318] Caches are synced for node config Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:23:16Z" level=info msg="Tunnel authorizer set Kubelet Port 10250" Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.593645 2966 kube.go:152] Node controller sync successful Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.594404 2966 vxlan.go:141] VXLAN config: VNI=1 Port=0 GBP=false Learning=false DirectRouting=false Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.601768 2966 apiserver.go:52] "Watching apiserver" Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.618410 2966 desired_state_of_world_populator.go:153] "Finished populating initial desired state of world" Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.622515 2966 reconciler.go:41] "Reconciler: start to sync state" Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:23:16Z" level=info msg="Wrote flannel subnet file to /run/flannel/subnet.env" Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:23:16Z" level=info msg="Running flannel backend." Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.632265 2966 iptables.go:290] generated 3 rules Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.632315 2966 vxlan_network.go:65] watching for new subnet leases Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.634285 2966 subnet.go:159] Batch elem [0] is { lease.Event{Type:0, Lease:lease.Lease{EnableIPv4:true, EnableIPv6:false, Subnet:ip.IP4Net{IP:0xaf40000, PrefixLen:0x18}, IPv6Subnet:ip.IP6Net{IP:(*ip.IP6)(nil), PrefixLen:0x0}, Attrs:lease.LeaseAttrs{PublicIP:0xa000002, PublicIPv6:(*ip.IP6)(nil), BackendType:"vxlan", BackendData:json.RawMessage{0x7b, 0x22, 0x56, 0x4e, 0x49, 0x22, 0x3a, 0x31, 0x2c, 0x22, 0x56, 0x74, 0x65, 0x70, 0x4d, 0x41, 0x43, 0x22, 0x3a, 0x22, 0x37, 0x65, 0x3a, 0x66, 0x37, 0x3a, 0x37, 0x63, 0x3a, 0x62, 0x36, 0x3a, 0x30, 0x30, 0x3a, 0x62, 0x37, 0x22, 0x7d}, BackendV6Data:json.RawMessage(nil)}, Expiration:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Asof:0}} } Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.636521 2966 iptables.go:290] generated 7 rules Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.665070 2966 iptables.go:283] bootstrap done Aug 1 19:23:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:23:16.693711 2966 iptables.go:283] bootstrap done Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[1]: Created slice User Slice of UID 0. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[1]: Starting User Runtime Directory /run/user/0... Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[1]: Finished User Runtime Directory /run/user/0. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[1]: Starting User Manager for UID 0... Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Queued start job for default target Main User Target. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Created slice User Application Slice. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Reached target Paths. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Reached target Timers. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Starting D-Bus User Message Bus Socket... Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on GnuPG network certificate management daemon. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers). Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on GnuPG cryptographic agent and passphrase cache (restricted). Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on GnuPG cryptographic agent (ssh-agent emulation). Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on GnuPG cryptographic agent and passphrase cache. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on debconf communication socket. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on REST API socket for snapd user session agent. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Listening on D-Bus User Message Bus Socket. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Reached target Sockets. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Reached target Basic System. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[1]: Started User Manager for UID 0. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[1]: Started Session 13 of User root. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Reached target Main User Target. Aug 1 19:23:37 test-cx22-pool-small-static-worker1 systemd[3139]: Startup finished in 151ms. Aug 1 19:23:54 test-cx22-pool-small-static-worker1 systemd[1]: session-13.scope: Deactivated successfully. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: Stopping User Manager for UID 0... Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Stopped target Main User Target. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Stopped target Basic System. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Stopped target Paths. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Stopped target Sockets. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Stopped target Timers. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed D-Bus User Message Bus Socket. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed GnuPG network certificate management daemon. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed GnuPG cryptographic agent and passphrase cache (access for web browsers). Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed GnuPG cryptographic agent and passphrase cache (restricted). Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed GnuPG cryptographic agent (ssh-agent emulation). Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed GnuPG cryptographic agent and passphrase cache. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed debconf communication socket. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Closed REST API socket for snapd user session agent. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Removed slice User Application Slice. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Reached target Shutdown. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Finished Exit the Session. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[3139]: Reached target Exit the Session. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: user@0.service: Deactivated successfully. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: Stopped User Manager for UID 0. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: Stopping User Runtime Directory /run/user/0... Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: run-user-0.mount: Deactivated successfully. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: user-runtime-dir@0.service: Deactivated successfully. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: Stopped User Runtime Directory /run/user/0. Aug 1 19:24:04 test-cx22-pool-small-static-worker1 systemd[1]: Removed slice User Slice of UID 0. Aug 1 19:24:06 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:06Z" level=error msg="Remotedialer proxy error" error="websocket: close 1006 (abnormal closure): unexpected EOF" Aug 1 19:24:11 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:11Z" level=info msg="Connecting to proxy" url="wss://10.0.0.2:6443/v1-k3s/connect" Aug 1 19:24:11 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:11Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.0.2:6443: connect: connection refused" Aug 1 19:24:11 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:11Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.0.2:6443: connect: connection refused" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:16Z" level=info msg="Connecting to proxy" url="wss://10.0.0.2:6443/v1-k3s/connect" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:16Z" level=error msg="Failed to connect to proxy. Empty dialer response" error="dial tcp 10.0.0.2:6443: connect: connection refused" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:16Z" level=error msg="Remotedialer proxy error" error="dial tcp 10.0.0.2:6443: connect: connection refused" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:24:16.818376 2966 controller.go:193] "Failed to update lease" err="Put \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-cx22-pool-small-static-worker1?timeout=10s\": read tcp 127.0.0.1:57348->127.0.0.1:6444: read: connection reset by peer" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:24:16.819609 2966 controller.go:193] "Failed to update lease" err="Put \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-cx22-pool-small-static-worker1?timeout=10s\": read tcp 127.0.0.1:57358->127.0.0.1:6444: read: connection reset by peer" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:24:16.820828 2966 controller.go:193] "Failed to update lease" err="Put \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-cx22-pool-small-static-worker1?timeout=10s\": read tcp 127.0.0.1:57368->127.0.0.1:6444: read: connection reset by peer" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:24:16.822305 2966 controller.go:193] "Failed to update lease" err="Put \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-cx22-pool-small-static-worker1?timeout=10s\": read tcp 127.0.0.1:57382->127.0.0.1:6444: read: connection reset by peer" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: E0801 19:24:16.823959 2966 controller.go:193] "Failed to update lease" err="Put \"https://127.0.0.1:6444/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/test-cx22-pool-small-static-worker1?timeout=10s\": read tcp 127.0.0.1:57394->127.0.0.1:6444: read: connection reset by peer" Aug 1 19:24:16 test-cx22-pool-small-static-worker1 k3s[2966]: I0801 19:24:16.824037 2966 controller.go:116] "failed to update lease using latest lease, fallback to ensure lease" err="failed 5 attempts to update lease" Aug 1 19:24:17 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:17Z" level=error msg="Error getting the node object: Get \"https://127.0.0.1:6444/api/v1/nodes/test-cx22-pool-small-static-worker1\": read tcp 127.0.0.1:57588->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:57344->127.0.0.1:6444: read: connection reset by peer" Aug 1 19:24:17 test-cx22-pool-small-static-worker1 k3s[2966]: time="2024-08-01T19:24:17Z" level=fatal msg="Get \"https://127.0.0.1:6444/api/v1/nodes/test-cx22-pool-small-static-worker1\": read tcp 127.0.0.1:57588->127.0.0.1:6444: read: connection reset by peer - error from a previous attempt: read tcp 127.0.0.1:57344->127.0.0.1:6444: read: connection reset by peer" Aug 1 19:24:17 test-cx22-pool-small-static-worker1 systemd[1]: k3s-agent.service: Main process exited, code=exited, status=1/FAILURE Aug 1 19:24:17 test-cx22-pool-small-static-worker1 systemd[1]: k3s-agent.service: Failed with result 'exit-code'. Aug 1 19:24:17 test-cx22-pool-small-static-worker1 systemd[1]: Failed to start Lightweight Kubernetes. Aug 1 19:24:17 test-cx22-pool-small-static-worker1 systemd[1]: k3s-agent.service: Consumed 3.319s CPU time. Aug 1 19:24:22 test-cx22-pool-small-static-worker1 systemd[1]: k3s-agent.service: Scheduled restart job, restart counter is at 4. Aug 1 19:24:22 test-cx22-pool-small-static-worker1 systemd[1]: Stopped Lightweight Kubernetes. Aug 1 19:24:22 test-cx22-pool-small-static-worker1 systemd[1]: k3s-agent.service: Consumed 3.319s CPU time. Aug 1 19:24:22 test-cx22-pool-small-static-worker1 systemd[1]: Starting Lightweight Kubernetes... Aug 1 19:24:22 test-cx22-pool-small-static-worker1 sh[3240]: + /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service Aug 1 19:24:23 test-cx22-pool-small-static-worker1 k3s[3244]: time="2024-08-01T19:24:23Z" level=info msg="Starting k3s agent v1.27.10+k3s2 (5b2ac881)" Aug 1 19:24:23 test-cx22-pool-small-static-worker1 k3s[3244]: time="2024-08-01T19:24:23Z" level=info msg="Adding server to load balancer k3s-agent-load-balancer: 10.0.0.2:6443" Aug 1 19:24:23 test-cx22-pool-small-static-worker1 k3s[3244]: time="2024-08-01T19:24:23Z" level=info msg="Running load balancer k3s-agent-load-balancer 127.0.0.1:6444 -> [10.0.0.2:6443] [default: 10.0.0.2:6443]" Aug 1 19:24:23 test-cx22-pool-small-static-worker1 k3s[3244]: time="2024-08-01T19:24:23Z" level=warning msg="Cluster CA certificate is not trusted by the host CA bundle, but the token does not include a CA hash. Use the full token from the server's node-token file to enable Cluster CA validation." Aug 1 19:24:23 test-cx22-pool-small-static-worker1 k3s[3244]: time="2024-08-01T19:24:23Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: https://127.0.0.1:6444/v1-k3s/serving-kubelet.crt: 503 Service Unavailable" Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[1]: Created slice User Slice of UID 0. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[1]: Starting User Runtime Directory /run/user/0... Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[1]: Finished User Runtime Directory /run/user/0. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[1]: Starting User Manager for UID 0... Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Queued start job for default target Main User Target. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Created slice User Application Slice. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Reached target Paths. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Reached target Timers. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Starting D-Bus User Message Bus Socket... Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on GnuPG network certificate management daemon. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on GnuPG cryptographic agent and passphrase cache (access for web browsers). Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on GnuPG cryptographic agent and passphrase cache (restricted). Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on GnuPG cryptographic agent (ssh-agent emulation). Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on GnuPG cryptographic agent and passphrase cache. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on debconf communication socket. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on REST API socket for snapd user session agent. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Listening on D-Bus User Message Bus Socket. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Reached target Sockets. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Reached target Basic System. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Reached target Main User Target. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[3258]: Startup finished in 189ms. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[1]: Started User Manager for UID 0. Aug 1 19:24:26 test-cx22-pool-small-static-worker1 systemd[1]: Started Session 15 of User root.Latest available version v1.30.3+k3s1:
`hetzner-k3s create --config vito_botta_test_cluster_config.yaml Validating configuration......configuration seems valid.
=== Creating infrastructure resources === Creating network...done. Creating firewall...done. Creating SSH key...done. Creating placement group test-masters...done. Creating placement group test-small-static-1...done. Creating server test-cx22-master1... Creating server test-cx22-pool-small-static-worker1... ...server test-cx22-master1 created. ...server test-cx22-pool-small-static-worker1 created. Server test-cx22-master1 already exists, skipping. Waiting for successful ssh connectivity with server test-cx22-master1... Server test-cx22-pool-small-static-worker1 already exists, skipping. Waiting for successful ssh connectivity with server test-cx22-pool-small-static-worker1... ...server test-cx22-master1 is now up. ...server test-cx22-pool-small-static-worker1 is now up.
=== Setting up Kubernetes === Deploying k3s to first master test-cx22-master1... [INFO] Using v1.30.3+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s.service [INFO] systemd: Enabling k3s unit [INFO] systemd: Starting k3s Waiting for the control plane to be ready... Saving the kubeconfig file to ~/test/kubeconfig... ...k3s has been deployed to first master test-cx22-master1 and the control plane is up. Deploying k3s to worker test-cx22-pool-small-static-worker1... [INFO] Using v1.30.3+k3s1 as release [INFO] Downloading hash https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/sha256sum-amd64.txt [INFO] Downloading binary https://github.com/k3s-io/k3s/releases/download/v1.30.3+k3s1/k3s [INFO] Verifying binary download [INFO] Installing k3s to /usr/local/bin/k3s [INFO] Skipping installation of SELinux RPM [INFO] Creating /usr/local/bin/kubectl symlink to k3s [INFO] Creating /usr/local/bin/crictl symlink to k3s [INFO] Creating /usr/local/bin/ctr symlink to k3s [INFO] Creating killall script /usr/local/bin/k3s-killall.sh [INFO] Creating uninstall script /usr/local/bin/k3s-agent-uninstall.sh [INFO] env: Creating environment file /etc/systemd/system/k3s-agent.service.env [INFO] systemd: Creating service file /etc/systemd/system/k3s-agent.service [INFO] systemd: Enabling k3s-agent unit [INFO] systemd: Starting k3s-agent`
Aug 1 20:04:23 test-cx22-pool-small-static-worker1 systemd[1]: systemd-hostnamed.service: Deactivated successfully. Aug 1 20:04:24 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:24Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:26 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:26Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:28 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:28Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:30 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:30Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:32 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:32Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:34 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:34Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:36 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:36Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:38 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:38Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:40 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:40Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:42 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:42Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:44 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:44Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:46 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:46Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:48 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:48Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:50 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:50Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:52 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:52Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:54 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:54Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:56 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:56Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:04:58 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:04:58Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:00 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:00Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:02 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:02Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:04 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:04Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:06 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:06Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:08 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:08Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:10 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:10Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:12 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:12Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:14 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:14Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:16 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:16Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:18 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:18Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:20 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:20Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:22 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:22Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:24 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:24Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:26 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:26Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:28 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:28Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:30 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:30Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:32 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:32Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:34 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:34Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:36 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:36Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:38 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:38Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:40 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:40Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:42 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:42Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:44 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:44Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:46 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:46Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:48 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:48Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:50 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:50Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:52 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:52Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:54 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:54Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:56 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:56Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:05:58 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:05:58Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:00 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:00Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:02 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:02Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:04 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:04Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:06 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:06Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:08 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:08Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:10 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:10Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:12 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:12Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:14 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:14Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:16 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:16Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:18 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:18Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:20 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:20Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:22 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:22Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:24 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:24Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:26 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:26Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:28 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:28Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:30 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:30Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:32 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:32Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:34 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:34Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:36 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:36Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:38 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:38Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:40 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:40Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:42 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:42Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:44 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:44Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:46 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:46Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:48 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:48Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:50 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:50Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:52 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:52Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:54 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:54Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:56 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:56Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:06:58 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:06:58Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:00 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:00Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:02 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:02Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:04 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:04Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:06 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:06Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:08 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:08Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:10 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:10Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:12 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:12Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:14 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:14Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:16 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:16Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:18 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:18Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:20 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:20Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:22 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:22Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:24 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:24Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:26 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:26Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:28 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:28Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:30 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:30Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:32 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:32Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:34 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:34Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:36 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:36Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:38 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:38Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:40 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:40Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:42 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:42Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:44 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:44Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:46 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:46Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:48 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:48Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:50 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:50Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:52 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:52Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:54 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:54Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:56 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:56Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:07:58 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:07:58Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:00 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:00Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:02 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:02Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:04 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:04Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:06 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:06Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:08 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:08Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:10 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:10Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:11 test-cx22-pool-small-static-worker1 systemd[1]: session-13.scope: Deactivated successfully. Aug 1 20:08:12 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:12Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:14 test-cx22-pool-small-static-worker1 systemd[1]: Starting Download data for packages that failed at package install time... Aug 1 20:08:14 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:14Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:14 test-cx22-pool-small-static-worker1 systemd[1]: update-notifier-download.service: Deactivated successfully. Aug 1 20:08:14 test-cx22-pool-small-static-worker1 systemd[1]: Finished Download data for packages that failed at package install time. Aug 1 20:08:16 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:16Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:18 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:18Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:20 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:20Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:22 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:22Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:24 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:24Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:26 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:26Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:28 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:28Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:30 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:30Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:32 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:32Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:34 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:34Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:36 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:36Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:38 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:38Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:40 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:40Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:42 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:42Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:44 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:44Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:46 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:46Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:48 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:48Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:50 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:50Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:52 test-cx22-pool-small-static-worker1 k3s[2304]: time="2024-08-01T20:08:52Z" level=info msg="Network policy controller waiting for removal of node.cloudprovider.kubernetes.io/uninitialized taint" Aug 1 20:08:52 test-cx22-pool-small-static-worker1 systemd[1]: Started Session 14 of User root.