multiple issues running on ubuntu 22

[sigurdblueface]

Environmental Info: K3s Version:

k3s version v1.25.3+k3s1 (f2585c16)
go version go1.19.2

Node(s) CPU architecture, OS, and Version:

Linux oa-node-0 5.15.0-52-generic #58-Ubuntu SMP Thu Oct 13 08:03:55 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux Cluster Configuration:

3 servers, 3 agents Describe the bug:

After update of host OS Ubuntu 20.04 -> 22.04 pods cannot connect to each other between nodes. Even within the same namespace. So NodePorts & Ingress do not properly work also. Steps To Reproduce:

Installed K3s: with custom ansible role. k3s config.yaml:

token: xxx
bind-address: 192.168.152.201
node-ip: 192.168.152.201
tls-san:
  - "oa-cluster.local"
disable: ['traefik', 'servicelb']
kubelet-arg: ['container-runtime=remote', 'container-log-max-files=3', 'container-log-max-size=10Mi']
kube-proxy-arg: [ 'conntrack-max-per-core=0', 'proxy-mode=iptables']

systemd unit:

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/lib/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s server --cluster-init --debug --flannel-external-ip --v 5

.env file:

HTTP_PROXY="http://192.168.152.9:3128"
HTTPS_PROXY="http://192.168.152.9:3128"

NO_PROXY=10.0.0.0/8,localhost,127.0.0.1,192.168.152.201,192.168.152.202,192.168.152.203,192.168.152.204

sysctl:

  - fs.inotify.max_queued_events= 32768
  - fs.inotify.max_user_instances= 2048
  - fs.inotify.max_user_watches= 16384
  - kernel.panic_on_oops= 0
  - kernel.panic= 10
  - net.ipv4.ip_forward= 1
  - vm.overcommit_memory= 1
  - vm.panic_on_oom= 0
  - net.core.somaxconn=65535
  - net.ipv4.ip_local_port_range=1024 65535
  - net.nf_conntrack_max=4194304
  - net.netfilter.nf_conntrack_max=4194304
  - fs.file-max=2097152
  - net.ipv4.tcp_syncookies=1
  - net.ipv4.tcp_max_syn_backlog=65535
  - net.ipv4.conf.cni0.rp_filter=0
  - net.ipv4.conf.ens18.rp_filter=0

  echo 0 > /proc/sys/net/ipv4/conf/flannel.1/rp_filter

Expected behavior:

A perfectly working cluster Actual behavior:

Pods/services cannot interoperate while located on different nodes. One pod cannot ping other pod even within the same namespace. At the same time the service network is accessible. Attempts to access resources via ingresses lead to 5хх errors.

Tried to flush iptables/switch to ipvs - no changes. NodePorts works only if desired pod is located on that particular node i.e my-pod works on node-2, has nodeport 30087 so node-2:30087 - works, and node-1:30087 - not

Additional context / logs:

nothing really meaningful/pointing into any issue in k3s logs

Please advise me what could I also check and try? Thanks in advance.

[brandond]

Does this affect only pods on different nodes, or even pods on the same nodes? Have you tried switching to iptables-legacy instead of iptables-nft? Have you confirmed that the vxlan module is available and loaded? Are you able to find any errors in the k3s or containerd logs?

[sigurdblueface]

It affects pods on different nodes. Pods on the same node can connect.

Yes, I use iptables-legacy.

Regarding vxlan:

lsmod | grep vxlan
vxlan                  86016  0
ip6_udp_tunnel         16384  1 vxlan
udp_tunnel             20480  1 vxlan

I see only errors regarding unability to connect to metrics-server even with --v 5 verbosity.

[brandond]

Did firewalld or ufw get accidentally re-enabled as part of the upgrade? Do you have anything else on the nodes that might be blocking vxlan traffic?

What sort of hardware is this on? There have been issues with hardware checksum offload corrupting vxlan packets on vmware when used with specific kernel versions.

[sigurdblueface]

@brandond, thank you very much, and I am sorry for wasting your time. I've managed to solve the issue. Found out that someone for some unclear reason enabled hypervisor-level firewall in Proxmox UI for nw devices... Unchecked that, performed reboot - everything works perfectly now.