k0s nodes - cannot communicate between different regions/clouds

EKami commented 1 month ago

Before creating an issue, make sure you've checked the following:

[X] You are running the latest released version of k0s
[X] Make sure you've searched for existing issues, both open and closed
[X] Make sure you've searched for PRs too, a fix might've been merged already
[X] You're looking at docs for the released version, "main" branch docs are usually ahead of released versions.

Platform

Linux 6.8.0-38-generic #38-Ubuntu SMP PREEMPT_DYNAMIC Fri Jun  7 15:25:01 UTC 2024 x86_64 GNU/Linux
PRETTY_NAME="Ubuntu 24.04 LTS"
NAME="Ubuntu"
VERSION_ID="24.04"
VERSION="24.04 LTS (Noble Numbat)"
VERSION_CODENAME=noble
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=noble
LOGO=ubuntu-logo

Version

v1.30.2+k0s.0

Sysinfo

`k0s sysinfo`

Total memory: 3.7 GiB (pass)
Disk space available for /var/lib/k0s: 33.2 GiB (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.8.0-38-generic (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": unknown (warning: insufficient permissions, try with elevated permissions)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

I have a cluster with 4 nodes, 3 are running in the Hetzner cloud, and 1 is running in AWS.

❯ k get nodes -o wide
NAME            STATUS   ROLES    AGE    VERSION       INTERNAL-IP      EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME
ip-172-31-7-1   Ready    <none>   159m   v1.30.2+k0s   172.31.7.1       <none>        Ubuntu 24.04 LTS   6.8.0-1010-aws     containerd://1.7.18
worker-1        Ready    <none>   166m   v1.30.2+k0s   162.55.48.10     <none>        Ubuntu 24.04 LTS   6.8.0-38-generic   containerd://1.7.18
worker-2        Ready    <none>   166m   v1.30.2+k0s   116.203.97.220   <none>        Ubuntu 24.04 LTS   6.8.0-38-generic   containerd://1.7.18
worker-3        Ready    <none>   166m   v1.30.2+k0s   195.201.224.90   <none>        Ubuntu 24.04 LTS   6.8.0-38-generic   containerd://1.7.18

worker-1/worker-2 and worker-3 are all located in the Hetzner cloud. And in AWS I have the node ip-172-31-7-1

In AWS the security group is open to the world so theoretically, accessible from any IP and port:

The issue is that I can easily communicate between worker-1 and worker-2 or worker-3 but it's impossible to communicate between worker-1/2/3 -> ip-172-31-7-1. I also checked the ACL of my VPC, everything seems fine on both the VPC and security groups level.

See below for more details.

Steps to reproduce

Using the following deployment file:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod-listener-worker-1
spec:
  containers:
  - name: test-container
    image: busybox
    command: ["sh", "-c", "echo 'Listening on port 12345' && while true; do nc -lk -p 12345; done"]
  nodeSelector:
    kubernetes.io/hostname: worker-1
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-worker-2
spec:
  containers:
  - name: test-container
    image: busybox
    command:
      - sleep
      - "3600"
  nodeSelector:
    kubernetes.io/hostname: worker-2
---
apiVersion: v1
kind: Pod
metadata:
  name: test-pod-gpu-pod
spec:
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule
  containers:
  - name: test-container
    image: busybox
    command:
      - sleep
      - "3600"
  nodeSelector:
    kubernetes.io/hostname: ip-172-31-7-1

I get:

❯ kubectl get pods -o wide
NAME                         READY   STATUS    RESTARTS      AGE    IP           NODE            NOMINATED NODE   READINESS GATES
test-pod-gpu-pod             1/1     Running   2 (27m ago)   147m   10.244.3.5   ip-172-31-7-1   <none>           <none>
test-pod-listener-worker-1   1/1     Running   0             152m   10.244.2.7   worker-1        <none>           <none>
test-pod-worker-2            1/1     Running   2 (32m ago)   152m   10.244.1.7   worker-2        <none>           <none>

I can send a message with echo "Hello from $(hostname)" | nc 10.244.2.7 12345 from test-pod-worker-2 but not from ip-172-30-0-144. The command actually hangs from ip-172-31-7-1. Ping has the same problem:

❯ kubectl exec -it test-pod-listener-worker-1 -- ping 10.244.1.7
PING 10.244.1.7 (10.244.1.7): 56 data bytes
64 bytes from 10.244.1.7: seq=0 ttl=62 time=1.688 ms
64 bytes from 10.244.1.7: seq=1 ttl=62 time=0.822 ms
64 bytes from 10.244.1.7: seq=2 ttl=62 time=0.868 ms
64 bytes from 10.244.1.7: seq=3 ttl=62 time=0.908 ms
❯ kubectl exec -it test-pod-listener-worker-1 -- ping 10.244.3.5
PING 10.244.3.5 (10.244.3.5): 56 data bytes
<hangs>

Also:

❯ kubectl logs -n kube-system -l k8s-app=kube-router
Defaulted container "kube-router" out of: kube-router, install-cni-bins (init), install-cniconf (init)
Defaulted container "kube-router" out of: kube-router, install-cni-bins (init), install-cniconf (init)
Defaulted container "kube-router" out of: kube-router, install-cni-bins (init), install-cniconf (init)
Defaulted container "kube-router" out of: kube-router, install-cni-bins (init), install-cniconf (init)
time="2024-07-22T04:20:46Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:20:57Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:08Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:19Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:31Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:42Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:54Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:04Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:18Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:29Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T01:47:34Z" level=info msg="Add a peer configuration" Key=162.55.48.10 Topic=Peer
time="2024-07-22T01:47:34Z" level=info msg="Add a peer configuration" Key=116.203.97.220 Topic=Peer
time="2024-07-22T01:47:34Z" level=info msg="Add a peer configuration" Key=195.201.224.90 Topic=Peer
I0722 01:47:34.549468    2705 network_policy_controller.go:175] Starting network policy controller full sync goroutine
time="2024-07-22T02:57:25Z" level=info msg="Can't find configuration for a new passive connection" Key=5.75.130.232 Topic=Peer
time="2024-07-22T03:01:17Z" level=warning msg="Closed an accepted connection" Key=162.55.48.10 State=BGP_FSM_IDLE Topic=Peer
time="2024-07-22T03:02:23Z" level=warning msg="Closed an accepted connection" Key=162.55.48.10 State=BGP_FSM_IDLE Topic=Peer
time="2024-07-22T03:03:00Z" level=warning msg="Closed an accepted connection" Key=162.55.48.10 State=BGP_FSM_IDLE Topic=Peer
time="2024-07-22T03:10:13Z" level=warning msg="Closed an accepted connection" Key=162.55.48.10 State=BGP_FSM_IDLE Topic=Peer
time="2024-07-22T04:17:56Z" level=info msg="Can't find configuration for a new passive connection" Key=172.170.167.157 Topic=Peer
time="2024-07-22T04:20:32Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:20:42Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:20:54Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:05Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:20Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:34Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:46Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:57Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:08Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:22Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:20:37Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:20:50Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:03Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:14Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:25Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:37Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:21:49Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:02Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:12Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer
time="2024-07-22T04:22:24Z" level=info msg="Can't find configuration for a new passive connection" Key=34.223.252.141 Topic=Peer

I'm not sure what I'm missing here. I have tried with an instance in a provider different from AWS (lambdalabs) and I'm facing the same issue. For that reason I think it's coming from my k0s or k8s configuration.

Expected behavior

For my pods to communicate between themselves

Actual behavior

Not all pods are able to communicate with each other

Additional context

❯ kubectl get networkpolicies --all-namespaces
No resources found

On all of my nodes with direct SSH access:

$ sudo ufw status
Status: inactive

from the worker-1 instance:

#!/bin/bash
echo "Test TCP on worker-2"
for port in 2380 6443 179 10250 9443 8132 8133; do
  nc -zvw3 162.55.45.10 $port
done

echo "Test UDP on worker-2"
nc -zvw3 -u 162.55.45.10 4789 && echo "UDP port 4789 is open" || echo "UDP port 4789 is closed"

echo "Test TCP on ip-172-30-0-144"
for port in 2380 6443 179 10250 9443 8132 8133; do
  nc -zvw3 34.223.252.141 $port
done

echo "Test UDP on ip-172-30-0-144"
nc -zvw3 -u 34.223.252.141 4789 && echo "UDP port 4789 is open" || echo "UDP port 4789 is closed"

the results are:

Test TCP on worker-2
nc: connect to 162.55.48.10 port 2380 (tcp) failed: Connection refused
nc: connect to 162.55.48.10 port 6443 (tcp) failed: Connection refused
Connection to 162.55.48.10 179 port [tcp/bgp] succeeded!
Connection to 162.55.48.10 10250 port [tcp/*] succeeded!
nc: connect to 162.55.48.10 port 9443 (tcp) failed: Connection refused
nc: connect to 162.55.48.10 port 8132 (tcp) failed: Connection refused
nc: connect to 162.55.48.10 port 8133 (tcp) failed: Connection refused
Test UDP on worker-2
UDP port 4789 is closed
Test TCP on ip-172-30-0-144
nc: connect to 34.223.252.141 port 2380 (tcp) failed: Connection refused
nc: connect to 34.223.252.141 port 6443 (tcp) failed: Connection refused
Connection to 34.223.252.141 179 port [tcp/bgp] succeeded!
Connection to 34.223.252.141 10250 port [tcp/*] succeeded!
nc: connect to 34.223.252.141 port 9443 (tcp) failed: Connection refused
nc: connect to 34.223.252.141 port 8132 (tcp) failed: Connection refused
nc: connect to 34.223.252.141 port 8133 (tcp) failed: Connection refused
Test UDP on ip-172-30-0-144
UDP port 4789 is closed

List of ports taken from the docs. This is my /etc/k0s/k0s.yaml configuration file on my controller node:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: null
  name: k0s
spec:
  api:
    address: 5.75.130.232
    k0sApiPort: 9443
    port: 6443
    sans:
    - 5.75.130.232
    - 2a01:4f8:1c1e:b6a0::1
    - fe80::9400:3ff:fe8c:feae
  controllerManager: {}
  extensions:
    helm:
      charts: null
      concurrencyLevel: 5
      repositories: null
    storage:
      create_default_storage_class: false
      type: external_storage
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    calico: null
    clusterDomain: cluster.local
    dualStack: {}
    kubeProxy:
      iptables:
        minSyncPeriod: 0s
        syncPeriod: 0s
      ipvs:
        minSyncPeriod: 0s
        syncPeriod: 0s
        tcpFinTimeout: 0s
        tcpTimeout: 0s
        udpTimeout: 0s
      metricsBindAddress: 0.0.0.0:10249
      mode: iptables
    kuberouter:
      autoMTU: true
      hairpin: Enabled
      ipMasq: false
      metricsPort: 8080
      mtu: 0
      peerRouterASNs: ""
      peerRouterIPs: ""
    nodeLocalLoadBalancing:
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
    podCIDR: 10.244.0.0/16
    provider: kuberouter
    serviceCIDR: 10.96.0.0/12
  scheduler: {}
  storage:
    etcd:
      externalCluster: null
      peerAddress: 5.75.130.232
    type: etcd
  telemetry:
    enabled: true

juanluisvaladas commented 1 month ago

By default, kuberouter sends all the traffic to the pods directly just using normal layer 3 routing. In other words, the package from the node A goes to the node B using the podIP.

Some infrastructure providers do not allow this, most certainly you can get this to work fix this this by setting the option: .spec.network.kuberouter.ipMasq to true.

EKami commented 1 month ago

By default, kuberouter sends all the traffic to the pods directly just using normal layer 3 routing. In other words, the package from the node A goes to the node B using the podIP.

Some infrastructure providers do not allow this, most certainly you can get this to work fix this this by setting the option: .spec.network.kuberouter.ipMasq to true.

Thank you for the pointer, but unfortunately it did not work. I also tried the following, but it didn't work either:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: null
  name: k0s
spec:
  api:
    address: 116.203.97.220
    k0sApiPort: 9443
    port: 6443
    sans:
    - 116.203.97.220
  controllerManager: {}
  extensions:
    helm:
      charts: null
      concurrencyLevel: 5
      repositories: null
    storage:
      create_default_storage_class: false
      type: external_storage
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    serviceCIDR: 10.96.0.0/12
    podCIDR: 10.244.0.0/16
    provider: calico
    calico:
      mode: ipip
      ipipMode: Always
    nodeLocalLoadBalancing:
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
  scheduler: {}
  storage:
    etcd:
      externalCluster: null
      peerAddress: 116.203.97.220
    type: etcd
  telemetry:
    enabled: true

In both cases I started with a fresh cluster to make sure the previous configuration wasn't applied on existing nodes.

❯ kubectl logs -n kube-system -l k8s-app=calico-node
Defaulted container "calico-node" out of: calico-node, install-cni (init)
Defaulted container "calico-node" out of: calico-node, install-cni (init)
Defaulted container "calico-node" out of: calico-node, install-cni (init)
2024-07-22 13:54:23.303 [INFO][86] felix/summary.go 100: Summarising 10 dataplane reconciliation loops over 1m2.9s: avg=17ms longest=37ms ()
2024-07-22 13:54:23.413 [INFO][84] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface eth0: 5.75.130.232/32
2024-07-22 13:55:23.426 [INFO][84] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface eth0: 5.75.130.232/32
2024-07-22 13:55:26.198 [INFO][86] felix/summary.go 100: Summarising 7 dataplane reconciliation loops over 1m2.9s: avg=12ms longest=18ms (resync-ipsets-v4)
2024-07-22 13:56:14.880 [INFO][86] felix/int_dataplane.go 1954: Received *proto.HostMetadataV4V6Update update from calculation graph msg=hostname:"worker-1" ipv4_addr:"5.75.130.232/32" labels:<key:"beta.kubernetes.io/arch" value:"amd64" > labels:<key:"beta.kubernetes.io/os" value:"linux" > labels:<key:"kubernetes.io/arch" value:"amd64" > labels:<key:"kubernetes.io/hostname" value:"worker-1" > labels:<key:"kubernetes.io/os" value:"linux" >
2024-07-22 13:56:15.527 [INFO][86] felix/int_dataplane.go 1954: Received *proto.HostMetadataV4V6Update update from calculation graph msg=hostname:"worker-2" ipv4_addr:"162.55.48.10/32" labels:<key:"beta.kubernetes.io/arch" value:"amd64" > labels:<key:"beta.kubernetes.io/os" value:"linux" > labels:<key:"kubernetes.io/arch" value:"amd64" > labels:<key:"kubernetes.io/hostname" value:"worker-2" > labels:<key:"kubernetes.io/os" value:"linux" >
2024-07-22 13:56:20.641 [INFO][86] felix/int_dataplane.go 1954: Received *proto.HostMetadataV4V6Update update from calculation graph msg=hostname:"ip-172-31-12-4" ipv4_addr:"172.31.12.4/20" labels:<key:"beta.kubernetes.io/arch" value:"amd64" > labels:<key:"beta.kubernetes.io/os" value:"linux" > labels:<key:"gpu-memory-MiB" value:"15360" > labels:<key:"kubernetes.io/arch" value:"amd64" > labels:<key:"kubernetes.io/hostname" value:"ip-172-31-12-4" > labels:<key:"kubernetes.io/os" value:"linux" > labels:<key:"nvidia.com/gpu.present" value:"true" >
2024-07-22 13:56:23.428 [INFO][84] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface eth0: 5.75.130.232/32
2024-07-22 13:56:28.955 [INFO][86] felix/summary.go 100: Summarising 12 dataplane reconciliation loops over 1m2.8s: avg=11ms longest=36ms ()
2024-07-22 13:57:23.443 [INFO][84] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface eth0: 5.75.130.232/32
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 50701)
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 48971)
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 55499)
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 54229)
2024-07-22 13:57:23.174 [INFO][88] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface eth0: 162.55.48.10/32
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 34559)
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 47097)
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 54185)
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 39407)
bird: BGP: Unexpected connect from unknown address 35.91.61.120 (port 37163)
2024-07-22 13:54:31.634 [INFO][97] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface ens5: 172.31.12.4/20
2024-07-22 13:54:53.332 [INFO][102] felix/summary.go 100: Summarising 8 dataplane reconciliation loops over 1m3.7s: avg=4ms longest=11ms ()
2024-07-22 13:55:31.645 [INFO][97] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface ens5: 172.31.12.4/20
2024-07-22 13:55:53.752 [INFO][102] felix/summary.go 100: Summarising 8 dataplane reconciliation loops over 1m0.4s: avg=4ms longest=10ms ()
2024-07-22 13:56:14.964 [INFO][102] felix/int_dataplane.go 1954: Received *proto.HostMetadataV4V6Update update from calculation graph msg=hostname:"worker-1" ipv4_addr:"5.75.130.232/32" labels:<key:"beta.kubernetes.io/arch" value:"amd64" > labels:<key:"beta.kubernetes.io/os" value:"linux" > labels:<key:"kubernetes.io/arch" value:"amd64" > labels:<key:"kubernetes.io/hostname" value:"worker-1" > labels:<key:"kubernetes.io/os" value:"linux" >
2024-07-22 13:56:15.613 [INFO][102] felix/int_dataplane.go 1954: Received *proto.HostMetadataV4V6Update update from calculation graph msg=hostname:"worker-2" ipv4_addr:"162.55.48.10/32" labels:<key:"beta.kubernetes.io/arch" value:"amd64" > labels:<key:"beta.kubernetes.io/os" value:"linux" > labels:<key:"kubernetes.io/arch" value:"amd64" > labels:<key:"kubernetes.io/hostname" value:"worker-2" > labels:<key:"kubernetes.io/os" value:"linux" >
2024-07-22 13:56:20.722 [INFO][102] felix/int_dataplane.go 1954: Received *proto.HostMetadataV4V6Update update from calculation graph msg=hostname:"ip-172-31-12-4" ipv4_addr:"172.31.12.4/20" labels:<key:"beta.kubernetes.io/arch" value:"amd64" > labels:<key:"beta.kubernetes.io/os" value:"linux" > labels:<key:"gpu-memory-MiB" value:"15360" > labels:<key:"kubernetes.io/arch" value:"amd64" > labels:<key:"kubernetes.io/hostname" value:"ip-172-31-12-4" > labels:<key:"kubernetes.io/os" value:"linux" > labels:<key:"nvidia.com/gpu.present" value:"true" >
2024-07-22 13:56:31.645 [INFO][97] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface ens5: 172.31.12.4/20
2024-07-22 13:56:58.649 [INFO][102] felix/summary.go 100: Summarising 12 dataplane reconciliation loops over 1m4.9s: avg=4ms longest=9ms (resync-nat-v4,resync-raw-v4)
2024-07-22 13:57:31.656 [INFO][97] monitor-addresses/autodetection_methods.go 103: Using autodetected IPv4 address on interface ens5: 172.31.12.4/20

EKami commented 1 month ago

This didn't help either :( :

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: null
  name: k0s
spec:
  api:
    address: 116.203.97.220
    k0sApiPort: 9443
    port: 6443
    sans:
    - 116.203.97.220
  controllerManager: {}
  extensions:
    helm:
      charts: null
      concurrencyLevel: 5
      repositories: null
    storage:
      create_default_storage_class: false
      type: external_storage
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    serviceCIDR: 10.96.0.0/12
    podCIDR: 10.244.0.0/16
    provider: calico
    calico:
      mode: vxlan
      vxlanPort: 4789
      crossSubnet: true
    nodeLocalLoadBalancing:
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
  scheduler: {}
  storage:
    etcd:
      externalCluster: null
      peerAddress: 116.203.97.220
    type: etcd
  telemetry:
    enabled: true

twz123 commented 1 month ago

This kind of setup requires source/destination checking to be turned off for your EC2 instance. Can you verify that this is the case? In the AWS Console, go to EC2, then select Actions -> Networking -> Change source/destination check. Check that the "Stop" box is checked.

change_source-destination_check

You can verify the status of the Source/Destination checks via the AWS CLI, too:

aws ec2 describe-instances --query "Reservations[].Instances[].[Tags[?Key=='Name'].Value | [0], InstanceId, SourceDestCheck]"

This needs to print false for the value of SourceDestCheck.

EKami commented 1 month ago

Thanks a lot for your help. I did exactly that, and the AWS command returns:

[
    [
        "gpu1",
        "i-0c5f8ba18d7b8616a",
        false
    ]
]

With the following k0s configuration file:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: null
  name: k0s
spec:
  api:
    address: 116.203.97.220
    k0sApiPort: 9443
    port: 6443
    sans:
      - 116.203.97.220
  controllerManager: {}
  extensions:
    helm:
      charts: null
      concurrencyLevel: 5
      repositories: null
    storage:
      create_default_storage_class: false
      type: external_storage
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    provider: kuberouter
    podCIDR: 10.244.0.0/16
    serviceCIDR: 10.96.0.0/12
    clusterDomain: cluster.local
    kubeProxy:
      iptables:
        minSyncPeriod: 0s
        syncPeriod: 0s
      ipvs:
        minSyncPeriod: 0s
        syncPeriod: 0s
        tcpFinTimeout: 0s
        tcpTimeout: 0s
        udpTimeout: 0s
      metricsBindAddress: 0.0.0.0:10249
      mode: iptables
    kuberouter:
      ipMasq: true
      autoMTU: true
      hairpin: Enabled
      metricsPort: 8080
    nodeLocalLoadBalancing:
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
  scheduler: {}
  storage:
    etcd:
      externalCluster: null
      peerAddress: 116.203.97.220
    type: etcd
  telemetry:
    enabled: true

But it still doesn't work :( . I guess the only way to bypass this limitation in any cloud is to use something like tailscale.

Is there a CNI that allows bypassing these limitations without a solution like tailscale?

EKami commented 1 month ago

I finally got it working with Cilium as a CNI coupled with Wireguard tunneling. Here is my final k0s.yaml configuration:

apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
metadata:
  creationTimestamp: null
  name: k0s
spec:
  api:
    address: {{ ansible_host }}
    k0sApiPort: 9443
    port: 6443
    sans:
      - {{ ansible_host }}
  controllerManager: {}
  extensions:
    helm:
      charts: null
      concurrencyLevel: 5
      repositories: null
    storage:
      create_default_storage_class: false
      type: external_storage
  installConfig:
    users:
      etcdUser: etcd
      kineUser: kube-apiserver
      konnectivityUser: konnectivity-server
      kubeAPIserverUser: kube-apiserver
      kubeSchedulerUser: kube-scheduler
  konnectivity:
    adminPort: 8133
    agentPort: 8132
  network:
    provider: custom
    kubeProxy:
      disabled: true
    nodeLocalLoadBalancing:
      envoyProxy:
        apiServerBindPort: 7443
        konnectivityServerBindPort: 7132
      type: EnvoyProxy
  scheduler: {}
  storage:
    etcd:
      externalCluster: null
      peerAddress: {{ ansible_host }}
    type: etcd
  telemetry:
    enabled: true

And here is how I installed cilium with Ansible:

- name: Check if Cilium is already installed
  stat:
    path: /etc/ansible/facts.d/cilium_installed.fact
  register: cilium_installed_fact

- block:
  - name: Ensure /etc/ansible/facts.d directory exists
    file:
      become: yes
      path: /etc/ansible/facts.d
      state: directory
      mode: '0755'

  - name: Get the latest Cilium CLI version
    shell: curl -s https://raw.githubusercontent.com/cilium/cilium-cli/main/stable.txt
    register: cilium_cli_version

  - name: Determine the CLI architecture
    set_fact:
      cli_arch: "{{ 'arm64' if ansible_architecture == 'aarch64' else 'amd64' }}"

  - name: Download the Cilium CLI tarball and its checksum
    get_url:
      url: "https://github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_version.stdout }}/cilium-linux-{{ cli_arch }}.tar.gz"
      dest: "/tmp/cilium-linux-{{ cli_arch }}.tar.gz"
    register: cilium_cli_tarball

  - name: Download the Cilium CLI checksum
    get_url:
      url: "https://github.com/cilium/cilium-cli/releases/download/{{ cilium_cli_version.stdout }}/cilium-linux-{{ cli_arch }}.tar.gz.sha256sum"
      dest: "/tmp/cilium-linux-{{ cli_arch }}.tar.gz.sha256sum"

  - name: Verify the checksum
    shell: sha256sum --check /tmp/cilium-linux-{{ cli_arch }}.tar.gz.sha256sum
    args:
      chdir: /tmp

  - name: Extract the Cilium CLI tarball
    become: yes
    unarchive:
      src: "/tmp/cilium-linux-{{ cli_arch }}.tar.gz"
      dest: /usr/local/bin
      remote_src: yes

  - name: Clean up the downloaded files
    file:
      path: "/tmp/cilium-linux-{{ cli_arch }}.tar.gz{{ item }}"
      state: absent
    loop:
      - ''
      - '.sha256sum'

  - name: Install Cilium
    become: yes
    shell: "cilium install --set encryption.enabled=true --set encryption.type=wireguard"
    environment:
      KUBECONFIG: "{{ ansible_facts.env.HOME }}/.kube/config"

  - name: Check Cilium status and wait until it is ready
    shell: "cilium status --wait"
    environment:
      KUBECONFIG: "{{ ansible_facts.env.HOME }}/.kube/config"
    retries: 5
    delay: 10
    register: cilium_status
    until: cilium_status.rc == 0

  - name: Test nodes connectivity with Cilium
    shell: "cilium connectivity test --test '/node-to-node/'"
    environment:
      KUBECONFIG: "{{ ansible_facts.env.HOME }}/.kube/config"
    retries: 5
    delay: 10
    register: connectivity_test
    until: connectivity_test.rc == 0

  - name: Test pods connectivity with Cilium
    shell: "cilium connectivity test --test '/pod-to-pod/'"
    environment:
      KUBECONFIG: "{{ ansible_facts.env.HOME }}/.kube/config"
    retries: 5
    delay: 10
    register: connectivity_test
    until: connectivity_test.rc == 0

  - name: Create a fact file indicating Cilium installation
    become: yes
    copy:
      dest: /etc/ansible/facts.d/cilium_installed.fact
      content: |
        [cilium]
        installed = true
  when: cilium_installed_fact.stat.exists == false

k0sproject / k0s