Ampere A1 cluster of four c+w - periodical RAM exhaustion

[lacrimal]

Hi, Have ambivalent feelings about k0s. Just now using four machines Ampere A1 (2CPU, 16GB ram) on OL8 ram was exhausted on 2 hosts just after 2h of cluster work without any additional deployment. Both hosts are NoSchedule, NoExecute and unreachable with last memory stats saying RAM was 97% full. Now, after restarting this hosts, there is not much more time, and one of two healthy hosts will die for the same reason and forced reboot is needed. I takes about 2h after fresh install of cluster to exhaust memory on random node making it unreachable. Every time it looks like this just before machine is unreachable:

Tasks: 235 total,   1 running, 234 sleeping,   0 stopped,   0 zombie
%Cpu(s): 15.6 us,  6.5 sy,  0.0 ni,  0.0 id, 77.1 wa,  0.5 hi,  0.3 si,  0.0 st
GiB Mem : 97.0/10.8     [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||   ]
GiB Swap:  0.0/0.0      [                                                                                                    ]

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   5182 kube-ap+  20   0 9626980   8.2g      0 S   0.3  76.5  73:43.35 /var/lib/k0s/bin/kube-apiserver --authorization-mode=Node,RBAC --requestheader-allowed-names=front-proxy-client --client-ca-file=/var/lib/k0s/pki/ca.crt --proxy+
   4214 etcd      20   0   11.4g 314988      0 S   0.0   2.8  26:45.99 /var/lib/k0s/bin/etcd --listen-peer-urls=https://10.0.0.140:2380 --name=instancea --cert-file=/var/lib/k0s/pki/etcd/server.crt --peer-cert-file=/var/lib/k0s/pki+

other node:

top - 08:29:09 up 10:13,  1 user,  load average: 35.80, 34.49, 26.77
Tasks: 234 total,   1 running, 233 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.3 us,  3.1 sy,  0.0 ni,  0.0 id, 94.5 wa,  0.5 hi,  0.5 si,  0.0 st
KiB Mem : 97.2/11298148 [|||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||   ]
KiB Swap:  0.0/0        [                                                                                                    ]

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
   5795 kube-ap+  20   0 9571424   8.4g      0 S   0.0  77.7  80:48.24 /var/lib/k0s/bin/kube-apiserver --service-cluster-ip-range=10.96.0.0/12 --service-account-signing-key-file=/var/lib/k0s/pki/sa.key --featu+
   4230 etcd      20   0   11.4g 611188      0 S   0.3   5.4  41:08.82 /var/lib/k0s/bin/etcd --peer-key-file=/var/lib/k0s/pki/etcd/peer.key --cipher-suites=TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECD+

cluster is created with this k0sctl:

---
apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: k0s-cluster
spec:
  hosts:
  - ssh:
      address: 10.0.0.5
      user: root
      port: 43221
      keyPath: /root/.ssh/id_ed25519
    role: controller+worker
    installFlags:
      - --no-taints
  - ssh:
      address: 10.0.0.10
      user: root
      port: 43221
      keyPath: /root/.ssh/id_ed25519
    role: controller+worker
    installFlags:
      - --no-taints        
  - ssh:
      address: 10.0.0.140
      user: root
      port: 43221
      keyPath: /root/.ssh/id_ed25519
    role: controller+worker
    installFlags:
      - --no-taints
  - ssh:
      address: 10.0.0.145
      user: root
      port: 43221
      keyPath: /root/.ssh/id_ed25519
    role: controller+worker
    installFlags:
      - --no-taints
  k0s:
    version: 1.26.2+k0s.1
    dynamicConfig: false
    config:
      apiVersion: k0s.k0sproject.io/v1beta1
      kind: Cluster
      metadata:
        name: k0s-cluster0
      spec:
        api:
          k0sApiPort: 9443
          port: 6443
          sans:
            - 10.0.0.5
            - 10.0.0.10
            - 10.0.0.140
            - 10.0.0.145
            - 127.0.0.1
        installConfig:
          users:
            etcdUser: etcd
            kineUser: kube-apiserver
            konnectivityUser: konnectivity-server
            kubeAPIserverUser: kube-apiserver
            kubeSchedulerUser: kube-scheduler
        konnectivity:
          adminPort: 8133
          agentPort: 8132
        network:
          kubeProxy:
            disabled: false
            mode: iptables
          kuberouter:
            autoMTU: true
            mtu: 0
            peerRouterASNs: ""
            peerRouterIPs: ""
          podCIDR: 10.244.0.0/16
          provider: kuberouter
          serviceCIDR: 10.96.0.0/12
        podSecurityPolicy:
          defaultPolicy: 00-k0s-privileged
        storage:
          type: etcd
        telemetry:
          enabled: false

Also created cluster of three x86 machines with one controller and can't observe this behaviour. So is my configuration too exotic? Please advise if I can deliver more detail to diagnose it - im fresh in k0s.

[juanluisvaladas]

We haven't seen leaks so my guess is that konnectivity isn't working properly due to the lack of a node balancer and that's causing konnectivity to open an unreasonable number of connections to kube-apiserver.

You can use the node local load balancer right in k0sctl: https://docs.k0sproject.io/head/nllb/#full-example-using-k0sctl

Also it's a very bad idea to have 4 controllers. Etcd should have an ODD number of replicas, etcd considers 3 or 5 optimal so you should have 3 nodes which are controller+worker and one node should be only worker.

If once you have the loadbalancer the problem persists I'll ask for a memory profile so that we can see what's going on internally.

[lacrimal]

Thank you for help, after creating cluster with odd number of controllers and lb it work just fine. Now i know why there is no lb in configuration of two nodes with one controller.