rancher / k3k

Kubernetes in Kubernetes
Apache License 2.0
131 stars 21 forks source link

k3k not working on AKS cluster 1.27.7 #94

Closed xinity closed 5 months ago

xinity commented 6 months ago

i've been trying k3k on my AKS cluster:

OS: linux (amd64)
OS Image: Ubuntu 22.04.4 LTS
Kernel version: 5.15.0-1056-azure
Container runtime: containerd://1.7.7-1
Kubelet version: v1.27.7

tried this yaml :

apiVersion: k3k.io/v1alpha1
kind: Cluster
metadata:
  name: single-server
spec:
  servers: 1
  agents: 3
  token: test
  version: v1.26.0-k3s2
  clusterCIDR: 10.30.0.0/16
  serviceCIDR: 10.31.0.0/16
  clusterDNS: 10.31.0.10
  serverArgs:
  - "--write-kubeconfig-mode=777"
  expose:
    ingress:
      enabled: true
      ingressClassName: "nginx"

the deployment fails because the readiness probe fails ( 127.0.0.1:6443 connection refused)

full log available here: https://pastebin.com/me8fhV8q

any hints on what's happening perhaps ? :)

galal-hussein commented 6 months ago

I can reproduce the problem on AKS cluster:

Readiness:      tcp-socket 127.0.0.1:6443 delay=60s timeout=10s period=10s #success=1 #failure=5

However I can see k3s is up and running normally:

/ # kubectl get nodes
NAME                STATUS   ROLES                       AGE     VERSION
demo-k3k-server-0   Ready    control-plane,etcd,master   9m21s   v1.26.1+k3s1
# netstat -tlnu
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10256         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10258         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:2381          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:2380          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:6444          0.0.0.0:*               LISTEN      
tcp        0      0 127.0.0.1:10010         0.0.0.0:*               LISTEN      
tcp        0      0 10.244.2.4:2379         0.0.0.0:*               LISTEN      
tcp        0      0 10.244.2.4:2380         0.0.0.0:*               LISTEN      
tcp        0      0 :::10250                :::*                    LISTEN      
tcp        0      0 :::6443                 :::*                    LISTEN   
galal-hussein commented 5 months ago

Tested and worked on AKS cluster k3k version: v0.2.1 chart-0.1.4-r1