search

k0sproject / k0s

k0s - The Zero Friction Kubernetes
https://docs.k0sproject.io
Other
3.65k stars 361 forks source link

dial unix /run/k0s/kine/kine.sock:2379: connect: no such file or directory #5079

Open fruworg opened 2 weeks ago

fruworg commented 2 weeks ago

Before creating an issue, make sure you've checked the following:

Platform

Linux 6.1.0-25-cloud-arm64 #1 SMP Debian 6.1.106-3 (2024-08-26) aarch64 GNU/Linux
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

Version

v1.31.1+k0s.0

Sysinfo

`k0s sysinfo` 
Total memory: 11.7 GiB (pass)
File system of /var/lib/k0s: ext4 (pass)
Disk space available for /var/lib/k0s: 38.1 GiB (pass)
Relative disk space available for /var/lib/k0s: 77% (pass)
Name resolution: localhost: [::1 127.0.0.1] (pass)
Operating system: Linux (pass)
  Linux kernel release: 6.1.0-25-cloud-arm64 (pass)
  Max. file descriptors per process: current: 1048576 / max: 1048576 (pass)
  AppArmor: active (pass)
  Executable in PATH: modprobe: /usr/sbin/modprobe (pass)
  Executable in PATH: mount: /usr/bin/mount (pass)
  Executable in PATH: umount: /usr/bin/umount (pass)
  /proc file system: mounted (0x9fa0) (pass)
  Control Groups: version 2 (pass)
    cgroup controller "cpu": available (is a listed root controller) (pass)
    cgroup controller "cpuacct": available (via cpu in version 2) (pass)
    cgroup controller "cpuset": available (is a listed root controller) (pass)
    cgroup controller "memory": available (is a listed root controller) (pass)
    cgroup controller "devices": available (device filters attachable) (pass)
    cgroup controller "freezer": available (cgroup.freeze exists) (pass)
    cgroup controller "pids": available (is a listed root controller) (pass)
    cgroup controller "hugetlb": available (is a listed root controller) (pass)
    cgroup controller "blkio": available (via io in version 2) (pass)
  CONFIG_CGROUPS: Control Group support: built-in (pass)
    CONFIG_CGROUP_FREEZER: Freezer cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_PIDS: PIDs cgroup subsystem: built-in (pass)
    CONFIG_CGROUP_DEVICE: Device controller for cgroups: built-in (pass)
    CONFIG_CPUSETS: Cpuset support: built-in (pass)
    CONFIG_CGROUP_CPUACCT: Simple CPU accounting cgroup subsystem: built-in (pass)
    CONFIG_MEMCG: Memory Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_HUGETLB: HugeTLB Resource Controller for Control Groups: built-in (pass)
    CONFIG_CGROUP_SCHED: Group CPU scheduler: built-in (pass)
      CONFIG_FAIR_GROUP_SCHED: Group scheduling for SCHED_OTHER: built-in (pass)
        CONFIG_CFS_BANDWIDTH: CPU bandwidth provisioning for FAIR_GROUP_SCHED: built-in (pass)
    CONFIG_BLK_CGROUP: Block IO controller: built-in (pass)
  CONFIG_NAMESPACES: Namespaces support: built-in (pass)
    CONFIG_UTS_NS: UTS namespace: built-in (pass)
    CONFIG_IPC_NS: IPC namespace: built-in (pass)
    CONFIG_PID_NS: PID namespace: built-in (pass)
    CONFIG_NET_NS: Network namespace: built-in (pass)
  CONFIG_NET: Networking support: built-in (pass)
    CONFIG_INET: TCP/IP networking: built-in (pass)
      CONFIG_IPV6: The IPv6 protocol: built-in (pass)
    CONFIG_NETFILTER: Network packet filtering framework (Netfilter): built-in (pass)
      CONFIG_NETFILTER_ADVANCED: Advanced netfilter configuration: built-in (pass)
      CONFIG_NF_CONNTRACK: Netfilter connection tracking support: module (pass)
      CONFIG_NETFILTER_XTABLES: Netfilter Xtables support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_REDIRECT: REDIRECT target support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_COMMENT: "comment" match support: module (pass)
        CONFIG_NETFILTER_XT_MARK: nfmark target and match support: module (pass)
        CONFIG_NETFILTER_XT_SET: set target and match support: module (pass)
        CONFIG_NETFILTER_XT_TARGET_MASQUERADE: MASQUERADE target support: module (pass)
        CONFIG_NETFILTER_XT_NAT: "SNAT and DNAT" targets support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: "addrtype" address type match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_CONNTRACK: "conntrack" connection tracking match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_MULTIPORT: "multiport" Multiple port match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_RECENT: "recent" match support: module (pass)
        CONFIG_NETFILTER_XT_MATCH_STATISTIC: "statistic" match support: module (pass)
      CONFIG_NETFILTER_NETLINK: module (pass)
      CONFIG_NF_NAT: module (pass)
      CONFIG_IP_SET: IP set support: module (pass)
        CONFIG_IP_SET_HASH_IP: hash:ip set support: module (pass)
        CONFIG_IP_SET_HASH_NET: hash:net set support: module (pass)
      CONFIG_IP_VS: IP virtual server support: module (pass)
        CONFIG_IP_VS_NFCT: Netfilter connection tracking: built-in (pass)
        CONFIG_IP_VS_SH: Source hashing scheduling: module (pass)
        CONFIG_IP_VS_RR: Round-robin scheduling: module (pass)
        CONFIG_IP_VS_WRR: Weighted round-robin scheduling: module (pass)
      CONFIG_NF_CONNTRACK_IPV4: IPv4 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_REJECT_IPV4: IPv4 packet rejection: module (pass)
      CONFIG_NF_NAT_IPV4: IPv4 NAT: unknown (warning)
      CONFIG_IP_NF_IPTABLES: IP tables support: module (pass)
        CONFIG_IP_NF_FILTER: Packet filtering: module (pass)
          CONFIG_IP_NF_TARGET_REJECT: REJECT target support: module (pass)
        CONFIG_IP_NF_NAT: iptables NAT support: module (pass)
        CONFIG_IP_NF_MANGLE: Packet mangling: module (pass)
      CONFIG_NF_DEFRAG_IPV4: module (pass)
      CONFIG_NF_CONNTRACK_IPV6: IPv6 connetion tracking support (required for NAT): unknown (warning)
      CONFIG_NF_NAT_IPV6: IPv6 NAT: unknown (warning)
      CONFIG_IP6_NF_IPTABLES: IP6 tables support: module (pass)
        CONFIG_IP6_NF_FILTER: Packet filtering: module (pass)
        CONFIG_IP6_NF_MANGLE: Packet mangling: module (pass)
        CONFIG_IP6_NF_NAT: ip6tables NAT support: module (pass)
      CONFIG_NF_DEFRAG_IPV6: module (pass)
    CONFIG_BRIDGE: 802.1d Ethernet Bridging: module (pass)
      CONFIG_LLC: module (pass)
      CONFIG_STP: module (pass)
  CONFIG_EXT4_FS: The Extended 4 (ext4) filesystem: built-in (pass)
  CONFIG_PROC_FS: /proc file system support: built-in (pass)

What happened?

Hi. When I try to deploy the cluster using k0sctl, k0s fails to start. Error: dial unix /run/k0s/kine/kine.sock:2379: connect: no such file or directory

Steps to reproduce

  1. copy k0sctl.yaml
  2. k0sctl apply --config k0sctl.yaml
  3. k0s server

Expected behavior

Works without error.

Actual behavior

It's not working

Screenshots and logs

k0s server

INFO[2024-10-06 11:35:53] using api address: 100.93.0.158              
INFO[2024-10-06 11:35:53] using listen port: 6443                      
INFO[2024-10-06 11:35:53] using sans: [100.93.0.158 alpha.<DOMAIN-REDACTED> 100.93.209.186 bravo.<DOMAIN-REDACTED> 127.0.0.1] 
INFO[2024-10-06 11:35:53] DNS address: 10.96.0.10                      
INFO[2024-10-06 11:35:53] using storage backend kine                   
WARN[2024-10-06 11:35:53] Files with key material for kube-apiserver user will be owned by root  error="failed to lookup UID for \"kube-apiserver\": user does not exist (exit status 1: id: ‘kube-apiserver’: no such user)"
INFO[2024-10-06 11:35:53] generate received request                     component=cfssl
INFO[2024-10-06 11:35:53] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:53] generating key: rsa-2048                      component=cfssl
INFO[2024-10-06 11:35:53] generate received request                     component=cfssl
INFO[2024-10-06 11:35:53] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:53] generating key: rsa-2048                      component=cfssl
INFO[2024-10-06 11:35:53] generate received request                     component=cfssl
INFO[2024-10-06 11:35:53] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:53] generating key: rsa-2048                      component=cfssl
INFO[2024-10-06 11:35:53] generate received request                     component=cfssl
INFO[2024-10-06 11:35:53] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:53] generating key: rsa-2048                      component=cfssl
INFO[2024-10-06 11:35:53] generate received request                     component=cfssl
INFO[2024-10-06 11:35:53] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:53] generating key: rsa-2048                      component=cfssl
INFO[2024-10-06 11:35:53] generate received request                     component=cfssl
INFO[2024-10-06 11:35:53] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:53] generating key: rsa-2048                      component=cfssl
WARN[2024-10-06 11:35:53] Files with key material for konnectivity-server user will be owned by root  error="failed to lookup UID for \"konnectivity-server\": user does not exist (exit status 1: id: ‘konnectivity-server’: no such user)"
INFO[2024-10-06 11:35:53] generate received request                     component=cfssl
INFO[2024-10-06 11:35:53] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:53] generating key: rsa-2048                      component=cfssl
WARN[2024-10-06 11:35:54] Files with key material for kube-scheduler user will be owned by root  error="failed to lookup UID for \"kube-scheduler\": user does not exist (exit status 1: id: ‘kube-scheduler’: no such user)"
INFO[2024-10-06 11:35:54] generate received request                     component=cfssl
INFO[2024-10-06 11:35:54] received CSR                                  component=cfssl
INFO[2024-10-06 11:35:54] generating key: rsa-2048                      component=cfssl
INFO[2024-10-06 11:35:55] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:55] signed certificate with serial number 165034088466474029612502487541600685615664972440  component=cfssl
INFO[2024-10-06 11:35:55] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:55] signed certificate with serial number 461781381686173583495964129565178613475985762900  component=cfssl
INFO[2024-10-06 11:35:55] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:55] signed certificate with serial number 477252895822943348312180408477736361127442553371  component=cfssl
INFO[2024-10-06 11:35:55] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:55] signed certificate with serial number 680474221602125930221492967835892021942151930746  component=cfssl
INFO[2024-10-06 11:35:55] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:55] signed certificate with serial number 248113658265481034429006418211240835736125477761  component=cfssl
INFO[2024-10-06 11:35:55] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:55] signed certificate with serial number 568973617032074489528896356302337073379027957732  component=cfssl
INFO[2024-10-06 11:35:55] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:55] signed certificate with serial number 180490754112672499911836309653984377747754321492  component=cfssl
INFO[2024-10-06 11:35:56] encoded CSR                                   component=cfssl
INFO[2024-10-06 11:35:56] signed certificate with serial number 22734747237181592705406339184513874154807757483  component=cfssl
INFO[2024-10-06 11:35:56] initializing Kine                            
INFO[2024-10-06 11:35:56] initializing Konnectivity                    
INFO[2024-10-06 11:35:56] initializing APIServer                       
INFO[2024-10-06 11:35:56] initializing K0sControllersLeaseCounter      
INFO[2024-10-06 11:35:56] initializing LeasePool                       
INFO[2024-10-06 11:35:56] initializing Manager                         
INFO[2024-10-06 11:35:56] initializing K0SControlAPI                   
INFO[2024-10-06 11:35:56] initializing CSRApprover                     
INFO[2024-10-06 11:35:56] initializing Status                          
INFO[2024-10-06 11:35:56] Listening address /run/k0s/status.sock        component=status
INFO[2024-10-06 11:35:56] initializing kine                            
WARN[2024-10-06 11:35:56] Running kine as root                          error="failed to lookup UID for \"kube-apiserver\": user does not exist (exit status 1: id: ‘kube-apiserver’: no such user)"
WARN[2024-10-06 11:35:56] Running konnectivity as root                  error="failed to lookup UID for \"konnectivity-server\": user does not exist (exit status 1: id: ‘konnectivity-server’: no such user)"
INFO[2024-10-06 11:35:56] Staging '/var/lib/k0s/bin/konnectivity-server' 
WARN[2024-10-06 11:35:56] Running Kubernetes API server as root         error="failed to lookup UID for \"kube-apiserver\": user does not exist (exit status 1: id: ‘kube-apiserver’: no such user)"
INFO[2024-10-06 11:35:56] Staging '/var/lib/k0s/bin/kube-apiserver'    
INFO[2024-10-06 11:35:56] Staging '/var/lib/k0s/bin/kine'              
INFO[2024-10-06 11:35:56] starting Kine                                
INFO[2024-10-06 11:35:56] Starting kine                                
INFO[2024-10-06 11:35:56] Starting to supervise                         component=kine
W1006 11:35:56.065287   73254 logging.go:55] [core] [Channel #1 SubChannel #2]grpc: addrConn.createTransport failed to connect to {Addr: "unix:/run/k0s/kine/kine.sock:2379", ServerName: "kine.sock:2379", }. Err: connection error: desc = "transport: Error while dialing: dial unix /run/k0s/kine/kine.sock:2379: connect: no such file or directory"
INFO[2024-10-06 11:35:56] Started successfully, go nuts pid 73267       component=kine
INFO[2024-10-06 11:35:56] time="2024-10-06T11:35:56.087802393Z" level=info msg="metrics server is starting to listen at :2380"  component=kine stream=stderr
INFO[2024-10-06 11:35:56] time="2024-10-06T11:35:56.088202915Z" level=fatal msg="error creating the metrics listener: listen tcp :2380: bind: address already in use"  component=kine stream=stderr
WARN[2024-10-06 11:35:56] Failed to wait for process                    component=kine error="exit status 1"

Additional context

k0sctl.yaml

apiVersion: k0sctl.k0sproject.io/v1beta1
kind: Cluster
metadata:
  name: <NAME-REDACTED>
spec:
  hosts:
  - role: controller+worker
    ssh:
      address: alpha.<DOMAIN-REDACTED>
      user: root
      keyPath: ~/.ssh/id_rsa
    privateAddress: 100.93.209.186
  - role: controller+worker
    ssh:
      address: bravo.<DOMAIN-REDACTED>
      user: root
      keyPath: ~/.ssh/id_rsa
    privateAddress: 100.93.0.158
  k0s:
    version: 1.31.1+k0s.0
    config:
      spec:
        storage:
          type: kine
          kine:
            dataSource: mysql://k0s:<PASSWORD-REDACTED>@tcp(127.0.0.1:4000)/k0s
        telemetry:
          enabled: false

k0s.yaml

# generated-by-k0sctl 2024-10-03T16:31:43+02:00
apiVersion: k0s.k0sproject.io/v1beta1
kind: ClusterConfig
spec:
  api:
    address: 100.93.0.158
    sans:
    - 100.93.0.158
    - alpha.<DOMAIN-REDACTED>
    - 100.93.209.186
    - bravo.<DOMAIN-REDACTED>
    - 127.0.0.1
  storage:
    etcd:
      peerAddress: 100.93.0.158
    kine:
      dataSource: mysql://k0s:<PASSWORD-REDACTED>@tcp(127.0.0.1:4000)/k0s
    type: kine
  telemetry:
    enabled: false
twz123 commented 1 week ago

Looking at the logs, it seems that something on the node is already using port 2380.

level=fatal msg="error creating the metrics listener: listen tcp :2380: bind: address already in use" component=kine stream=stderr

fruworg commented 6 days ago

Hi, @twz123. Oh, yes, I see. Is there a way to set up a custom port for Kine?