k3s-io / k3s

Lightweight Kubernetes
https://k3s.io
Apache License 2.0
28.22k stars 2.36k forks source link

When installing k3s server on multiple centos9 machines, 10.43.0.1 cannot be accessed #8822

Closed w7team closed 10 months ago

w7team commented 1 year ago

I created two centos9 virtual machines on the same network segment under pve, namely 10.0.0.201 and 10.0.0.202, and installed two independent k3s servers on the two virtual machines. It was found that only one of the two independent clusters could be started successfully. Through investigation, it was found that 10.43.0.1 was affected. If the two clusters set the same apiserver address 10.43.0.1, host 2 would be able to access host 1 https://10.43.0.1:443 causes k3s of host 2 to fail to start. I tested the installation under Ubuntu and there was no such problem. The problem occurred 100% under Centos9. Centos9 does not have iptables installed by default. Could this be related?

我在pve下创建两个两个同网段的centos9虚拟机,分别为10.0.0.20110.0.0.202,并在两个虚拟机上安装了两个独立的k3s server。发现两个独立集群只有一个可以成功启动,通过排查发现10.43.0.1影响到了,如果两个集群设置了相同的apiserver地址10.43.0.1,会导致主机2可以访问主机1的https://10.43.0.1:443的地址,从而导致主机2的k3s无法启动。我在ubuntu下测试安装没有这个问题,centos9下就100%出现问题,centos9默认没有安装iptables,会不会跟这个有关系?

K3s Version: v1.27.4+k3s1 os version: CentOS-Stream-9

k3s check-config

cat: /sys/kernel/security/apparmor/profiles: No such file or directory

Verifying binaries in /var/lib/rancher/k3s/data/b20b68fab02222c85f08adc9d865ac2052f2f2f831a883e3b42654cf04acf31c/bin:

  • sha256sum: good
  • aux/ip6tables: symlink to xtables-nft-multi
  • aux/ip6tables-restore: symlink to xtables-nft-multi
  • aux/ip6tables-save: symlink to xtables-nft-multi
  • aux/iptables: symlink to xtables-nft-multi
  • aux/iptables-restore: symlink to xtables-nft-multi
  • aux/iptables-save: symlink to xtables-nft-multi
  • links: good

System:

  • /var/lib/rancher/k3s/data/b20b68fab02222c85f08adc9d865ac2052f2f2f831a883e3b42654cf04acf31c/bin/aux iptables v1.8.8 (nf_tables): ok
  • swap: disabled
  • routes: ok

Limits:

  • /proc/sys/kernel/keys/root_maxkeys: 1000000

modprobe: FATAL: Module configs not found in directory /lib/modules/5.14.0-366.el9.x86_64 info: reading kernel config from /boot/config-5.14.0-366.el9.x86_64 ...

Generally Necessary:

  • cgroup hierarchy: cgroups V2 mounted, cpu|cpuset|memory controllers status: good
  • CONFIG_NAMESPACES: enabled
  • CONFIG_NET_NS: enabled
  • CONFIG_PID_NS: enabled
  • CONFIG_IPC_NS: enabled
  • CONFIG_UTS_NS: enabled
  • CONFIG_CGROUPS: enabled
  • CONFIG_CGROUP_PIDS: enabled
  • CONFIG_CGROUP_CPUACCT: enabled
  • CONFIG_CGROUP_DEVICE: enabled
  • CONFIG_CGROUP_FREEZER: enabled
  • CONFIG_CGROUP_SCHED: enabled
  • CONFIG_CPUSETS: enabled
  • CONFIG_MEMCG: enabled
  • CONFIG_KEYS: enabled
  • CONFIG_VETH: enabled (as module)
  • CONFIG_BRIDGE: enabled (as module)
  • CONFIG_BRIDGE_NETFILTER: enabled (as module)
  • CONFIG_IP_NF_FILTER: enabled (as module)
  • CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_IPVS: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_COMMENT: enabled (as module)
  • CONFIG_NETFILTER_XT_MATCH_MULTIPORT: enabled (as module)
  • CONFIG_IP_NF_NAT: enabled (as module)
  • CONFIG_NF_NAT: enabled (as module)
  • CONFIG_POSIX_MQUEUE: enabled

Optional Features:

  • CONFIG_USER_NS: enabled
  • CONFIG_SECCOMP: enabled
  • CONFIG_BLK_CGROUP: enabled
  • CONFIG_BLK_DEV_THROTTLING: enabled
  • CONFIG_CGROUP_PERF: enabled
  • CONFIG_CGROUP_HUGETLB: enabled
  • CONFIG_NET_CLS_CGROUP: enabled
  • CONFIG_CGROUP_NET_PRIO: enabled
  • CONFIG_CFS_BANDWIDTH: enabled
  • CONFIG_FAIR_GROUP_SCHED: enabled
  • CONFIG_RT_GROUP_SCHED: missing
  • CONFIG_IP_NF_TARGET_REDIRECT: enabled (as module)
  • CONFIG_IP_SET: enabled (as module)
  • CONFIG_IP_VS: enabled (as module)
  • CONFIG_IP_VS_NFCT: enabled
  • CONFIG_IP_VS_PROTO_TCP: enabled
  • CONFIG_IP_VS_PROTO_UDP: enabled
  • CONFIG_IP_VS_RR: enabled (as module)
  • CONFIG_EXT4_FS: enabled (as module)
  • CONFIG_EXT4_FS_POSIX_ACL: enabled
  • CONFIG_EXT4_FS_SECURITY: enabled
  • Network Drivers:
    • "overlay":
    • CONFIG_VXLAN: enabled (as module) Optional (for encrypted networks):
      • CONFIG_CRYPTO: enabled
      • CONFIG_CRYPTO_AEAD: enabled
      • CONFIG_CRYPTO_GCM: enabled
      • CONFIG_CRYPTO_SEQIV: enabled
      • CONFIG_CRYPTO_GHASH: enabled
      • CONFIG_XFRM: enabled
      • CONFIG_XFRM_USER: enabled
      • CONFIG_XFRM_ALGO: enabled
      • CONFIG_INET_ESP: enabled (as module)
      • CONFIG_INET_XFRM_MODE_TRANSPORT: missing
  • Storage Drivers:
    • "overlay":
    • CONFIG_OVERLAY_FS: enabled (as module)

STATUS: pass

node2 curl https://10.43.0.1:443 -k

{ "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "Unauthorized", "reason": "Unauthorized", "code": 401 }

w7team commented 1 year ago

cc @brandond

rbrtbnfgl commented 1 year ago

from the curl seems that node2 can reach 10.43.0.1 and you get unauthorized because you need to use the right token to get the access to the api server. Are you trying to create two different cluster or a cluster with two servers?

w7team commented 1 year ago

two different cluster

------------------ Original ------------------ From: Roberto Bonafiglia @.> Date: Tue,Nov 28,2023 11:30 PM To: k3s-io/k3s @.> Cc: 微擎团队 @.>, Author @.> Subject: Re: [k3s-io/k3s] When installing k3s server on multiple centos9machines, 10.43.0.1 cannot be accessed (Issue #8822)

from the curl seems that node2 can reach 10.43.0.1 and you get unauthorized because you need to use the right token to get the access to the api server. Are you trying to create two different cluster or a cluster with two servers?

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

rbrtbnfgl commented 1 year ago

Are you starting K3s with a custom configuration or default one?

w7team commented 1 year ago

[root@localhost ~]# cat /etc/systemd/system/k3s.service

[Unit]
Description=Lightweight Kubernetes
Documentation=https://k3s.io
Wants=network-online.target
After=network-online.target

[Install]
WantedBy=multi-user.target

[Service]
Type=notify
EnvironmentFile=-/etc/default/%N
EnvironmentFile=-/etc/sysconfig/%N
EnvironmentFile=-/etc/systemd/system/k3s.service.env
KillMode=process
Delegate=yes
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNOFILE=1048576
LimitNPROC=infinity
LimitCORE=infinity
TasksMax=infinity
TimeoutStartSec=0
Restart=always
RestartSec=5s
ExecStartPre=/bin/sh -xc '! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service'
ExecStartPre=-/sbin/modprobe br_netfilter
ExecStartPre=-/sbin/modprobe overlay
ExecStart=/usr/local/bin/k3s \
    server \
        '--write-kubeconfig-mode' \
        '644' \
        '--tls-san' \
        '***' \
        '--node-external-ip' \
        '***' \
        '--system-default-registry' \
        'registry.cn-hangzhou.aliyuncs.com' \
        '--kubelet-arg=image-gc-high-threshold=70' \
        '--kubelet-arg=image-gc-low-threshold=60'

我的k3s就这些配置,没有修改过ip段 My k3s has these configurations, and the IP segment has not been modified.

github-actions[bot] commented 10 months ago

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.