search

rancher / rke2

https://docs.rke2.io/
Apache License 2.0
1.51k stars 264 forks source link

Can't access to pods on a specific node #6567

Open vfouqueron opened 1 month ago

vfouqueron commented 1 month ago

Environmental Info: RKE2 Version: rke2 version v1.28.12+rke2r1 (27989db0c83739c9f560e0567a3bf76177947122) go version go1.22.5 X:boringcrypto

Node(s) CPU architecture, OS, and Version: Linux VMALLKUB03 6.8.0-40-generic #40-Ubuntu SMP PREEMPT_DYNAMIC Fri Jul 5 10:34:03 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux

Cluster Configuration: I have only 3 servers, no agents

Describe the bug: The pods in node-1 and node-2 are reachable from all 3-nodes, but pods in node-3 are not reachable from node-1 and node-2.

I test this by doing pings to pods in the different nodes.

Steps To Reproduce: Here is my configurations :

Node 1

disable:
  - rke2-ingress-nginx
node-ip: 192.168.9.31
server: https://rancher.predell.priv:9345
service-node-port-range:
  - 0-32767
tls-san:
  - rancher.predell.priv
  - 192.168.9.30
  - 192.168.9.31
  - 192.168.9.32
token: <token>

Node 2

disable:
  - rke2-ingress-nginx
node-ip: 192.168.9.32
server: https://rancher.predell.priv:9345
service-node-port-range:
  - 0-32767
tls-san:
  - rancher.predell.priv
  - 192.168.9.30
  - 192.168.9.31
  - 192.168.9.32
token: <token>

Node 3

disable:
  - rke2-ingress-nginx
node-ip: 192.168.9.30
server: https://rancher.predell.priv:9345
service-node-port-range:
  - 0-32767
tls-san:
  - rancher.predell.priv
  - 192.168.9.30
  - 192.168.9.31
  - 192.168.9.32
token: <token>

Expected behavior: All pods are reachable from all nodes

Actual behavior: Pods on node-3 are not reachable

Additional context / logs:

On node 1 : 

$ sudo ip -details link show flannel.1

230: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether ca:e2:03:38:55:7b brd ff:ff:ff:ff:ff:ff promiscuity 0  allmulti 0 minmtu 68 maxmtu 65535 
    vxlan id 1 local 192.168.9.31 dev ens18 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536

On node 2 : 

$ sudo ip -details link show flannel.1

660: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether f6:41:70:85:c8:2a brd ff:ff:ff:ff:ff:ff promiscuity 0  allmulti 0 minmtu 68 maxmtu 65535 
    vxlan id 1 local 192.168.9.32 dev ens18 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536

On node 3 : 

$ sudo ip -details link show flannel.1

391: flannel.1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default 
    link/ether b6:10:1e:4a:bf:0b brd ff:ff:ff:ff:ff:ff promiscuity 0  allmulti 0 minmtu 68 maxmtu 65535 
    vxlan id 1 local 192.168.9.30 dev ens18 srcport 0 0 dstport 8472 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535 tso_max_size 65536 tso_max_segs 65535 gro_max_size 65536

rke-canal logs : 

2024-08-13 15:56:45.543 [INFO][9] startup/startup.go 431: Early log level set to info
2024-08-13 15:56:45.543 [INFO][9] startup/utils.go 126: Using NODENAME environment for node name vmallkub03
2024-08-13 15:56:45.543 [INFO][9] startup/utils.go 138: Determined node name: vmallkub03
2024-08-13 15:56:45.543 [INFO][9] startup/startup.go 95: Starting node vmallkub03 with version v3.28.0
2024-08-13 15:56:45.545 [INFO][9] startup/startup.go 436: Checking datastore connection
2024-08-13 15:56:45.581 [INFO][9] startup/startup.go 460: Datastore connection verified
2024-08-13 15:56:45.581 [INFO][9] startup/startup.go 105: Datastore is ready
2024-08-13 15:56:45.596 [WARNING][9] startup/winutils.go 150: Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-08-13 15:56:45.788 [INFO][9] startup/startup.go 830: Selected default IP pool is '172.16.0.0/16'
2024-08-13 15:56:45.788 [INFO][9] startup/startup.go 680: FELIX_IPV6SUPPORT is false through environment variable
2024-08-13 15:56:45.878 [INFO][9] startup/startup.go 215: Using node name: vmallkub03
2024-08-13 15:56:45.976 [INFO][18] tunnel-ip-allocator/param_types.go 718: StringSliceParam StringSliceParam raw="docker+"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "prometheusmetricsenabled"="true"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "failsafeoutboundhostports"=""
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthenabled"="true"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "failsafeinboundhostports"=""
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "iptablesrefreshinterval"="60"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "logseverityscreen"="info"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "iptablesbackend"="auto"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "iptablesmarkmask"="0xffff0000"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "ipv6support"="false"
2024-08-13 15:56:45.977 [INFO][18] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "xdpenabled"="false"
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 504: Merging in config from environment variable: map[defaultendpointtohostaction:ACCEPT failsafeinboundhostports: failsafeoutboundhostports: healthenabled:true iptablesbackend:auto iptablesmarkmask:0xffff0000 iptablesrefreshinterval:60 ipv6support:false logseverityscreen:info prometheusmetricsenabled:true xdpenabled:false]
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="failsafeoutboundhostports" source=environment variable
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="failsafeinboundhostports" source=environment variable
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for IptablesMarkMask: 0xffff0000 (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for IptablesMarkMask: 4294901760 (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for IptablesRefreshInterval: 60 (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for IptablesRefreshInterval: 1m0s (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:45.978 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for LogSeverityScreen: info (from environment variable)
2024-08-13 15:56:45.979 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for LogSeverityScreen: INFO (from environment variable)
2024-08-13 15:56:45.979 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:45.979 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:45.979 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:45.979 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:45.979 [INFO][18] tunnel-ip-allocator/config_params.go 635: Parsing value for XDPEnabled: false (from environment variable)
2024-08-13 15:56:45.979 [INFO][18] tunnel-ip-allocator/config_params.go 671: Parsed value for XDPEnabled: false (from environment variable)
CALICO_NETWORKING_BACKEND is none - no BGP daemon running
Calico node started successfully
2024-08-13 15:56:47.196 [WARNING][49] cni-config-monitor/winutils.go 150: Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-08-13 15:56:47.197 [INFO][54] status-reporter/startup.go 431: Early log level set to info
2024-08-13 15:56:47.198 [INFO][48] tunnel-ip-allocator/param_types.go 718: StringSliceParam StringSliceParam raw="docker+"
2024-08-13 15:56:47.198 [INFO][54] status-reporter/watchersyncer.go 89: Start called
2024-08-13 15:56:47.199 [INFO][54] status-reporter/watchersyncer.go 130: Sending status update Status=wait-for-ready
2024-08-13 15:56:47.199 [INFO][54] status-reporter/watchersyncer.go 149: Starting main event processing loop
2024-08-13 15:56:47.199 [INFO][54] status-reporter/watchercache.go 181: Full resync is required ListRoot="/calico/resources/v3/projectcalico.org/caliconodestatuses"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "prometheusmetricsenabled"="true"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "failsafeoutboundhostports"=""
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "healthenabled"="true"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "failsafeinboundhostports"=""
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "iptablesrefreshinterval"="60"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "logseverityscreen"="info"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "iptablesbackend"="auto"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "iptablesmarkmask"="0xffff0000"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "ipv6support"="false"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/env_var_loader.go 40: Found felix environment variable: "xdpenabled"="false"
2024-08-13 15:56:47.199 [INFO][48] tunnel-ip-allocator/config_params.go 504: Merging in config from environment variable: map[defaultendpointtohostaction:ACCEPT failsafeinboundhostports: failsafeoutboundhostports: healthenabled:true iptablesbackend:auto iptablesmarkmask:0xffff0000 iptablesrefreshinterval:60 ipv6support:false logseverityscreen:info prometheusmetricsenabled:true xdpenabled:false]
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="failsafeinboundhostports" source=environment variable
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="failsafeoutboundhostports" source=environment variable
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for LogSeverityScreen: info (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for LogSeverityScreen: INFO (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for IptablesMarkMask: 0xffff0000 (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for IptablesMarkMask: 4294901760 (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for XDPEnabled: false (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for XDPEnabled: false (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for IptablesRefreshInterval: 60 (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for IptablesRefreshInterval: 1m0s (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 635: Parsing value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:47.200 [INFO][48] tunnel-ip-allocator/config_params.go 671: Parsed value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:47.226 [INFO][50] felix/logutils.go 82: Early screen log level set to info
2024-08-13 15:56:47.226 [INFO][50] felix/daemon.go 146: Felix starting up GOMAXPROCS=14 builddate="2024-06-25T18:33:58+0000" gitcommit="413e6f5593a8e76d34cf5338a642265da52561ce" version="v3.28.0"
2024-08-13 15:56:47.230 [INFO][50] felix/daemon.go 165: Kubernetes server override env vars. KUBERNETES_SERVICE_HOST="10.43.0.1" KUBERNETES_SERVICE_PORT="443"
2024-08-13 15:56:47.230 [INFO][50] felix/daemon.go 170: Loading configuration...
2024-08-13 15:56:47.231 [INFO][50] felix/param_types.go 718: StringSliceParam StringSliceParam raw="docker+"
2024-08-13 15:56:47.231 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "prometheusmetricsenabled"="true"
2024-08-13 15:56:47.231 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "etcdscheme"=""
2024-08-13 15:56:47.231 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "failsafeoutboundhostports"=""
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "etcdkeyfile"=""
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "healthenabled"="true"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "failsafeinboundhostports"=""
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "iptablesrefreshinterval"="60"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "etcdcafile"=""
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "etcdendpoints"=""
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "felixhostname"="vmallkub03"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "datastoretype"="kubernetes"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "etcdcertfile"=""
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "etcdaddr"=""
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "defaultendpointtohostaction"="ACCEPT"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "logseverityscreen"="info"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "iptablesbackend"="auto"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "iptablesmarkmask"="0xffff0000"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "ipv6support"="false"
2024-08-13 15:56:47.232 [INFO][50] felix/env_var_loader.go 40: Found felix environment variable: "xdpenabled"="false"
2024-08-13 15:56:47.232 [INFO][50] felix/daemon.go 196: Loading config file: /etc/calico/felix.cfg
2024-08-13 15:56:47.232 [INFO][50] felix/config_params.go 504: Merging in config from environment variable: map[datastoretype:kubernetes defaultendpointtohostaction:ACCEPT etcdaddr: etcdcafile: etcdcertfile: etcdendpoints: etcdkeyfile: etcdscheme: failsafeinboundhostports: failsafeoutboundhostports: felixhostname:vmallkub03 healthenabled:true iptablesbackend:auto iptablesmarkmask:0xffff0000 iptablesrefreshinterval:60 ipv6support:false logseverityscreen:info prometheusmetricsenabled:true xdpenabled:false]
2024-08-13 15:56:47.232 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdcafile" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdaddr" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="failsafeoutboundhostports" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdkeyfile" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="failsafeinboundhostports" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdcertfile" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdendpoints" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 513: Ignoring empty configuration parameter. Use value 'none' if your intention is to explicitly disable the default value. name="etcdscheme" source=environment variable
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 635: Parsing value for IptablesMarkMask: 0xffff0000 (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 671: Parsed value for IptablesMarkMask: 4294901760 (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 635: Parsing value for LogSeverityScreen: info (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 671: Parsed value for LogSeverityScreen: INFO (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 635: Parsing value for XDPEnabled: false (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 671: Parsed value for XDPEnabled: false (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 635: Parsing value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 671: Parsed value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 635: Parsing value for DatastoreType: kubernetes (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 671: Parsed value for DatastoreType: kubernetes (from environment variable)
2024-08-13 15:56:47.233 [INFO][50] felix/config_params.go 635: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 671: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 635: Parsing value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 671: Parsed value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 635: Parsing value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 671: Parsed value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 635: Parsing value for FelixHostname: vmallkub03 (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 671: Parsed value for FelixHostname: vmallkub03 (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 635: Parsing value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 671: Parsed value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 635: Parsing value for IptablesRefreshInterval: 60 (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 671: Parsed value for IptablesRefreshInterval: 1m0s (from environment variable)
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 504: Merging in config from config file: map[LogFilePath:None LogSeverityFile:None LogSeveritySys:None MetadataAddr:None]
2024-08-13 15:56:47.234 [INFO][50] felix/config_params.go 635: Parsing value for DatastoreType: kubernetes (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for DatastoreType: kubernetes (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for IptablesMarkMask: 0xffff0000 (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for IptablesMarkMask: 4294901760 (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for LogSeverityScreen: info (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for LogSeverityScreen: INFO (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for XDPEnabled: false (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for XDPEnabled: false (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for IptablesBackend: auto (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for IptablesRefreshInterval: 60 (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for IptablesRefreshInterval: 1m0s (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for DefaultEndpointToHostAction: ACCEPT (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for PrometheusMetricsEnabled: true (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 635: Parsing value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:47.235 [INFO][50] felix/config_params.go 671: Parsed value for HealthEnabled: true (from environment variable)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 635: Parsing value for FelixHostname: vmallkub03 (from environment variable)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 671: Parsed value for FelixHostname: vmallkub03 (from environment variable)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 635: Parsing value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 671: Parsed value for Ipv6Support: false (from environment variable)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 635: Parsing value for LogSeveritySys: None (from config file)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 652: Value set to 'none', replacing with zero-value: "".
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 671: Parsed value for LogSeveritySys:  (from config file)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 635: Parsing value for MetadataAddr: None (from config file)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 652: Value set to 'none', replacing with zero-value: "".
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 671: Parsed value for MetadataAddr:  (from config file)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 635: Parsing value for LogFilePath: None (from config file)
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 652: Value set to 'none', replacing with zero-value: "".
2024-08-13 15:56:47.236 [INFO][50] felix/config_params.go 671: Parsed value for LogFilePath:  (from config file)
2024-08-13 15:56:47.237 [INFO][50] felix/config_params.go 635: Parsing value for LogSeverityFile: None (from config file)
2024-08-13 15:56:47.237 [INFO][50] felix/config_params.go 652: Value set to 'none', replacing with zero-value: "".
2024-08-13 15:56:47.237 [INFO][50] felix/config_params.go 671: Parsed value for LogSeverityFile:  (from config file)
2024-08-13 15:56:47.237 [INFO][50] felix/health.go 379: Health enabled.  Starting server. host="localhost" port=9099
2024-08-13 15:56:47.237 [INFO][50] felix/config_params.go 816: Encap disabled, disabling node poll (if KDD is in use).
2024-08-13 15:56:47.237 [INFO][50] felix/daemon.go 228: Connecting to datastore datastore="kubernetes"
2024-08-13 15:56:47.238 [INFO][50] felix/daemon.go 235: Created datastore client
2024-08-13 15:56:47.296 [INFO][49] cni-config-monitor/token_watch.go 226: Update of CNI kubeconfig triggered based on elapsed time.
2024-08-13 15:56:47.296 [WARNING][49] cni-config-monitor/winutils.go 150: Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
2024-08-13 15:56:47.297 [INFO][49] cni-config-monitor/token_watch.go 281: Wrote updated CNI kubeconfig file. path="/host/etc/cni/net.d/calico-kubeconfig"
2024-08-13 15:56:47.302 [INFO][50] felix/daemon.go 950: Loaded ready flag kind="ClusterInformation" name="default" ready=true
2024-08-13 15:56:47.329 [INFO][54] status-reporter/watchercache.go 294: Sending synced update ListRoot="/calico/resources/v3/projectcalico.org/caliconodestatuses"
2024-08-13 15:56:47.329 [INFO][54] status-reporter/watchersyncer.go 209: Received InSync event from one of the watcher caches
2024-08-13 15:56:47.329 [INFO][54] status-reporter/watchersyncer.go 130: Sending status update Status=resync
2024-08-13 15:56:47.329 [INFO][54] status-reporter/watchersyncer.go 221: All watchers have sync'd data - sending data and final sync
2024-08-13 15:56:47.329 [INFO][54] status-reporter/watchersyncer.go 130: Sending status update Status=in-sync
2024-08-13 15:56:47.394 [INFO][50] felix/daemon.go 930: No config of this type kind="FelixConfiguration" name="node.vmallkub03"
2024-08-13 15:56:47.395 [INFO][50] felix/health.go 294: Reporter is not ready: reporting non-ready. name="FelixStartup"
2024-08-13 15:56:47.396 [INFO][50] felix/health.go 336: Overall health status changed: live=true ready=false
+--------------+---------+----------------+---------------------+--------+
|  COMPONENT   | TIMEOUT |    LIVENESS    |      READINESS      | DETAIL |
+--------------+---------+----------------+---------------------+--------+
| FelixStartup | -       | reporting live | reporting non-ready |        |
+--------------+---------+----------------+---------------------+--------+
dfaltum commented 1 month ago

Hi there! Can you try this one? https://github.com/rancher/rke2/issues/6307#issuecomment-2223536624

vfouqueron commented 1 month ago

Thank you @dfaltum for your answer !

If I'm not mistaken port 4789 is for Windows nodes (as described there : https://docs.rke2.io/install/requirements#inbound-network-rules). There is currently no firewall on nodes, but there is also nothing listening on port 4789on the 192.168.9.30 at least. However, there is a service listening on port 8472 as expected and reachable from other nodes.

I runned /usr/sbin/ethtool -K flannel.1 tx-checksum-ip-generic off but pods are still unreachable from other hosts