search

projectcalico / calico

Cloud native networking and network security
https://docs.tigera.io/calico/latest/about/
Apache License 2.0
5.94k stars 1.32k forks source link

Fail to change the node interface #8602

Closed baowj-678 closed 7 months ago

baowj-678 commented 7 months ago

I want to change the interface used by calico in k8s cluster. According to the document, I update the Installation in namespace tigera-operator. The updated installation is as follows:

apiVersion: v1
items:
- apiVersion: operator.tigera.io/v1
  kind: Installation
  metadata:
    annotations:
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"operator.tigera.io/v1","kind":"Installation","metadata":{"annotations":{},"name":"default"},"spec":{"calicoNetwork":{"ipPools":[{"blockSize":26,"cidr":"100.64.0.0/10","encapsulation":"VXLANCrossSubnet","natOutgoing":"Enabled","nodeSelector":"all()"}]},"registry":"sea.hub:5000"}}
    creationTimestamp: "2022-12-02T07:20:32Z"
    generation: 5
    name: default
    resourceVersion: "149771195"
    uid: 37f7c1e4-165d-4e6f-8bc6-4d4d7f8a9b4d
  spec:
    calicoNetwork:
      bgp: Enabled
      hostPorts: Enabled
      ipPools:
      - blockSize: 26
        cidr: 100.64.0.0/10
        encapsulation: VXLANCrossSubnet
        natOutgoing: Enabled
        nodeSelector: all()
      linuxDataplane: Iptables
      multiInterfaceMode: None
      nodeAddressAutodetectionV4:
        interface: eno12399.*
    cni:
      ipam:
        type: Calico
      type: Calico
    controlPlaneReplicas: 2
    flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
    nodeUpdateStrategy:
      rollingUpdate:
        maxUnavailable: 1
      type: RollingUpdate
    nonPrivileged: Disabled
    registry: sea.hub:5000/
    variant: Calico
  status:
    computed:
      calicoNetwork:
        bgp: Enabled
        hostPorts: Enabled
        ipPools:
        - blockSize: 26
          cidr: 100.64.0.0/10
          encapsulation: VXLANCrossSubnet
          natOutgoing: Enabled
          nodeSelector: all()
        linuxDataplane: Iptables
        multiInterfaceMode: None
        nodeAddressAutodetectionV4:
          interface: eno12399.*
      cni:
        ipam:
          type: Calico
        type: Calico
      controlPlaneReplicas: 2
      flexVolumePath: /usr/libexec/kubernetes/kubelet-plugins/volume/exec/
      nodeUpdateStrategy:
        rollingUpdate:
          maxUnavailable: 1
        type: RollingUpdate
      nonPrivileged: Disabled
      registry: sea.hub:5000/
      variant: Calico
    mtu: 1450
    variant: Calico
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""

Expected Behavior

Calico should use eno12399.* interface (it's bandwith is 10Gbps). The network speed between pods on different nodes should be 10Gbps.

Current Behavior

The network speed between pods on different nodes is just 1Gbps (the bandwith of the interface before changed is 1Gbps). And the calico nodes' log is as follow:

2024-03-10 09:07:24.487 [INFO][4296] felix/route_table.go 432: Trying to connect to netlink
2024-03-10 09:07:24.487 [WARNING][4296] felix/route_table.go 1138: Failed to access interface but it appears to be up error=netlink update operation failed ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 link=&netlink.Device{LinkAttrs:netlink.LinkAttrs{Index:4, MTU:1500, TxQLen:1000, Name:"eno12399np0", HardwareAddr:net.HardwareAddr{0x0, 0x62, 0xb, 0x78, 0x7, 0x30}, Flags:0x13, RawFlags:0x11043, ParentIndex:0, MasterIndex:0, Namespace:interface {}(nil), Alias:"", Statistics:(*netlink.LinkStatistics)(0xc000bfb800), Promisc:0, Allmulti:0, Multi:1, Xdp:(*netlink.LinkXdp)(0xc000f06f48), EncapType:"ether", Protinfo:(*netlink.Protinfo)(nil), OperState:0x6, PhysSwitchID:4269279024, NetNsID:-1, NumTxQueues:74, NumRxQueues:74, GSOMaxSize:0x10000, GSOMaxSegs:0xffff, Vfs:[]netlink.VfInfo(nil), Group:0x0, Slave:netlink.LinkSlave(nil)}} tableIndex=0
2024-03-10 09:07:24.487 [WARNING][4296] felix/route_table.go 575: Failed to sync routes to interface even after retries. Leaving it dirty, requiring a full sync. ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.487 [WARNING][4296] felix/route_table.go 588: Some interfaces still out-of sync. ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.488 [WARNING][4296] felix/int_dataplane.go 1836: Failed to synchronize routing table, will retry...
2024-03-10 09:07:24.595 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.82.46.64/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.595 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.82.46.64/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.595 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.67.102.128/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.595 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.67.102.128/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.595 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.65.120.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.595 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.65.120.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.78.150.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.78.150.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.91.133.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.91.133.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.107.178.0/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.107.178.0/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 432: Trying to connect to netlink
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.107.178.0/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.107.178.0/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.82.46.64/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.82.46.64/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.67.102.128/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.67.102.128/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.65.120.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.65.120.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.78.150.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.78.150.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 910: Deleting from expected targets cidr=100.91.133.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [INFO][4296] felix/route_table.go 916: No pending target update, adding back in as an update cidr=100.91.133.192/26 ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [WARNING][4296] felix/route_table.go 690: Failed to add route error=file exists ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [WARNING][4296] felix/route_table.go 690: Failed to add route error=file exists ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [WARNING][4296] felix/route_table.go 690: Failed to add route error=file exists ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [WARNING][4296] felix/route_table.go 690: Failed to add route error=file exists ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [WARNING][4296] felix/route_table.go 690: Failed to add route error=file exists ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.596 [WARNING][4296] felix/route_table.go 690: Failed to add route error=file exists ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.597 [INFO][4296] felix/route_table.go 432: Trying to connect to netlink
2024-03-10 09:07:24.597 [WARNING][4296] felix/route_table.go 1138: Failed to access interface but it appears to be up error=netlink update operation failed ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 link=&netlink.Device{LinkAttrs:netlink.LinkAttrs{Index:4, MTU:1500, TxQLen:1000, Name:"eno12399np0", HardwareAddr:net.HardwareAddr{0x0, 0x62, 0xb, 0x78, 0x7, 0x30}, Flags:0x13, RawFlags:0x11043, ParentIndex:0, MasterIndex:0, Namespace:interface {}(nil), Alias:"", Statistics:(*netlink.LinkStatistics)(0xc000b53440), Promisc:0, Allmulti:0, Multi:1, Xdp:(*netlink.LinkXdp)(0xc000f07ed8), EncapType:"ether", Protinfo:(*netlink.Protinfo)(nil), OperState:0x6, PhysSwitchID:4269279024, NetNsID:-1, NumTxQueues:74, NumRxQueues:74, GSOMaxSize:0x10000, GSOMaxSegs:0xffff, Vfs:[]netlink.VfInfo(nil), Group:0x0, Slave:netlink.LinkSlave(nil)}} tableIndex=0
2024-03-10 09:07:24.597 [WARNING][4296] felix/route_table.go 575: Failed to sync routes to interface even after retries. Leaving it dirty, requiring a full sync. ifaceName="eno12399np0" ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.597 [WARNING][4296] felix/route_table.go 588: Some interfaces still out-of sync. ifaceRegex="^eno12399np0$" ipVersion=0x4 tableIndex=0
2024-03-10 09:07:24.597 [WARNING][4296] felix/int_dataplane.go 1836: Failed to synchronize routing table, will retry...

Calico's STATUS is normal:

NAME                                       READY   STATUS    RESTARTS   AGE
calico-kube-controllers-79f7986874-4j6xd   1/1     Running   0          26m
calico-node-bjgcl                          1/1     Running   0          42m
calico-node-jc5pn                          1/1     Running   0          42m
calico-node-jglkq                          1/1     Running   0          42m
calico-node-mrm67                          1/1     Running   0          42m
calico-node-qnb8q                          1/1     Running   0          41m
calico-node-t27x5                          1/1     Running   0          41m
calico-node-xfqkd                          1/1     Running   0          42m
calico-node-xhbvq                          1/1     Running   0          41m
calico-typha-547df8dd69-6gklm              1/1     Running   0          26m
calico-typha-547df8dd69-gt5t8              1/1     Running   0          25m
calico-typha-547df8dd69-njs9q              1/1     Running   0          26m

Your Environment

baowj-678 commented 7 months ago

There are some unchanged routes. I deleted them manually and solved it.

$ route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         210.28.132.1    0.0.0.0         UG    101    0        0 eno8303
100.65.120.192  192.28.132.14   255.255.255.192 UG    0      0        0 eno12399np0
100.67.102.128  210.28.132.18   255.255.255.192 UG    0      0        0 eno8303
100.78.150.192  210.28.132.16   255.255.255.192 UG    0      0        0 eno8303
100.82.46.64    210.28.132.21   255.255.255.192 UG    0      0        0 eno8303
100.91.133.192  210.28.132.19   255.255.255.192 UG    0      0        0 eno8303
100.101.133.192 0.0.0.0         255.255.255.192 U     0      0        0 *
100.101.133.193 0.0.0.0         255.255.255.255 UH    0      0        0 cali286c0765f1a
100.101.133.194 0.0.0.0         255.255.255.255 UH    0      0        0 cali60a8b08031f
100.101.133.196 0.0.0.0         255.255.255.255 UH    0      0        0 calid81897a64d7
100.101.133.198 0.0.0.0         255.255.255.255 UH    0      0        0 cali35b1fd9510c
100.101.133.206 0.0.0.0         255.255.255.255 UH    0      0        0 calieb98b93c4e0
100.101.133.211 0.0.0.0         255.255.255.255 UH    0      0        0 cali5eff01a0bf0
100.101.133.212 0.0.0.0         255.255.255.255 UH    0      0        0 cali9584adfa0ec
100.101.133.228 0.0.0.0         255.255.255.255 UH    0      0        0 cali6b9ed818504
100.101.133.229 0.0.0.0         255.255.255.255 UH    0      0        0 calibaacc18f08b
100.101.133.231 0.0.0.0         255.255.255.255 UH    0      0        0 calibe67bf8342e
100.101.133.237 0.0.0.0         255.255.255.255 UH    0      0        0 cali638b6f991f8
100.101.133.241 0.0.0.0         255.255.255.255 UH    0      0        0 calibc3e27b104c
100.101.133.255 0.0.0.0         255.255.255.255 UH    0      0        0 cali80c68bf3aa3
100.107.178.0   210.28.132.17   255.255.255.192 UG    0      0        0 eno8303
100.112.39.128  192.28.132.20   255.255.255.192 UG    0      0        0 eno12399np0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
192.28.132.0    0.0.0.0         255.255.255.0   U     100    0        0 eno12399np0
192.168.122.0   0.0.0.0         255.255.255.0   U     0      0        0 virbr0
210.28.132.0    0.0.0.0         255.255.255.0   U     101    0        0 eno8303