Open kkgty opened 2 years ago
Can you elaborate on how you create your topology? Do you have a list of instructions to reproduce this?
thanks for your reply, here is my topology, and all configurations are here: https://github.com/kkgty/topo-ospf
apiVersion: v1
kind: List
items:
################### pod #######################
###### router-1 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-1
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-1
###### router-2 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-2
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-2
###### router-3 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-3
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-3
###### router-4 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-4
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-4
###### router-5 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-5
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-5
###### router-6 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-6
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-6
###### router-7 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-7
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-7
###### router-8 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-8
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-8
###### router-9 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-9
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-9
###### router-10 ######
- apiVersion: v1
kind: Pod
metadata:
name: router-10
labels:
name: tunnel
spec:
containers:
- name: tunnel
image: frrouting/frr:v8.2.2
resources:
limits:
memory: "128Mi"
cpu: "500m"
securityContext:
privileged: true
volumeMounts:
- name: config
mountPath: /etc/frr/daemons
subPath: daemons
- name: config
mountPath: /etc/frr/frr.conf
subPath: frr.conf
volumes:
- name: config
configMap:
name: router-10
################### topo #######################
###### router-1 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-1
spec:
links:
- uid: 2218012
peer_pod: router-2
local_intf: eth1
peer_intf: eth1
local_ip: 10.0.1.1/24
peer_ip: 10.0.1.2/24
- uid: 2218013
peer_pod: router-3
local_intf: eth2
peer_intf: eth1
local_ip: 10.0.2.1/24
peer_ip: 10.0.2.2/24
###### router-2 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-2
spec:
links:
- uid: 2218012
peer_pod: router-1
local_intf: eth1
peer_intf: eth1
local_ip: 10.0.1.2/24
peer_ip: 10.0.1.1/24
- uid: 2218023
peer_pod: router-3
local_intf: eth2
peer_intf: eth2
local_ip: 10.0.3.1/24
peer_ip: 10.0.3.2/24
###### router-3 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-3
spec:
links:
- uid: 2218013
peer_pod: router-1
local_intf: eth1
peer_intf: eth2
local_ip: 10.0.2.2/24
peer_ip: 10.0.2.1/24
- uid: 2218023
peer_pod: router-2
local_intf: eth2
peer_intf: eth2
local_ip: 10.0.3.2/24
peer_ip: 10.0.3.1/24
- uid: 2218034
peer_pod: router-4
local_intf: eth3
peer_intf: eth1
local_ip: 10.0.4.1/24
peer_ip: 10.0.4.2/24
- uid: 2218035
peer_pod: router-5
local_intf: eth4
peer_intf: eth1
local_ip: 10.0.5.1/24
peer_ip: 10.0.5.2/24
- uid: 2218036
peer_pod: router-6
local_intf: eth5
peer_intf: eth1
local_ip: 10.0.6.1/24
peer_ip: 10.0.6.2/24
###### router-4 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-4
spec:
links:
- uid: 2218034
peer_pod: router-3
local_intf: eth1
peer_intf: eth3
local_ip: 10.0.4.2/24
peer_ip: 10.0.4.1/24
- uid: 2218045
peer_pod: router-5
local_intf: eth2
peer_intf: eth2
local_ip: 10.0.7.1/24
peer_ip: 10.0.7.2/24
- uid: 2218047
peer_pod: router-7
local_intf: eth3
peer_intf: eth1
local_ip: 10.0.8.1/24
peer_ip: 10.0.8.2/24
###### router-5 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-5
spec:
links:
- uid: 2218035
peer_pod: router-3
local_intf: eth1
peer_intf: eth4
local_ip: 10.0.5.2/24
peer_ip: 10.0.5.1/24
- uid: 2218045
peer_pod: router-4
local_intf: eth2
peer_intf: eth2
local_ip: 10.0.7.2/24
peer_ip: 10.0.7.1/24
- uid: 2218058
peer_pod: router-8
local_intf: eth3
peer_intf: eth2
local_ip: 10.0.10.1/24
peer_ip: 10.0.10.2/24
###### router-6 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-6
spec:
links:
- uid: 2218036
peer_pod: router-3
local_intf: eth1
peer_intf: eth5
local_ip: 10.0.6.2/24
peer_ip: 10.0.6.1/24
- uid: 2218068
peer_pod: router-8
local_intf: eth2
peer_intf: eth1
local_ip: 10.0.9.1/24
peer_ip: 10.0.9.2/24
###### router-7 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-7
spec:
links:
- uid: 2218047
peer_pod: router-4
local_intf: eth1
peer_intf: eth3
local_ip: 10.0.8.2/24
peer_ip: 10.0.8.1/24
- uid: 2218078
peer_pod: router-8
local_intf: eth2
peer_intf: eth3
local_ip: 10.0.11.1/24
peer_ip: 10.0.11.2/24
###### router-8 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-8
spec:
links:
- uid: 2218058
peer_pod: router-5
local_intf: eth2
peer_intf: eth3
local_ip: 10.0.10.2/24
peer_ip: 10.0.10.1/24
- uid: 2218068
peer_pod: router-6
local_intf: eth1
peer_intf: eth2
local_ip: 10.0.9.2/24
peer_ip: 10.0.9.1/24
- uid: 2218078
peer_pod: router-7
local_intf: eth3
peer_intf: eth2
local_ip: 10.0.11.2/24
peer_ip: 10.0.11.1/24
- uid: 2218089
peer_pod: router-9
local_intf: eth5
peer_intf: eth1
local_ip: 10.0.13.1/24
peer_ip: 10.0.13.2/24
- uid: 2218080
peer_pod: router-10
local_intf: eth4
peer_intf: eth1
local_ip: 10.0.12.1/24
peer_ip: 10.0.12.2/24
###### router-9 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-9
spec:
links:
- uid: 2218089
peer_pod: router-8
local_intf: eth1
peer_intf: eth5
local_ip: 10.0.13.2/24
peer_ip: 10.0.13.1/24
- uid: 2218090
peer_pod: router-10
local_intf: eth2
peer_intf: eth2
local_ip: 10.0.14.1/24
peer_ip: 10.0.14.2/24
###### router-10 ######
- apiVersion: networkop.co.uk/v1beta1
kind: Topology
metadata:
name: router-10
spec:
links:
- uid: 2218080
peer_pod: router-8
local_intf: eth1
peer_intf: eth4
local_ip: 10.0.12.2/24
peer_ip: 10.0.12.1/24
- uid: 2218090
peer_pod: router-9
local_intf: eth2
peer_intf: eth2
local_ip: 10.0.14.2/24
peer_ip: 10.0.14.1/24
Can you also provide the output of ip -d link show
from inside of one of the routers with neighbors in Init/DROther
state?
e.g. kubectl exec -it router-4 -- ip -d link show
Sure, here's the output from router4
node-d ➜ ~ kubectl exec -n tunnel router-4 -- ip -d link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
3: eth0@if178: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether e6:a1:b0:66:3f:99 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
188: eth3@if188: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 66:d8:31:8b:88:15 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
vxlan id 2223047 remote 192.168.22.7 dev if2 srcport 0 0 dstport 4789 l2miss l3miss ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
189: eth1@if189: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 7e:ca:8c:a9:70:11 brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
vxlan id 2223034 remote 192.168.22.7 dev if2 srcport 0 0 dstport 4789 l2miss l3miss ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
191: eth2@if191: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 86:1d:2c:1d:1d:dc brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
vxlan id 2223045 remote 192.168.22.7 dev if2 srcport 0 0 dstport 4789 l2miss l3miss ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
and router-5
node-d ➜ ~ kubectl exec -n tunnel router-5 -- ip -d link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0 minmtu 0 maxmtu 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
3: eth0@if194: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether e2:4b:59:44:b2:ac brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
205: eth1@if204: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether b2:be:cd:69:92:cd brd ff:ff:ff:ff:ff:ff link-netnsid 1 promiscuity 0 minmtu 68 maxmtu 65535
veth addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
211: eth2@if211: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 56:89:62:6e:79:ab brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
vxlan id 2223045 remote 192.168.22.3 dev eth0 srcport 0 0 dstport 4789 l2miss l3miss ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
215: eth3@if215: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 86:fc:ef:8b:1f:3e brd ff:ff:ff:ff:ff:ff link-netnsid 0 promiscuity 0 minmtu 68 maxmtu 65535
vxlan id 2223058 remote 192.168.22.3 dev eth0 srcport 0 0 dstport 4789 l2miss l3miss ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx addrgenmode eui64 numtxqueues 1 numrxqueues 1 gso_max_size 65536 gso_max_segs 65535
node-d ➜ ~ kubectl exec -n tunnel router-5 -- vtysh -c "show ip ospf neighbor"
Neighbor ID Pri State Up Time Dead Time Address Interface RXmtL RqstL DBsmL
10.224.100.87 1 Full/Backup 2h18m15s 39.314s 10.0.5.1 eth1:10.0.5.2 0 0 0
10.224.71.140 1 Full/Backup 2h18m20s 37.147s 10.0.10.2 eth3:10.0.10.1 0 0 0
Everything seems fine. I even deployed your topology locally and was able to see all adjacencies established. The only issue you've got is router-4
missing ip ospf area
statement under its eth3
interface. But otherwise, everything looks correct.
The Init/DROther
is a really weird state. It's as if the packets are being dropped in one direction. I'd suggest checking if you have any ACLs on the host (or somewhere in the network) that might drop UDP/VXLAN packets between hosts. Check the host logs for any errors during Pod deployment. As an experiment you can try and create a pair of VXLAN interfaces manually, give them an IP and try pinging between them. Other than that, I don't have any other ideas.
Thank you very much for your suggestions. I'll try to run some other tests.
I have an OSPF topology with 10 nodes and they all run the same frrouting image
I want to test this topology with meshnet-cni
My k8s cluster has a total of 4 nodes connected through calico BGP mode
I create my topology and distribute it on two nodes
For example, Router4 should establish a neighbor relationship with R3, R5, and R7, but they are not established and cannot ping through each other
However, if all the pods are running on the same node and connected via veth, this is successful
Can someone help me?