pjdruddy
commented
3 years ago can you share the linux plumbing configuration, and maybe a quick topology sketch?
Closed getabc closed 3 years ago
pjdruddy
commented
3 years ago can you share the linux plumbing configuration, and maybe a quick topology sketch?
taspelund
commented
3 years ago I'm with @pjdruddy. Understanding the interface/bridge config on the Linux side would be helpful here.
The TX drops on the vxlan interface make me suspect we don't have a route to the remote VTEP.
It would also be good to see whether routes have been learned for the EVPN next-hops:
show ip route
getabc
commented
3 years ago 
nb2700-site-008
~ $ ip -d link show wwan0
19: wwan0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN mode DEFAULT group default qlen 1000
link/none promiscuity 0 minmtu 0 maxmtu 65535 addrgenmode eui64 numtxqueues 1 gso_max_size 65536 gso_max_segs 65535
alias usbwwan0-0~ $ ip -d link show vxlan6001
21: vxlan6001: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master lan0 state UNKNOWN mode DEFAULT group default qlen 1000
link/ether 2a:5b:58:72:2e:91 brd ff:ff:ff:ff:ff:ff promiscuity 1 minmtu 68 maxmtu 65535
vxlan id 6001 local 10.168.236.8 srcport 0 0 dstport 4789 nolearning ttl auto ageing 300 udpcsum noudp6zerocsumtx noudp6zerocsumrx
bridge_slave state forwarding priority 32 cost 100 hairpin off guard off root_block off fastleave off learning on flood on port_id 0x10060bd1 port_no 0x10060bdf designated_port 32773 designated_cost 0 designated_bridge 8000.00:11:2B:02:A3:AF designated_root 8000.00:11:2B:02:A3:AF hold_timer 0.00 message_age_timer 0.00 forward_delay_timer 0.00 topology_change_ack 0 config_pending 0 proxy_arp off proxy_arp_wifi off mcast_router 1 mcast_fast_leave off mcast_flood on neigh_suppress off group_fwd_mask 0x10060d24 group_fwd_mask_str 0x0 vlan_tunnel off isolated off addrgenmode eui64 numtxqueues 1 gso_max_size 65536 gso_max_segs 65535~ $ bridge fdb show
01:00:5e:00:00:01 dev dev lan4 self permanent
01:00:5e:00:00:01 dev dev sw0 self permanent
00:11:2b:02:a3:af dev dev eth0 vlan 1 master lan0 permanent
00:11:2b:02:a3:af dev dev eth0 master lan0 permanent
33:33:00:00:00:01 dev dev wlan0 self permanent
33:33:00:00:00:01 dev dev lan0 self permanent
01:00:5e:00:00:01 dev dev lan0 self permanent
2a:5b:58:72:2e:91 dev dev vxlan6001 vlan 1 master lan0 permanent
2a:5b:58:72:2e:91 dev dev vxlan6001 master lan0 permanent~ $ ifconfig wwan0
wwan0 Link encap:UNSPEC HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.168.236.8 P-t-P:10.168.236.8 Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST MTU:1500 Metric:1
RX packets:195928 errors:0 dropped:0 overruns:0 frame:0
TX packets:125895 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:150494781 (143.5 MiB) TX bytes:6221518 (5.9 MiB)bgpd
ubuntuserver# sh bgp neighbors 10.168.236.8
BGP neighbor is *10.168.236.8, remote AS 65000, local AS 65000, internal link
Hostname: nb2700-site-008
Member of peer-group fabric for session parameters
Belongs to the subnet range group: 10.168.236.0/24
BGP version 4, remote router ID 10.168.236.8, local router ID 10.168.234.140
BGP state = Established, up for 00:08:59
Last read 00:00:59, Last write 00:00:59
Hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
AddPath:
L2VPN EVPN: RX advertised L2VPN EVPN and received
Route refresh: advertised and received(old & new)
Address Family L2VPN EVPN: advertised and received
Hostname Capability: advertised (name: ubuntuserver,domain name: n/a) received (name: nb2700-site-008,domain name: n/a)
Graceful Restart Capabilty: advertised and received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart information:
End-of-RIB send: L2VPN EVPN
End-of-RIB received: L2VPN EVPN
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 5 2
Keepalives: 9 9
Route Refresh: 0 0
Capability: 0 0
Total: 15 12
Minimum time between advertisement runs is 0 seconds
For address family: L2VPN EVPN
fabric peer-group member
Update group 6, subgroup 29
Packet Queue length 0
Route-Reflector Client
NEXT_HOP is propagated unchanged to this neighbor
Community attribute sent to this neighbor(all)
advertise-all-vni
1 accepted prefixes
Connections established 1; dropped 0
Last reset 00:08:59, No AFI/SAFI activated for peer
Local host: 10.168.234.140, Local port: 179
Foreign host: 10.168.236.8, Foreign port: 48128
Nexthop: 10.168.234.140
Nexthop global: fe80::20c:29ff:febd:d2b4
Nexthop local: fe80::20c:29ff:febd:d2b4
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 40 ms
Read thread: on Write thread: on FD used: 23nb2700-site-008# sh bgp neighbors 10.168.234.140
BGP neighbor is 10.168.234.140, remote AS 65000, local AS 65000, internal link
Hostname: ubuntuserver
Member of peer-group fabric for session parameters
BGP version 4, remote router ID 10.168.234.140, local router ID 10.168.236.8
BGP state = Established, up for 00:09:40
Last read 00:00:40, Last write 00:00:40
Hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
AddPath:
L2VPN EVPN: RX advertised L2VPN EVPN and received
Route refresh: advertised and received(old & new)
Address Family L2VPN EVPN: advertised and received
Hostname Capability: advertised (name: nb2700-site-008,domain name: n/a) received (name: ubuntuserver,domain name: n/a)
Graceful Restart Capability: advertised and received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart information:
End-of-RIB send: L2VPN EVPN
End-of-RIB received: L2VPN EVPN
Local GR Mode: Helper*
Remote GR Mode: Helper
R bit: False
Timers:
Configured Restart Time(sec): 120
Received Restart Time(sec): 120
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 2 5
Keepalives: 10 10
Route Refresh: 0 0
Capability: 0 0
Total: 13 16
Minimum time between advertisement runs is 0 seconds
For address family: L2VPN EVPN
fabric peer-group member
Update group 1, subgroup 1
Packet Queue length 0
NEXT_HOP is propagated unchanged to this neighbor
Community attribute sent to this neighbor(all)
advertise-all-vni
1 accepted prefixes
Connections established 1; dropped 0
Last reset 00:09:41, Waiting for peer OPEN
Local host: 10.168.236.8, Local port: 48128
Foreign host: 10.168.234.140, Foreign port: 179
Nexthop: 10.168.236.8
Nexthop global: ::
Nexthop local: ::
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 35 ms
Read thread: on Write thread: on FD used: 20nb2700-site-009# sh bgp neighbors 10.168.234.140
BGP neighbor is 10.168.234.140, remote AS 65000, local AS 65000, internal link
Hostname: ubuntuserver
Member of peer-group fabric for session parameters
BGP version 4, remote router ID 10.168.234.140, local router ID 10.168.236.9
BGP state = Established, up for 00:12:43
Last read 00:00:07, Last write 00:00:43
Hold time is 180, keepalive interval is 60 seconds
Neighbor capabilities:
4 Byte AS: advertised and received
AddPath:
L2VPN EVPN: RX advertised L2VPN EVPN and received
Route refresh: advertised and received(old & new)
Address Family L2VPN EVPN: advertised and received
Hostname Capability: advertised (name: nb2700-site-009,domain name: n/a) received (name: ubuntuserver,domain name: n/a)
Graceful Restart Capability: advertised and received
Remote Restart timer is 120 seconds
Address families by peer:
none
Graceful restart information:
End-of-RIB send: L2VPN EVPN
End-of-RIB received: L2VPN EVPN
Local GR Mode: Helper*
Remote GR Mode: Helper
R bit: False
Timers:
Configured Restart Time(sec): 120
Received Restart Time(sec): 120
Message statistics:
Inq depth is 0
Outq depth is 0
Sent Rcvd
Opens: 1 1
Notifications: 0 0
Updates: 2 7
Keepalives: 13 13
Route Refresh: 0 0
Capability: 0 0
Total: 16 21
Minimum time between advertisement runs is 0 seconds
For address family: L2VPN EVPN
fabric peer-group member
Update group 1, subgroup 1
Packet Queue length 0
NEXT_HOP is propagated unchanged to this neighbor
Community attribute sent to this neighbor(all)
advertise-all-vni
1 accepted prefixes
Connections established 1; dropped 0
Last reset 00:12:44, Waiting for peer OPEN
Local host: 10.168.236.9, Local port: 43248
Foreign host: 10.168.234.140, Foreign port: 179
Nexthop: 10.168.236.9
Nexthop global: ::
Nexthop local: ::
BGP connection: non shared network
BGP Connect Retry Timer in Seconds: 120
Estimated round trip time: 124 ms
Read thread: on Write thread: on FD used: 20I'm with @pjdruddy. Understanding the interface/bridge config on the Linux side would be helpful here.
The TX drops on the vxlan interface make me suspect we don't have a route to the remote VTEP. It would also be good to see whether routes have been learned for the EVPN next-hops:
show ip route
You are correct - when I manually create the fdb forwarding entry on nb2700-site-00x the vxlan interface will transmit data.
taspelund
commented
3 years ago Could you provide the following:
show ip route
show evpn
show evpn vni detailOver LTE
nb2700-site-009# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
K>* 0.0.0.0/0 [0/0] is directly connected, wwan0, 00:02:03
K>* 10.168.234.140/32 [0/0] via 10.168.236.9, wwan0, 00:02:03
C>* 10.168.236.9/32 is directly connected, wwan0, 00:02:03
C>* 192.168.1.0/24 is directly connected, lan0, 00:02:03nb2700-site-009# show evpn
L2 VNIs: 1
L3 VNIs: 0
Advertise gateway mac-ip: No
Advertise svi mac-ip: No
Duplicate address detection: Enable
Detection max-moves 5, time 180nb2700-site-009# show evpn vni detail
VNI: 6001
Type: L2
Tenant VRF: default
VxLAN interface: vxlan6001
VxLAN ifIndex: 21
Local VTEP IP: 10.168.236.9
Mcast group: 0.0.0.0
No remote VTEPs known for this VNI
Number of MACs (local and remote) known for this VNI: 1
Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 1
Advertise-gw-macip: NoOver WiFi More devices active
nb2700-site-009# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
K>* 0.0.0.0/0 [0/0] via 10.10.10.254, lan3, 00:00:25
K>* 10.168.234.140/32 [0/0] via 10.10.10.254, lan3, 00:00:25
C>* 10.10.10.0/24 is directly connected, lan3, 00:00:25
C>* 192.168.1.0/24 is directly connected, lan0, 00:00:25nb2700-site-009# show evpn
L2 VNIs: 2
L3 VNIs: 0
Advertise gateway mac-ip: No
Advertise svi mac-ip: No
Duplicate address detection: Enable
Detection max-moves 5, time 180nb2700-site-009# show evpn vni detail
VNI: 6001
Type: L2
Tenant VRF: default
VxLAN interface: vxlan6001
VxLAN ifIndex: 23
Local VTEP IP: 10.10.10.8
Mcast group: 0.0.0.0
Remote VTEPs for this VNI:
10.10.10.8 flood: HER
10.10.10.3 flood: HER
10.10.10.2 flood: HER
10.10.10.1 flood: HER
Number of MACs (local and remote) known for this VNI: 2
Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 8
Advertise-gw-macip: NoCan you try ip nht resolve-via-default ? It looks to me like the issue is that we aren't considering the Type 3 learned over LTE to be a remote VTEP because our only route to get to that peer is via the default route:
nb2700-site-009# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
K>* 0.0.0.0/0 [0/0] is directly connected, wwan0, 00:02:03 <<<<<<<<
K>* 10.168.234.140/32 [0/0] via 10.168.236.9, wwan0, 00:02:03
C>* 10.168.236.9/32 is directly connected, wwan0, 00:02:03 <<<<<< no subnet route here, only /32
C>* 192.168.1.0/24 is directly connected, lan0, 00:02:03Whereas on wifi we have a connected /24 which is more specific than the default:
nb2700-site-009# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,
O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP,
T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP,
F - PBR, f - OpenFabric,
> - selected route, * - FIB route, q - queued, r - rejected, b - backup
K>* 0.0.0.0/0 [0/0] via 10.10.10.254, lan3, 00:00:25
K>* 10.168.234.140/32 [0/0] via 10.10.10.254, lan3, 00:00:25
C>* 10.10.10.0/24 is directly connected, lan3, 00:00:25 <<<<<<<
C>* 192.168.1.0/24 is directly connected, lan0, 00:00:25My guess is that we'll see the Type 3 get accepted by allowing next-hop tracking to mark the BGP next-hop valid when it's only resolvable through a default route.
Can you give that a shot and let us know if the outcome changes?
getabc
commented
3 years ago That worked perfectly - thank you so much!
The devices are working across the private LTE network.
taspelund
commented
3 years ago Excellent, that's great news! And you're welcome, happy to help.
@polychaeta autoclose 1 day
donaldsharp
commented
3 years ago @polychaeta autoclose 1 day
Route reflector
VTEPs
Can we use VXLAN BGP EVPN over private LTE?
When I have it running in my lab on a mixture of WLAN and LAN interfaces everything works fine. However, when I try to run in on our private LTE test lab it doesn't work. The customer has an L2 network sending small amounts of data around, and the LTE network only supports L3.
L2VPN eBGP works on the route reflector.
bgp evpn route on the route reflector
When I compare the neighbour status on the route reflector from the LTE to the WLAN, I noticed the following was missing.
I can normally see the endpoints in the 'bridge fdb show' command but doesn't work over LTE
I'm getting only dropped packets on the vxlan6001 interface
Route Reflector
VTEP
Any help would be appreciated.