search

m13253 / VxWireguard-Generator

Utility to generate VXLAN over Wireguard mesh SD-WAN configuration
MIT License
150 stars 18 forks source link

Example Documentation #4

Closed bmullan closed 4 years ago

bmullan commented 4 years ago

I am confused by the Example in the current README.md ... and how/where I need to modify the following for my own 3 VM testing use case:

# Set the address pool of the network vwgen set wg-meshvpn pool-ipv4 172.20.10.0/24 pool-ipv6 2001:db8:42::/64

# Add 3 nodes # They will have IPv4 addresses fixed and IPv6 addresses dynamically calculated on demand vwgen add wg-meshvpn node1 node2 node3

# Set endpoint of node1 and node2, leave empty for node3 so it will do auto-discovery vwgen set wg-meshvpn node node1 endpoint '[2001:db8:1::1]:1234' listen-port 1234 vwgen set wg-meshvpn node node2 endpoint '[2001:db8:2::1]:2345' listen-port 2345 vwgen set wg-meshvpn node node3 listen-port 3456

# Show all information we have so far vwgen show wg-meshvpn

# Generate a configuration for node1 # It's a bad habit to log into server with root, this is just a demo vwgen showconf wg-meshvpn node1 > node1.conf scp node1.conf 'root@[2001:db8:1::1]:/etc/wireguard/wg-meshvpn.conf' ssh root@2001:db8:1::1 chmod 600 /etc/wireguard/wg-meshvpn.conf \; wg-quick up wg-meshvpn

I have 3 KVM VM's installed (all ubuntu 18.04)

node1 ipv4: 192.168.122.70/24 iov6 fe80::5054:ff:fe65:8203 prefixlen 64 scopeid 0x20

node2 ipv4: 192.168.122.89/24 ipv6 fe80::5054:ff:fe9b:2ece prefixlen 64 scopeid 0x20

node3 ipv4: 192.168.122.32/24 ipv6: fe80::5054:ff:feaa:4a12 prefixlen 64 scopeid 0x20

Now obviously none of my 3 VMs have an address space matching your "Example"

# Set the address pool of the network vwgen set wg-meshvpn pool-ipv4 172.20.10.0/24 pool-ipv6 2001:db8:42::/64

So do I "ignore" the above address pool configuration? Is that used only if you create VMs/Servers afterwards and manually allocate the "vwgen set wg-meshvpn pool-ipv4 and ipv6" to those?

If I should ignore the above pool allocation statement then do I use my existing ipv4 and ipv6 VM addresses in your "example" configuration statements shown below:

# Add 3 nodes # They will have IPv4 addresses fixed and IPv6 addresses dynamically calculated on demand vwgen add wg-meshvpn node1 node2 node3

# Set endpoint of node1 and node2, leave empty for node3 so it will do auto-discovery vwgen set wg-meshvpn node node1 endpoint '[2001:db8:1::1]:1234' listen-port 1234 vwgen set wg-meshvpn node node2 endpoint '[2001:db8:2::1]:2345' listen-port 2345 vwgen set wg-meshvpn node node3 listen-port 3456

thanks Brian

m13253 commented 4 years ago

vwgen has three layers of IP addresses:

  1. The endpoint. They are the addresses your ISP provided you. Wireguard uses these address to establish tunnels between nodes.
  2. The mid-layer. They are allocated randomly from 169.254.0.0/16 link-local address space, used to build the VXLAN on top of it.
  3. The inside address. They are allocated from pool-ipv4 and pool-ipv6. Since all nodes are now connected to one LAN, preferably you will want these address to reside in one single subnet (or you will have to use some routing protocol to make routing work). If you are not satisfied with the automatic allocation, for IPv4, you can edit the address manally; for IPv6, you can add more than one address and set it as primary.

Please tell me if you need more explanations. I will do my best to help you.

m13253 commented 4 years ago

If all your 3 nodes are inside a single, secured network. Let's say, a virtual bridge. What you really want may be a single VLAN or VXLAN (instead of vwgen). Depending on your virtualization platform, you might also be able to choose from NVGRE and Geneve besides them.

bmullan commented 4 years ago

@m13253 Actually my end-goal architecture overview might help you understand more. See the attached PNG. Each my goal is a multi-tenant, multi-node/host/server, multi-hybrid-cloud environment. Tenant's run in LXD "system" Containers (https://linuxcontainers.org/#LXD). Just in case you've not worked with LXD Containers, an LXD container runs a complete Linux distro with init, systemd etc but uses the Kernel of the Host/Node/Server. LXD supports containers of many Distro's (fedora, CentOS, Debian, Ubuntu, Alpine, Oracle etc). I use Ubuntu 18.04 for both Host/Node/Server and for the LXD Containers.

LXD normally would assign a "default" bridge (lxdbr0) to connect containers to their Host/Node/Server and the Internet. Normally, lxdbr0 is assigned a random 10.x.x.1 Network address and range of IP addresses for Containers on that Host to be assigned.

In my use-case Goal I create a individual per-tenant OVS bridges (tenant-A-br, tenant-B-br etc) and when creating an LXD container for Tenant A on Site 1,2,3 server or any Cloud Instance for Tenant A... I create those containers for Tenant A to use the tenant-A-br.

The tenant-A-br OVS bridge is created with a VXLAN VNID for Tenant A, as is similarly planned for Tenant B, C etc.

I am looking at 3 possible solutions.
full-mesh, auto-learning VPN

OVS/VXLAN w/ a ipsec or other VPN for encryption

DM-EVPN

I hope this helps you better understand what I'd like to accomplish as its a bit more complicated than just 3 VMs on a single host.

Click on the following to Enlarge the Picture...

CIAB Overview Diagram

Thanks Brian

m13253 commented 4 years ago

Thank you for your clear explanation! And I think I understood your situation.

I think VxWireguard is able to satisfy your demands. (Although I am not sure whether VpnCloud does this better or not.)

You only need to run Wireguard and VXLAN on your host machine. Since VxWireguard provides an Ethernet interface (starting with “vwg-”) for you to either run routing or bridging on, you can actually allocate IP address for your LXD containers using either DHCP or manual assignment.

1. Bridge

If you want to bridge your host network to the visualized network, you can simply use a single subnet across sites. For example:

Site 1 host: vwg-tenant-a: 10.0.0.1/16 (configure with vwgen)
             vwg-tenant-b: 10.1.0.1/16
             vwg-tenant-c: 10.2.0.1/16
Container A: eth0:         10.0.0.2/16 (configure manually or by DHCP)
Container B: eth0:         10.1.0.2/16
Container C: eth0:         10.2.0.2/16
Bridges: vwg-tenant-a ←→ Container A eth0
         vwg-tenant-b ←→ Container B eth0
         vwg-tenant-c ←→ Container C eth0

Site 2 host: vwg-tenant-a: 10.0.1.1/16
             vwg-tenant-b: 10.1.1.1/16
             vwg-tenant-c: 10.2.1.1/16
Container A: eth0:         10.0.1.2/16
Container B: eth0:         10.1.1.2/16
Container C: eth0:         10.2.1.2/16
Bridges: vwg-tenant-a ←→ Container A eth0
         vwg-tenant-b ←→ Container B eth0
         vwg-tenant-c ←→ Container C eth0

Site 3 host: vwg-tenant-a: 10.0.2.1/16
             vwg-tenant-b: 10.1.2.1/16
             vwg-tenant-c: 10.2.2.1/16
Container A: eth0:         10.0.2.2/16
Container B: eth0:         10.1.2.2/16
Container C: eth0:         10.2.2.2/16
Bridges: vwg-tenant-a ←→ Container A eth0
         vwg-tenant-b ←→ Container B eth0
         vwg-tenant-c ←→ Container C eth0

2. Route

If you want to route between these visualized networks, you can use alternative methods to allocate IPs. You can use either iBGP, RIP, OSPF, or EIGRP for automatic routing. In my lab network, I use Babel for routing, which is sometimes buggy and not suitable for production, but you can try it if you are running the network for experiments. For example:

Site 1 host: vwg-tenant-a: 10.0.0.1/24 (configure with vwgen)
             vwg-tenant-b: 10.1.0.1/24
             vwg-tenant-c: 10.2.0.1/24
Container A: eth0:         10.0.1.1/24 (configure manually or by DHCP)
Container B: eth0:         10.1.1.1/24
Container C: eth0:         10.2.1.1/24
Routing table: 10.0.2.0/24 via 10.0.0.2 dev vwg-tenant-a
               10.0.3.0/24 via 10.0.0.3 dev vwg-tenant-a
               10.1.2.0/24 via 10.1.0.2 dev vwg-tenant-b
               10.1.3.0/24 via 10.1.0.3 dev vwg-tenant-b
               10.2.2.0/24 via 10.2.0.2 dev vwg-tenant-c
               10.2.3.0/24 via 10.2.0.3 dev vwg-tenant-c

Site 2 host: vwg-tenant-a: 10.0.0.2/24
             vwg-tenant-b: 10.1.0.2/24
             vwg-tenant-c: 10.2.0.2/24
Container A: eth0:         10.0.2.1/24
Container B: eth0:         10.1.2.1/24
Container C: eth0:         10.2.2.1/24
Routing table: 10.0.1.0/24 via 10.0.0.1 dev vwg-tenant-a
               10.0.3.0/24 via 10.0.0.3 dev vwg-tenant-a
               10.1.1.0/24 via 10.1.0.1 dev vwg-tenant-b
               10.1.3.0/24 via 10.1.0.3 dev vwg-tenant-b
               10.2.1.0/24 via 10.2.0.1 dev vwg-tenant-c
               10.2.3.0/24 via 10.2.0.3 dev vwg-tenant-c

Site 3 host: vwg-tenant-a: 10.0.0.3/24
             vwg-tenant-b: 10.1.0.3/24
             vwg-tenant-c: 10.2.0.3/24
Container A: eth0:         10.0.3.1/24
Container B: eth0:         10.1.3.1/24
Container C: eth0:         10.2.3.1/24
Routing table: 10.0.1.0/24 via 10.0.0.1 dev vwg-tenant-a
               10.0.2.0/24 via 10.0.0.2 dev vwg-tenant-a
               10.1.1.0/24 via 10.1.0.1 dev vwg-tenant-b
               10.1.2.0/24 via 10.1.0.2 dev vwg-tenant-b
               10.2.1.0/24 via 10.2.0.1 dev vwg-tenant-c
               10.2.2.0/24 via 10.2.0.2 dev vwg-tenant-c

3. Raw Ethernet

You can also just plug the Ethernet into the container and let your tenant assign IP addresses by themselves. Since each VXLAN is isolated, they can do whatever they want in the virtualized network.

Also, just to mention that you can view or tune any settings with text editor, especially MTU. Wireguard + VXLAN take up 82 bytes of your MTU, so it might be better to do sufficient benchmark especially on a high-loss network or when the firewall drops fragments.

bmullan commented 4 years ago

@m13253 Thanks so much for the feedback.

The LXD basic/usual "default" networking is that all container's ETH0 interface connects to a single bridge named LXDBR0.

There is also DNSMASQ running which does the DHCP and DNS for each container attached to the LXDBR0 bridge.

In my use-case concept, each Multi-Tenant's containers would get their own individual, manually configured replacement of LXDBR0 and that Tenant's Bridge would provide the DNSMASQ for "that" Tenant's containers.

Hopefully, I can get Vx-WireGuard-Generator setup/configured correctly now that you've provided this good explanation/information.

Thanks Brian

bmullan commented 4 years ago

@m13253

Just wanted you to know that I got everything working for my use-case for a Multi-Tenant, Multi-Node/Server/Host, Multi-Cloud/Hybrid environment with all TenantN compute/storage in LXD Containers.

Click to enlarge:

CIAB Overview Diagram (generic)

I am using VxWireGuard-Generator to create the WireGuard configurations that include the VxLAN configs.

I am using BGP with BGP Route Reflectors configured to provide Routing information to each Node regarding the "other" NODE(s) LXD TenantN IP subnets as each TenantN sits behind their own indivualized & customized LXD managed Bridge.

The BGP will also enable Mobility of both Servers and TenantN LXD containers.

I did encounter a couple odd things with VxWireguard Generator that I will try to write up and let you know about.

As to performance... the following is throughput Testing using IPERF

On Hetzner Cloud Server (Germany) - In an LXD Container CN1, I ran IPERF in CLIENT Mode

root@cn1:~# iperf -c 10.0.1.19 -t 10

Client connecting to 10.0.1.19, TCP port 5001 TCP window size: 85.0 KByte (default)

[ 3] local 10.0.2.4 port 52974 connected with 10.0.1.19 port 5001 [ ID] Interval Transfer Bandwidth [ 3] 0.0-10.0 sec 191 MBytes 160 Mbits/sec

On Digital Ocean (U.S.A East Coast) Server - In an LXD Container CN0 -- I ran IPERF in SERVER Mode

root@cn0:~# iperf -s -b 500M

Server listening on TCP port 5001 TCP window size: 128 KByte (default)

[ 4] local 10.0.1.19 port 5001 connected with 169.254.216.90 port 52974 [ ID] Interval Transfer Bandwidth [ 4] 0.0-10.0 sec 191 MBytes 160 Mbits/sec

I have repeated tests many times using different servers/tenants and the throughput has been consistently 150-170Mbps

Not bad for transatlantic Traffic :-)

Brian Raleigh, North Carolina USA

If you'd like to use email for some communication about this: bmullan.mail@gmail.com

m13253 commented 4 years ago

Thank you for your sharing!

bmullan commented 4 years ago

@m13253

I decided to see if I could get FRR's Babel working to see how it performed compared to FRR's BGP.

I've installed vxwireguard-generator and have 3 nodes configured. As I'd mentioned earlier I'd previously installed FRR and when using BGPd it works fine

But for the life of me and after spending over a week trying all sorts of combinations of babel configuration I cannot get babel to work with this.

I am using FRR's babeld

I a putting my babeld configuration in to the /etc/frr/frr.conf file not the /etc/babeld.conf

Do you happen to have a FRR babel config file example that would work for the interfaces that get setup by vxwireguard ?

In my case, all three nodes I have the same interfaces

eth0 t1-br (a bridged interface to LXD containers) vwg-meshvpn wg-meshvpn

I remember you stating earlier:

In my lab network, I use Babel for routing, which is sometimes buggy and not suitable for production, but you can try it if you are running the network for experiments.

Were you just installing FRR's babel and using it?

So I hope I've not made stupid novice mistakes in configuring FRR babel. I'm still learning FRR and now Babel also.

thanks

m13253 commented 4 years ago

I haven't tried FRR. I will try it in the future.

The keypoint is to let FRR run Babel on multicast address of vwg-meshvpn. This way, every peer can discover each other automatically.

bmullan commented 4 years ago

I "think" I am running babel on vwg-meshvpn If I configured it right

Brian

On Sat, Feb 22, 2020, 1:06 AM Star Brilliant notifications@github.com wrote:

I haven't tried FRR. I will try it in the future.

The keypoint is to let FRR run Babel on multicast address of vwg-meshvpn. This way, every peer can discover each other automatically.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/m13253/VxWireguard-Generator/issues/4?email_source=notifications&email_token=AAM23JYVEIU5JAAIBP3AXRDREC6GXA5CNFSM4JMITPDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMUYFCY#issuecomment-589922955, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM23J5JUCTBOXWHI6DD3PDREC6GXANCNFSM4JMITPDA .

Jamesits commented 4 years ago

A very basic FRRouting 7.x config for babel:

ip forwarding
ipv6 forwarding
router babel
 network vwg-meshvpn

Wait a moment and you should see the neighbor coming up in vtysh:

(this is my lab environment, interface names and IPs are different)

router1# show babel neighbor
Neighbour fe80::250:1ff:fe00:200 dev ens3 reach ffff rxcost 256 txcost 256 rtt 0.000 rttcost 0.

And as long as you are running tunnels you should be careful with redistribute [ipv4|ipv6] connected and do filter out tunnel endpoints using route-map.

It is not recommended to manually edit FRRouting config file, and FRR's config is not compatible with Babeld config. Use vtysh's configure terminal mode to edit the config online, and use write memory to save the config.

bmullan commented 4 years ago

James

Thanks so much for replying!

I'd had a couple email exchanges one of the frr devs.

One question I'd asked was why some of the few config examples I read used only the /etc/frr/frr.conf file and others separated the Babel config into a 2nd file babeld.conf

His reply was that you can do either? But he didn't elaborate on that.

So I started searching/reading more and found out you can use either a "integrated" /etc/frr/frr.conf or separate config files for each Daemon.

One thing I've learned with both FRR and Babel is documentation is murky sometimes 😉

So just fyi.... on this topic of config files:

From this section of one of the FRR web pages: http://docs.frrouting.org/en/latest/vtysh.html http://docs.frrouting.org/en/latest/vtysh.html

it seems to indicate using separate .conf files for each daemon you activate via the /etc/frr/daemons file:

We recommend you do not mix the use of the two types of files.

Further, it is better not to use the integrated frr.conf file, as any syntax error in it can lead to all of your daemons being unable to start up.

Per daemon files are more robust as impact of errors in configuration are limited to the daemon in whose file the error is made.

BUT... Then I see a web page like this: http://labn.net/frr.html/Integrated-configuration-mode.html

which says...

Integrated configuration mode uses a single configuration file, frr.conf,

for all daemons. This replaces the individual files like zebra.conf or bgpd.conf.

frr.conf is located in /usr/local/etc.

All daemons check for the existence of this file at startup, and if it exists will not load their individual configuration files.

Instead, vtysh -b must be invoked to process frr.conf and apply its settings to the individual daemons.

AND this... https://www.2stacks.net/blog/getting-started-with-frr-on-eveng/ https://www.2stacks.net/blog/getting-started-with-frr-on-eveng/

where it gives a bit more info on use of "integrated" vs "non-integrated" config files and how vtysh.conf is involved:

Disable integrated configuration file (optional) https://www.2stacks.net/blog/getting-started-with-frr-on-eveng/#disable-integrated-configuration-file-optionalBy default FRR uses an Integrated configuration mode http://docs.frrouting.org/en/latest/vtysh.html#integrated-configuration-mode where all configs are kept in a single frr.conf file.

Though optional, it is recommended to use a configuration file per daemon. To disable the use of the integrated frr.conf and instead use a separate configuration file per daemon, first edit the vtysh.conf file.

$ sudo sed -i 's/^service/no service/g' /etc/frr/vtysh.conf

https://www.2stacks.net/blog/getting-started-with-frr-on-eveng/#disable-integrated-configuration-file-optional

I'm still having no luck pinging from one node to another's t1-br interface. But I'll keep looking/trying to fix it. I may punt and delete the servers on hetzner and digital ocean clouds (node0 and node1) and also the local VM (node2) and start over clean in case some artifact from all of my failed attempts so far is still on one or more of these systems.

brian

On Sat, Feb 22, 2020 at 6:15 AM James Swineson notifications@github.com wrote:

A very basic FRRouting 7.x config for babel:

ip forwarding ipv6 forwarding router babel network vwg-meshvpn

Wait a moment and you should see the neighbor coming up in vtysh:

(this is my lab environment, interface names and IPs are different)

router1# show babel neighbor Neighbour fe80::250:1ff:fe00:200 dev ens3 reach ffff rxcost 256 txcost 256 rtt 0.000 rttcost 0.

And as long as you are running tunnels you should be careful with redistribute [ipv4|ipv6] connected and do filter out tunnel endpoints using route-map.

It is not recommended to manually edit FRRouting config file, and it is not compatible with Babeld config. Use vtysh's configure terminal mode to edit the config online, and use write memory to save the config.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/m13253/VxWireguard-Generator/issues/4?email_source=notifications&email_token=AAM23J6HXYKRZ7B7BVKSZZLREECNTA5CNFSM4JMITPDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMU53FQ#issuecomment-589946262, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM23J3L4SXSRKYVRJDS62LREECNTANCNFSM4JMITPDA .

Jamesits commented 4 years ago

OK, so there is actually an option to use separate config files for each daemon. Still I'd recommend using the intergrated config mode since FRR itself is built as a Cisco IOS copycat and this is what most people use.

About your testbed, I have a few questions:

bmullan commented 4 years ago

James

Node0 - on Hetzner Cloud (Germany)

node0# sh babel neighbor Neighbour fe80::e4ef:f9ff:fe13:1a1a dev vwg-meshvpn reach 0002 rxcost 65535 txcost 65535 rtt 0.000 rttcost 0. Neighbour fe80::e0d0:cbff:fe87:3f51 dev vwg-meshvpn reach 3ff8 rxcost 65535 txcost 292 rtt 0.000 rttcost 0.

node0# sh ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric,

  • selected route, * - FIB route, q - queued route, r - rejected route

K> 0.0.0.0/0 [0/0] via 172.31.1.1, eth0, 00:07:13 C> 10.0.1.0/24 is directly connected, t1-br, 00:07:13 A> 10.0.2.0/24 [100/0] unreachable (ICMP unreachable), 00:03:00 A> 10.0.3.0/24 [100/0] unreachable (ICMP unreachable), 00:02:18 A> 64.227.0.0/20 [100/0] unreachable (ICMP unreachable), 00:03:00 C> 88.198.118.208/32 is directly connected, eth0, 00:07:13 C> 169.254.0.0/16 is directly connected, wg-meshvpn, 00:07:13 C> 172.20.10.0/24 is directly connected, vwg-meshvpn, 00:07:13 K> 172.31.1.1/32 [0/0] is directly connected, eth0, 00:07:13 A> 192.168.122.0/24 [100/0] unreachable (ICMP unreachable), 00:02:18

Node1 - on Digital Ocean (USA)

node1# sh babel neigh Neighbour fe80::e0d0:cbff:fe87:3f51 dev vwg-meshvpn reach 001f rxcost 65535 txcost 65535 rtt 0.000 rttcost 0. Neighbour fe80::6488:2aff:fe5e:5948 dev vwg-meshvpn reach 001f rxcost 65535 txcost 65535 rtt 0.000 rttcost 0.

node1# sho ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric,

  • selected route, * - FIB route, q - queued route, r - rejected route

K> 0.0.0.0/0 [0/0] via 64.227.0.1, eth0, 00:06:58 A> 10.0.1.0/24 [100/0] unreachable (ICMP unreachable), 00:00:13 C> 10.0.2.0/24 is directly connected, t1-br, 00:06:58 A> 10.0.3.0/24 [100/0] unreachable (ICMP unreachable), 00:00:21 C> 64.227.0.0/20 is directly connected, eth0, 00:06:58 A> 88.198.118.208/32 [100/0] unreachable (ICMP unreachable), 00:00:13 C> 169.254.0.0/16 is directly connected, wg-meshvpn, 00:06:58 C> 172.20.10.0/24 is directly connected, vwg-meshvpn, 00:06:58 A>* 192.168.122.0/24 [100/0] unreachable (ICMP unreachable), 00:00:21

Node2 - VM on my home server

node2# sh babel neigh Neighbour fe80::e4ef:f9ff:fe13:1a1a dev vwg-meshvpn reach 0010 rxcost 65535 txcost 65535 rtt 0.000 rttcost 0. Neighbour fe80::6488:2aff:fe5e:5948 dev vwg-meshvpn reach 001e rxcost 65535 txcost 96 rtt 0.000 rttcost 0.

node2# sh ip route Codes: K - kernel route, C - connected, S - static, R - RIP, O - OSPF, I - IS-IS, B - BGP, E - EIGRP, N - NHRP, T - Table, v - VNC, V - VNC-Direct, A - Babel, D - SHARP, F - PBR, f - OpenFabric,

  • selected route, * - FIB route, q - queued route, r - rejected route

K> 0.0.0.0/0 [0/100] via 192.168.122.1, ens3, src 192.168.122.228, 00:07:48 A> 10.0.1.0/24 [100/0] unreachable (ICMP unreachable), 00:01:01 A> 10.0.2.0/24 [100/0] unreachable (ICMP unreachable), 00:01:02 C> 10.0.3.0/24 is directly connected, t1-br, 00:07:48 A> 64.227.0.0/20 [100/0] unreachable (ICMP unreachable), 00:01:02 A> 88.198.118.208/32 [100/0] unreachable (ICMP unreachable), 00:01:01 C> 169.254.0.0/16 is directly connected, wg-meshvpn, 00:07:48 C> 172.20.10.0/24 is directly connected, vwg-meshvpn, 00:07:48 C> 192.168.122.0/24 is directly connected, ens3, 00:07:48 K> 192.168.122.1/32 [0/100] is directly connected, ens3, 00:07:48

On Sat, Feb 22, 2020 at 9:58 AM James Swineson notifications@github.com wrote:

OK, so there is actually an option to use separate config files for each daemon. Still I'd recommend using the intergrated config mode since FRR itself is built as a Cisco IOS copycat and this is what most people use.

About your testbed, I have a few questions:

  • From any node, can you ping the adjacent node using its VXLAN interface IP?
  • Can you do a show ip route and a show babel neighbor in vtysh on any of your nodes?

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/m13253/VxWireguard-Generator/issues/4?email_source=notifications&email_token=AAM23J7R5OFQKXFRQHXPPR3REE4QFA5CNFSM4JMITPDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEMVCETI#issuecomment-589963853, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAM23J5NTSN2DYVTOJOR3CLREE4QFANCNFSM4JMITPDA .

Jamesits commented 4 years ago

I think rxcost 65535 simply means it is unidirectionly unreachable since the rxcost is valid between 1 and 65534. Also babel is injecting unreachable routes into your routing table which means it has lost contact with its peer. I think there are probably some connectivity issue, high packet drop rate, or MTU mismatch between your nodes. Can they ping each other using their IPv6 link local addresses? For example, on node 0:

ping fe80::e4ef:f9ff:fe13:1a1a%vwg-meshvpn
ping fe80::e0d0:cbff:fe87:3f51%vwg-meshvpn
bmullan commented 4 years ago

@Jamesits thanks for your comment... I went back and looked at IPv6 addresses and somehow I'd put the same on 2 of the 3 different servers. So I'm sure Babel was really confused about that.

bmullan commented 4 years ago

@Jamesits James Wanted to let you know that after spending two weeks trying to get babel to work I've given up.

FRR Babel may work for alot of people but I must missing something critical.

VxWireguard, FRR, BGPd works great.

So my experiment w babel is over for now.

Thanks Brian