zerotier / ZeroTierOne

A Smart Ethernet Switch for Earth
https://zerotier.com
Other
14.56k stars 1.7k forks source link

Multipath bond show command outputs no data #1727

Open grapexy opened 2 years ago

grapexy commented 2 years ago

I have 2 boxes in the network, both of them having direct link and using version 1.10.1:

  1. Opnsense with 2 WAN links (through vlan04 and vlan05 interfaces) - 172.22.0.1
  2. Opnsense on VPS with 1 WAN - 172.22.1.1

And another linux box connected to OP1. I'm trying to send all outgoing traffic for linux box through the OP1, using both links with multipath, via OP2 on VPS configured as gateway.

  1. OP1 configuration:

local.conf:

{
    "physical": {
        "192.168.0.0/16": {
            "blacklist": true
        },
        "10.0.0.0/8": {
            "blacklist": true
        }
    },
    "settings": {
        "portMappingEnabled": false,
        "defaultBondingPolicy": "custom-balance-aware",
        "policies": {
            "custom-balance-aware": {
                "basePolicy": "balance-aware",
                "links": {
                    "vlan04": {"speed": 10000},
                    "vlan05": {"speed": 1000}
                }
            }
        }
    }
}

Outbound NAT rule: ZeroTier interface > ZT interface address (172.22.0.1) Gateway for linux box: 172.22.1.1

  1. OP2 (VPS) local.conf:
    {
    "physical": {},
    "settings": {
        "portMappingEnabled": false
    }
    }

This makes all outgoing connections on linux box to exit via OP2 WAN, and I can confirm that exit IP is OP2's public IP.

However, multipath is not working. All traffic is sent out using single WAN from OP1.

On OP1, zerotier-cli bond {VPS_ID} show output:

root@firewall:~ # zerotier-cli bond {VPS_ID} show
Peer               : {VPS_ID}
Bond               : balance-aware
Link Select Method : 0
Links              : 6/6
Failover Interval  : 5000 (ms)
Up Delay           : 0 (ms)
Down Delay         : 0 (ms)
Packets Per Link   : 0 (ms)

     Interface Name                      Path    Alive
--------------------------------------------------------------------------------
[0]          vlan04                           {VPS_PUBLIC_IP}/9993            0
[1]          vlan04                          {VPS_PUBLIC_IP}/58807            0
[2]          vlan05                           {VPS_PUBLIC_IP}/9993            0
[3]          vlan05                          {VPS_PUBLIC_IP}58807            0
[4]          vlan04                          {VPS_PUBLIC_IP}/58807            0
[5]          vlan05                           {VPS_PUBLIC_IP}/9993            0

        Latency     Jitter     Loss     Error        Speed   Alloc
--------------------------------------------------------------------------------
[0]       0.000      0.000    0.000     0.000            0       0
[1]       0.000      0.000    0.000     0.000            0       0
[2]       0.000      0.000    0.000     0.000            0       0
[3]       0.000      0.000    0.000     0.000            0       0
[4]       0.000      0.000    0.000     0.000            0       0
[5]       0.000      0.000    0.000     0.000            0       0

These numbers are not affected in any way when the traffic is generated. I've also tried different bonding policies and same thing is happening. Is there something I'm missing?

grapexy commented 2 years ago

I figured out why aggregation was not happening - remote VPS peer had to have the same aggregation policy set. I don't know why remote's policy (which only has a single wan) affects another peer's policy, but changing local.conf on remote to the following enabled aggregation:

{
    "physical": {},
    "settings": {
         "portMappingEnabled": false,
        "defaultBondingPolicy": "custom-balance-aware",
        "policies": {
            "custom-balance-aware": {
                "rebalanceStrategy": "aggressive",
                "basePolicy": "balance-aware"
            }
        }
    }
}

zerotier-cli bond {VPS_ID} show still shows all 0s though.