vrrp virtual IP address not available through gateway with L3PROTECT enabled

[ghost]

Hi There,

http://wiki.smartos.org/display/DOC/Managing+NICs#ManagingNICs-AddingVRRPnicstoVMs

Following the above guide to do VRRP, but with public IPs instead of internal ones with a gateway defined, the VIP IP address comes up inaccessible from the wider internet. It is however reachable from hosts on the same subnet.

I had a hunch, and thought it might be related to ip_strict_src_multihoming.

So I checked "/usr/sbin/ndd /dev/ip ip_strict_src_multihoming", and this was returning 1.

I tried doing this:

/usr/sbin/ndd -set /dev/ip ip_strict_src_multihoming 0 /usr/sbin/ndd -set /dev/ip ip_strict_src_multihoming 1

Then the IP sprang back to life. It's 100% reproducible - simply halt the zone, boot it, IP comes up but inaccessible from the net. Run the above, and it springs into life.

It's also possible to work around the problem by disabling L3PROTECT.

So this is some kind of weird interaction between L3PROTECT, ip_strict_src_multihoming and VRRP IP addresses.

[AlainODea]

I had a similar problem, but with private IPs. They were inaccessible regardless of subnet. This was resolvable for me when I added "allow_ip_spoofing": true to the NIC whose ip matches vrrp_primary_ip.

Original VM Payload that Exhibited the Problem

For example I have this payload:

{
  "alias": "proxy",
  "dataset_uuid": "84cb7edc-3f22-11e2-8a2a-3f2a7b148699",
  "nics": [
    {
      "nic_tag": "external",
      "vlan_id": 7,
      "ip": "10.101.0.4",
      "netmask": "255.255.0.0",
      "gateway": "10.101.0.1",
      "vrrp_vrid": 1,
      "vrrp_primary_ip": "10.101.0.6"
    },
    {
      "nic_tag": "external",
      "vlan_id": 7,
      "ip": "10.101.0.6",
      "netmask": "255.255.0.0",
      "gateway": "10.101.0.1",
      "primary": true
    }
  ]
}

I then enabled VRRP:

zlogin <UUID> vrrpadm create-router -V 1 -l net1 -A inet router0

I then checked reachability with ICMP ECHO:

ping 10.101.0.6

It returned 10.101.0.6 is alive.

I then checked reachability to the VRRP VIP with ICMP ECHO:

ping 10.101.0.4

It returned no answer from 10.101.0.4.

Modified VM Payload that Worked

I then tried adding "allow_ip_spoofing": true to the non-VRRP NIC in the payload:

{
  "alias": "proxy",
  "dataset_uuid": "84cb7edc-3f22-11e2-8a2a-3f2a7b148699",
  "nics": [
    {
      "nic_tag": "external",
      "vlan_id": 7,
      "ip": "10.101.0.4",
      "netmask": "255.255.0.0",
      "gateway": "10.101.0.1",
      "vrrp_vrid": 1,
      "vrrp_primary_ip": "10.101.0.6"
    },
    {
      "nic_tag": "external",
      "vlan_id": 7,
      "ip": "10.101.0.6",
      "netmask": "255.255.0.0",
      "gateway": "10.101.0.1",
      "primary": true,
      "allow_ip_spoofing": true
    }
  ]
}

I then enabled VRRP:

zlogin <UUID> vrrpadm create-router -V 1 -l net1 -A inet router0

I then checked reachability to the zone's own IP with ICMP ECHO:

ping 10.101.0.6

It returned 10.101.0.6 is alive.

I then checked reachability to the VRRP VIP with ICMP ECHO:

ping 10.101.0.4

It returned 10.101.0.4 is alive.

Experiments with Crossbow Configuration

I then re-enabled L3PROTECT on net1:

protection=$(dladm show-linkprop -z <UUID> net1 -p protection -o value | tail +2 | tr -d '\n')
dladm set-linkprop -z <UUID> net1 -p protection=$protection,ip-nospoof

I then checked reachability to the VRRP VIP with ICMP ECHO:

ping 10.101.0.4

It returned no answer from 10.101.0.4.

I tried to trick my way around this by allowing the VRRP VIP on net1:

ips=$(dladm show-linkprop -z <UUID> net1 -p allowed-ips -o value | tail +2 | tr -d '\n')
dladm set-linkprop -z <UUID> net1 -p allowed-ips=$ips,10.101.0.4

I then checked reachability to the VRRP VIP with ICMP ECHO:

ping 10.101.0.4

It returned no answer from 10.101.0.4.

So gaming allowed-ips did not work.

Known Workaround

When add "allow_ip_spoofing": true to the NIC whose ip matches vrrp_primary_ip.

Future Work

Is there a way to see the L3PROTECT logs in the GZ to see what the VM is attempting that is being rejected?

[AlainODea]

I have updated http://wiki.smartos.org/display/DOC/Managing+NICs#ManagingNICs-AddingVRRPnicstoVMs to include my workaround so it will work out of the box for people trying it.

Is there a way that L3PROTECT can be made to work with VRRP?

[alcir]

If I configure two zones on two different servers, I cannot ping the vrrp IP address from the GZ where the backup vrrp router is located, as well from all the zones located on it. I think that this happens because the GZ have that the mac address listed between her nics...