[BUG] Prod virts don't have a route out to internet

[FatmanUK]

They can ping the host (which shouldn't be possible with user mode networking?) so it might be a physical host routing issue. Not iptables as that's down currently.

[FatmanUK]

This needs to end up in a san role.

Restart libvirt and record the iptables, then reload with mods. Backport into project.

[FatmanUK]

Appears I can bring iptables up with libvirt and add these instructions to access the virts, at least:

iptables -I LIBVIRT_FWO 1 -s 10.2.0.0/16 -i virbr0 -j ACCEPT
iptables -I LIBVIRT_FWI 1 -d 10.2.0.0/16 -o virbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
iptables -D LIBVIRT_FWO 3
iptables -D LIBVIRT_FWI 3
iptables -tnat -I LIBVIRT_PRT 1 -j RETURN -s 10.2.0.0/16 -d 255.255.255.255/32
iptables -tnat -I LIBVIRT_PRT 1 -j RETURN -s 10.2.0.0/16 -d 224.0.0.0/24
iptables -tnat -I LIBVIRT_PRT 5 -j MASQUERADE -s 10.2.0.0/16 ! -d 10.2.0.0/16 -p tcp --to-ports 1024-65535
iptables -tnat -I LIBVIRT_PRT 5 -j MASQUERADE -s 10.2.0.0/16 ! -d 10.2.0.0/16 -p udp --to-ports 1024-65535
iptables -tnat -I LIBVIRT_PRT 5 -j MASQUERADE -s 10.2.0.0/16 ! -d 10.2.0.0/16

(Not 100% sure on this, needs testing.)

[FatmanUK]

https://serverfault.com/questions/1010806/iptables-port-forward-not-working-with-libvirtd-vm

The problem here comes from the following chains:

-A LIBVIRT_FWI -o virbr0 -j REJECT --reject-with icmp-port-unreachable -A LIBVIRT_FWO -i virbr0 -j REJECT --reject-with icmp-port-unreachable

They are added by libvirtd according to the network config in libvirt, if forward mode is set to nat.

One possible solution is to change the mode into open. However, be careful because you have to manually add forward rules by the following command:

virsh net-edit default

and setting something like the following:

<network> <name>default</name> <uuid>......</uuid> <forward mode='open'/> [...] </network>

[FatmanUK]

/etc/libvirt/qemu/networks/default.xml

❯ sudo virsh net-edit default
❯ sudo virsh net-destroy default
❯ sudo virsh net-start default

need to define networks and set autostart

host:

<network>
  <name>default</name>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <domain name='default'/>
  <ip address='10.2.1.0' netmask='255.255.255.0'>
  </ip>
  ...
</network>

control:

<network>
  <name>default</name>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <ip address='10.1.1.254' prefix='24'>
  </ip>
  ...
</network>

[FatmanUK]

Seems to work now, kinda.

[FatmanUK]

Interface virbr0 looks happier on all environments, and ...

ansible -i inventories/dev virts_cluster -mcommand -a'ping -c 4 208.67.222.222'
ansible -i inventories/prod virts_cluster -mcommand -a'ping -c 4 208.67.222.222'

... makes a smiley face.