sandia-minimega / minimega

minimega
GNU General Public License v3.0
148 stars 66 forks source link

bug: taps don't work like vms in ns bridges on multiple nodes #1386

Open mkunz7 opened 4 years ago

mkunz7 commented 4 years ago

Describe your environment

  1. minimega version latest
  2. Linux distro/version ubuntu 18.04.3 desktop
  3. Go compiler version 1.12.13
  4. VM types all

Describe the bug Taps don't work properly on ns bridges with multiple nodes

To Reproduce Launch minimega on three nodes in the same context.

ns add-hosts all
ns bridge test
vm config net test,foo
vm config memory 256
vm config cdrom tinycore.iso
vm launch kvm 6
vm start all
tap create foo bridge test ip 1.0.0.1/24
dnsmasq start 1.0.0.1 1.0.0.2 1.0.0.254

the vms will not get dhcp reservations, although ifconfig's packet counts are increasing for both send and receive

on a vm set a static ip of 1.0.0.2 and ping 1.0.0.1

sudo ifconfig eth0 1.0.0.2
ping 1.0.0.1

ping will fail

On the head node start a ping

ping 1.0.0.2

After a few seconds ping will succeed and the ping from the vm will succeed back to the head node, but vms will not get dhcp reservations.

Same issue exists with vxlan tunnels.

Same commands work fine if I only use one node instead of three.

With three nodes, why is it dhcp traffic happens over layer2 when I boot a vm in the cluster?

clear vm config
vm config kernel minirouter.kernel
vm config initrd minirouter.initrd
vm config net test,foo
vm launch kvm r0
vm start r0

wait for it to boot

router r0 interface 0 2.0.0.1/24
router r0 dhcp 2.0.0.1 range 2.0.0.2 2.0.0.254
router r0 commit

vm should get ip addresses in 2.0.0.0/24

So layer 2 traffic is happening on a vm in a gre tunnel, but doesn't happen when a tap is made on that same gre tunnel.

Expected behavior From my understanding by default layer 2 traffic shouldn't be transmitted on a layer 3 tunnel which would explain why I can't get DHCP to happen, but ping works. Taps should either allow layer 2 traffic on gre tunnels or vms should not allow layer 2 traffic on gre tunnels. Ideally I think the taps would function the same as a VM in the cluster. I feel like I'm going crazy here.