Can't get networking to work

[togoetha]

Hi,

Currently trying to integrate OSv unikernels into a larger network, but experiencing some issues. Unikernels are launched through the golang library, setting an IP generated by an orchestrator rather than using DHCP. I boiled it down to the most basic scenario, cutting out every other piece of the software (in fact, I can just copypaste the generated command and get the same results from bash):

• use virbr0
• set image IP to a static 192.168.122.2 (bridge is .1), gateway .1
• qemu bridge helper runs correctly, ACL file OK

IP is set correctly according to this screenshot: (nevermind the program's garbage output - see dummy code at the bottom)

However, I can't ping the VM or do anything else - Wireshark doesn't register anything in or out, not even ARP. ARP entry stays "(incomplete)".
This scenario is actually a bit worse than what I had first, when the software created the bridge and TAP devices for QEMU/KVM to use. In that case, Wireshark showed ARP requests for the VM IP and correct responses from the VM (!), but these weren't properly forwarded to/processed by the bridge for some reason. Manually adding the ARP record got ping requests to the TAP device, but no responses. No clue if/where they were dropped after that. There were some custom eBPF programs attached to the bridge, but logging correctly showed they didn't influence the ARP frames/ping packets.

I feel like I'm missing something really basic here, or could this be some sort of issue with specific parameters I'm using? Since most of the problems happen right at the TAP device, I suspect it's more of an OSv issue than QEMU/KVM.

Details:

Command I used (some boilerplate stuff, some requirements for the higher level software):

/usr/bin/qemu-system-x86_64 \
  -m 1024 \
  -smp 1 \
  -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0 \
  -drive file=/root/.capstan/instances/qemu/vmtest_debug_app2/disk.qcow2,if=none,id=hd0,aio=threads,cache=none \
  -chardev stdio,mux=on,id=stdio,signal=off \
  -device isa-serial,chardev=stdio \
  -netdev bridge,id=hn0,br=virbr0,helper=/usr/lib/qemu/qemu-bridge-helper \
  -device virtio-net-pci,netdev=hn0,id=nic1,mac=f2:f4:22:f5:82:0f \
  -chardev socket,id=charmonitor,path=/root/.capstan/instances/qemu/vmtest_debug_app2/osv.monitor,server=on,wait=off \
  -mon chardev=charmonitor,id=monitor,mode=control \
  -enable-kvm \
  -cpu host,+x2apic \
  -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/shm,share=on \
  -numa node,memdev=mem 

And the code it runs, which is basically just an endless sleep (which I sure hope isn't a case of PEBCAK):

int main()
{
  printf("Hello Edge\n");
  fflush(stdout);
  sleep(INT_MAX);
  return 0;
}

[togoetha]

Some incomprehensible deep magic just happened and it works as designed with the orchestrator handling things. Can be ignored for now while I look for reproducibility.

[togoetha]

Stuck again in the above situation after rebooting the development machine, pings make it to the TAP (and presumably VM) but no response. So it's down to some sort of network setting that affects the TAP interface. Suggestions are welcome. Using Ubuntu 24.04, tried iptables accept all INPUT/OUTPUT but didn't fix the issue.

[togoetha]

Additional information: The VM is still started using the same cmdline:

Setting cmdline: --rootfs=zfs --verbose --ip=eth0,10.57.228.18,255.255.255.0 --defaultgw=10.57.228.16 /main
Invoking QEMU at: /usr/bin/qemu-system-x86_64 with arguments:
  -vnc unix:/root/.capstan/instances/qemu/vmtest_debug_app/vnc-domain-socket
  -m 1024
  -smp 1
  -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0
  -drive file=/root/.capstan/instances/qemu/vmtest_debug_app/disk.qcow2,if=none,id=hd0,aio=threads,cache=none
  -chardev stdio,mux=on,id=stdio,signal=off
  -device isa-serial,chardev=stdio
  -netdev tap,id=hn0,ifname=vtapdf56,script=no,downscript=no
  -device virtio-net-pci,netdev=hn0,id=nic1,mac=4b:b6:0c:c0:be:2b
  -chardev socket,id=charmonitor,path=/root/.capstan/instances/qemu/vmtest_debug_app/osv.monitor,server=on,wait=off
  -mon chardev=charmonitor,id=monitor,mode=control
  -enable-kvm
  -cpu host,+x2apic
  -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/shm,share=on
  -numa node,memdev=mem

When pinging from the host, correct ARP responses from the VM get to the TAP device, but aren't processed by the bridge (stays incomplete in arp):

When fixing the ARP table manually, it just gets stuck like this:

[togoetha]

Case closed! Was generating bad MAC addresses some of the time.