search

cloudius-systems / osv

OSv, a new operating system for the cloud.
osv.io
Other
4.11k stars 604 forks source link

iperf3 - failed to open /dev/urandom #1333

Open togoetha opened 1 month ago

togoetha commented 1 month ago

Doing some lightweight throughput tests for a Kubernetes network architecture that supports OSv, but running into an odd problem where iperf3 can't find /dev/urandom. The images are built with Capstan, most basic possible config requiring osv.iperf3 and start command /tools/iperf3 -s .

The image starts fine, but as soon as any client connects I get the following output:

sudo /usr/bin/qemu-system-x86_64 -vnc unix:/root/.capstan/instances/qemu/vmtest_debug_app/vnc-domain-socket    -m 1024    -smp 1    -device virtio-blk-pci,id=blk0,bootindex=0,drive=hd0    -drive file=/root/.capstan/instances/qemu/vmtest_debug_app/disk.qcow2,if=none,id=hd0,aio=threads,cache=none    -chardev stdio,mux=on,id=stdio,signal=off    -device isa-serial,chardev=stdio    -netdev tap,id=hn0,ifname=vtap3391,script=no,downscript=no   -device virtio-net-pci,netdev=hn0,id=nic1,mac=46:0c:3b:7f:c0:c7    -chardev socket,id=charmonitor,path=/root/.capstan/instances/qemu/vmtest_debug_app/osv.monitor,server=on,wait=off    -mon chardev=charmonitor,id=monitor,mode=control    -enable-kvm    -cpu host,+x2apic    -object memory-backend-file,id=mem,size=1024M,mem-path=/dev/shm,share=on    -numa node,memdev=mem  
OSv v0.57.0-183-g9c152813
1 CPUs detected
Firmware vendor: Unknown
bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
vga: Add VGA device instance
eth0: ethernet address: 46:0c:3b:7f:c0:c7
devfs: created device vblk0.1 for a partition at offset:6291456 with size:10731126784
virtio-blk: Add blk device instances 0 as vblk0, devsize=10737418240
random: intel drng, rdrand registered as a source.
random: <Software, Yarrow> initialized
VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
BSD shrinker: event handler list found: 0x600000fed900
    BSD shrinker found: 1
BSD shrinker: unlocked, running
eth0: 10.184.66.26
Booted up in 136.05 ms
Cmdline: /tools/iperf3 -s
Running from /init/40-kubernetes: /kubernetes.so !

kubernetes: failed to open /run/kubernetes/configmaps.
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
random: device unblocked.
Accepted connection from 10.184.66.16, port 53268
iperf3: error - failed to open /dev/urandom: No such file or directory

program exited with status 1
VFS: unmounting /dev
VFS: unmounting /proc
Warning: unmount_rootfs: failed to unmount /proc, error = Invalid argument
VFS: unmounting /
Powering off.

From what I can see in the code OSv should definitely be creating the urandom device. Is this an image/OSv issue or is the Capstan iperf package/app no longer the way to go and should I just compile iperf3 straight from source instead?

Edit: looking at the iperf app, seems like it's just a pie recompile of default iperf3. I'll just modify iperf3 source to work around this in the meantime, although that's a bit hacky.

wkozaczuk commented 1 month ago

I have just built an image with the iperf3 from apps and it seems not to work:

OSv v0.57.0-183-g9c152813
1 CPUs detected
Firmware vendor: SeaBIOS
: ignoring missing symbol fsgsbase_avail
bsd: initializing - done
VFS: mounting ramfs at /
VFS: mounting devfs at /dev
net: initializing - done
vga: Add VGA device instance
eth0: ethernet address: 52:54:00:12:34:56
devfs: created device vblk0.1 for a partition at offset:6291456 with size:530579456
virtio-blk: Add blk device instances 0 as vblk0, devsize=536870912
random: virtio-rng registered as a source.
random: intel drng, rdrand registered as a source.
random: <Software, Yarrow> initialized
VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
BSD shrinker: event handler list found: 0x600000f45800
    BSD shrinker found: 1
BSD shrinker: unlocked, running
[I/17 dhcp]: Broadcasting DHCPDISCOVER message with xid: [2035976780]
[I/17 dhcp]: Waiting for IP...
[I/191 dhcp]: Received DHCPOFFER message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15
[I/191 dhcp]: Broadcasting DHCPREQUEST message with xid: [2035976780] to SELECT offered IP: 192.168.122.15
[I/191 dhcp]: Received DHCPACK message from DHCP server: 192.168.122.1 regarding offerred IP address: 192.168.122.15
[I/191 dhcp]: Server acknowledged IP 192.168.122.15 for interface eth0random: device unblocked.
 with time to lease in seconds: 86400
eth0: 192.168.122.15
[I/191 dhcp]: Configuring eth0: ip 192.168.122.15 subnet mask 255.255.255.0 gateway 192.168.122.1 MTU 1500
Booted up in 303.56 ms
Cmdline: /tools/iperf3 -s
warning: this system does not seem to support IPv6 - trying IPv4
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.122.1, port 38372
[  5] local 192.168.122.15 port 5201 connected to 192.168.122.1 port 38386
iperf3: getsockopt - Invalid argument
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   585 MBytes  4.91 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   1.00-2.00   sec   661 MBytes  5.55 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   2.00-3.00   sec   652 MBytes  5.47 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   3.00-4.00   sec   661 MBytes  5.54 Gbits/sec                  
iperf3: getsockopt - Invalid argument
[  5]   4.00-5.00   sec   652 MBytes  5.47 Gbits/sec                  
iperf3: getsockopt - Invalid argument
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-5.00   sec  3.14 GBytes  5.38 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
iperf3: interrupt - the server has terminated
program exited with status 1
[I/193 dhcp]: Unicasting DHCPRELEASE message with xid: [1722528183] from client: 192.168.122.15 to server: 192.168.122.1
VFS: unmounting /dev
VFS: unmounting /proc
VFS: unmounting /
Powering off.

Comparing your output with mine, I see these in mine:

VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys
BSD shrinker: event handler list found: 0x600000f45800

and yours:

VFS: unmounting /dev
zfs: driver has been initialized!
VFS: mounting zfs at /zfs
zfs: mounting osv/zfs from device /dev/vblk0.1
BSD shrinker: event handler list found: 0x600000fed900

so these 3 lines are missing in yours:

VFS: mounting devfs at /dev
VFS: mounting procfs at /proc
VFS: mounting sysfs at /sys

This suggests these 3 filesystems, including /dev/urandom, were not mounted. I wonder why?

togoetha commented 1 month ago

Odd indeed. I checked with run.py to see if any qemu parameters were missing, but that doesn't seem to be it. Next I reduced the build pipeline to this:

- git clone "https://github.com/cloudius-systems/capstan.git" "capstan-src" && cd "capstan-src"
- GOOS=linux GOARCH=amd64 go build -a -ldflags "-X main.VERSION='v1.0.0' -w -s" -tags netgo -v -o "dist/amd64/capstan" "github.com/cloudius-systems/capstan"
- cp "dist/amd64/capstan" "/usr/local/bin/capstan" && chmod +x "/usr/local/bin/capstan"
- cd "${CI_PROJECT_DIR}/iperf"
- git clone --recurse-submodules "https://github.com/cloudius-systems/osv.git" "osv-src" && cd osv-src
- ./scripts/build-capstan-mpm-packages kernel
- ./scripts/build-capstan-mpm-packages iperf3
- cd "${CI_PROJECT_DIR}/iperf"
- capstan package list
- cp -r "${CI_PROJECT_DIR}/common/init" "${CI_PROJECT_DIR}/iperf/capstan/."
- cp -r "${CI_PROJECT_DIR}/common/run" "${CI_PROJECT_DIR}/iperf/capstan/."
- cp "${CI_PROJECT_DIR}/common/kubernetes.so" "${CI_PROJECT_DIR}/iperf/capstan/."
- cd ${CI_PROJECT_DIR}/iperf/capstan && capstan package compose --verbose iperf

Which still gives the same result. The init/kubernetes part is just a script that checks a specific directory for config files, it runs after boot anyway. Capstan package is minimal:

name: iperf
title: iperf3 flint image
author: Tom Goethals
created: "2024-09-13T18:29:52+02:00"
require:
- osv.iperf3

For now it works with a fork of iperf3 that uses /dev/random instead of urandom, so I'll figure it out as needed.

togoetha commented 1 month ago

Could you post the two commands used to build and run the iperf image? I tried from scratch on my desktop, but for the life of me I can't get the extra mounts in there.