Closed sokool101 closed 7 years ago
pr1ntf
commented
7 years ago Thanks for the detailed bug report!
Can I get just one more thing? Can I get the output of zfs list | grep iohyve?
Also, have you tried this with a FreeBSD guest by chance? If not that is fine. I'm pulling down an Ubuntu ISO right now to give it a go and see if I can replicate.
sokool101
commented
7 years ago I have not tried any other guest other than Ubuntu 14.04.5
zfs list | grep iohyve
Recordings/iohyve 2.62T 15.7G 88K /mnt/iohyve/Recordings
Recordings/iohyve/ubuntu 2.62T 15.7G 88K /mnt/iohyve/Recordings/ubuntu
Recordings/iohyve/ubuntu/disk1 2.62T 2.63T 56K -
Storage/iohyve 133G 2.64T 128K /mnt/iohyve
Storage/iohyve/Firmware 117K 2.64T 117K /mnt/iohyve/Firmware
Storage/iohyve/ISO 608M 2.64T 117K /mnt/iohyve/ISO
Storage/iohyve/ISO/ubuntu-14.04.5-server-amd64.iso 608M 2.64T 608M /mnt/iohyve/ISO/ubuntu-14.04.5-server-amd64.iso
Storage/iohyve/ubuntu 132G 2.64T 117K /mnt/iohyve/ubuntu
Storage/iohyve/ubuntu/disk0 132G 2.77T 1.79G -Can you provide the output of sh -x /usr/local/sbin/iohyve start ubuntu?
sokool101
commented
7 years ago [root@SERVER] ~# sh -x /usr/local/sbin/iohyve start ubuntu
+ PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/root/bin:/usr/local/fusion-io:/bin:/usr/bin:/usr/local/bin:/sbin:/usr/sbin:/usr/local/sbin
+ grep POPCNT /var/run/dmesg.boot
+ [ -n ' Features2=0x7dbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX,F16C,RDRAND>' ]
+ grep CPU: /var/run/dmesg.boot
+ grep Intel
+ [ -n 'CPU: Intel(R) Xeon(R) CPU E3-1220 V2 @ 3.10GHz (3100.09-MHz K8-class CPU)' ]
+ grep VT-x: /var/run/dmesg.boot
+ grep UG
+ [ -z ' VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID' ]
+ [ -e ./lib/ioh-cmd ]
+ [ -e /usr/local/lib/iohyve ]
+ LIB=/usr/local/lib/iohyve
+ . /etc/rc.subr
+ : 59814
+ export RC_PID
+ [ -z '' ]
+ _rc_subr_loaded=YES
+ SYSCTL=/sbin/sysctl
+ SYSCTL_N='/sbin/sysctl -n'
+ SYSCTL_W=/sbin/sysctl
+ ID=/usr/bin/id
+ IDCMD='if [ -x /usr/bin/id ]; then /usr/bin/id -un; fi'
+ PS='/bin/ps -ww'
+ /bin/ps -ww -p 59814 -o jid=
+ JID=0
+ _rc_namevarlist='program chroot chdir env flags fib nice user group groups prepend'
+ _rc_subr_loaded=:
+ . /usr/local/lib/iohyve/ioh-cmd
+ [ ]
+ . /usr/local/lib/iohyve/ioh-setup
+ . /usr/local/lib/iohyve/ioh-zfs
+ . /usr/local/lib/iohyve/ioh-console
+ . /usr/local/lib/iohyve/ioh-iso
+ . /usr/local/lib/iohyve/ioh-firmware
+ . /usr/local/lib/iohyve/ioh-guest
+ . /usr/local/lib/iohyve/ioh-disk
+ . /usr/local/lib/iohyve/ioh-tap
+ [ -z start ]
+ __root_req_cmd start
+ return 0
+ whoami
+ [ root != root ]
+ __parse_cmd start ubuntu
+ [ 2 -gt 0 ]
+ __guest_start start ubuntu
+ local name=ubuntu
+ [ -z ubuntu ]
+ __guest_exist ubuntu
+ zfs get -H -s local,received -o value iohyve:name
+ grep '^ubuntu$'
+ local flag=
+ local pci=
+ local runmode=1
+ zfs get -H -s local,received -o name,value -t filesystem iohyve:name
+ printf '\t'
+ grep ' ubuntu$'
+ cut -f1
+ local dataset=Storage/iohyve/ubuntu
+ zfs get -H -o value iohyve:loader Storage/iohyve/ubuntu
+ local loader=grub-bhyve
+ zfs get -H -o value iohyve:template Storage/iohyve/ubuntu
+ local template=NO
+ [ NO = YES ]
+ [ grub-bhyve = uefi ]
+ zfs get -H -o value mountpoint Storage/iohyve/ubuntu
+ local mountpoint=/mnt/iohyve/ubuntu
+ [ -d /mnt/iohyve/ubuntu ]
+ pgrep -fx 'bhyve: ioh-ubuntu'
+ local running=
+ [ -z ]
+ runmode=1
+ echo 'Starting ubuntu... (Takes 15 seconds for FreeBSD guests)'
Starting ubuntu... (Takes 15 seconds for FreeBSD guests)
+ __guest_prepare ubuntu
+ local name=ubuntu
+ zfs get -H -s local,received -o name,value -t filesystem iohyve:name
+ printf '\t'
+ grep ' ubuntu$'
+ cut -f1
+ local dataset=Storage/iohyve/ubuntu
+ __zfs_get_pcidev_conf Storage/iohyve/ubuntu
+ local pool=Storage/iohyve/ubuntu
+ local 'oldifs=
'
+ IFS='
'
+ zfs get -H -o property,value all Storage/iohyve/ubuntu
+ grep iohyve:pcidev:
+ sort
+ echo 'iohyve:pcidev:1 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1'
+ cut -f2-
+ IFS='
'
+ local pci=ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1
+ zfs get -H -o value iohyve:tap Storage/iohyve/ubuntu
+ local listtap=tap0
+ echo tap0
+ sed -n 1p
+ tr , '\n'
+ [ tap0 ]
+ [ tap0 != - ]
+ ifconfig -l
+ tr ' ' '\n'
+ grep -F -w tap0
+ local tapif=tap0
+ [ -z tap0 ]
+ zfs get -H -o value iohyve:mac_tap0 Storage/iohyve/ubuntu
+ local mac=-
+ [ - = - ]
+ pci='ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ pci='ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ pci='hostbridge lpc ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ echo hostbridge lpc ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0
+ pci='hostbridge lpc ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ __guest_boot ubuntu 1 'hostbridge lpc ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ local name=ubuntu
+ [ -z ubuntu ]
+ __guest_exist ubuntu
+ zfs get -H -s local,received -o value iohyve:name
+ grep '^ubuntu$'
+ local runmode=1
+ local 'pci=hostbridge lpc ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ zfs get -H -s local,received -o name,value -t filesystem iohyve:name
+ printf '\t'
+ grep ' ubuntu$'
+ cut -f1
+ local dataset=Storage/iohyve/ubuntu
+ zfs get -H -o value iohyve:ram Storage/iohyve/ubuntu
+ local ram=16G
+ zfs get -H -o value iohyve:con Storage/iohyve/ubuntu
+ local con=nmdm0
+ zfs get -H -o value iohyve:cpu Storage/iohyve/ubuntu
+ local cpu=4
+ zfs get -H -o value iohyve:persist Storage/iohyve/ubuntu
+ local persist=1
+ zfs get -H -o value iohyve:bargs Storage/iohyve/ubuntu
+ local bargexist=-A_-H_-P
+ echo -A_-H_-P
+ sed -e 's/_/ /g'
+ local 'bargs=-A -H -P'
+ zfs set iohyve:install=no Storage/iohyve/ubuntu
+ __guest_get_bhyve_cmd 'hostbridge lpc ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ local 'devices=hostbridge lpc ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 virtio-net,tap0'
+ local pci_slot_count=0
+ echo '-s 0,hostbridge'
+ pci_slot_count=1
+ echo '-s 1,lpc'
+ pci_slot_count=2
+ echo '-s 2,ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0'
+ pci_slot_count=3
+ echo '-s 3,ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1'
+ pci_slot_count=4
+ echo '-s 4,virtio-net,tap0'
+ pci_slot_count=5
+ local 'pci_args=-s 0,hostbridge
-s 1,lpc
-s 2,ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0
-s 3,ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1
-s 4,virtio-net,tap0'
+ local runstate=1
+ exit
+ [ 1 = 1 ]
+ __guest_load ubuntu /dev/zvol/Storage/iohyve/ubuntu/disk0
+ local name=ubuntu
+ local media=/dev/zvol/Storage/iohyve/ubuntu/disk0
+ [ -z /dev/zvol/Storage/iohyve/ubuntu/disk0 ]
+ __guest_exist ubuntu
+ zfs get -H -s local,received -o value iohyve:name
[root@SERVER] ~# + grep '^ubuntu$'
+ zfs get -H -t volume -o name,value iohyve:name
+ printf 'disk0\t'
+ grep 'disk0 ubuntu$'
+ cut -f1
+ local disk=Storage/iohyve/ubuntu/disk0
+ zfs get -H -o name,value -s local,received -t filesystem iohyve:name
+ printf '\t'
+ grep ' ubuntu$'
+ cut -f1
+ local dataset=Storage/iohyve/ubuntu
+ zfs get -H -o value iohyve:ram Storage/iohyve/ubuntu
+ local ram=16G
+ zfs get -H -o value iohyve:con Storage/iohyve/ubuntu
+ local con=nmdm0
+ zfs get -H -o value iohyve:loader Storage/iohyve/ubuntu
+ local loader=grub-bhyve
+ zfs get -H -o value iohyve:install Storage/iohyve/ubuntu
+ local install=no
+ zfs get -H -o value iohyve:os Storage/iohyve/ubuntu
+ local os=debian
+ zfs get -H -o value iohyve:autogrub Storage/iohyve/ubuntu
+ local autogrub=-
+ zfs get -H -o value iohyve:bargs Storage/iohyve/ubuntu
+ local bargexist=-A_-H_-P
+ echo -A_-H_-P
+ sed -e 's/_/ /g'
+ local 'bargs=-A -H -P'
+ local test_for_wire_memory=-S
+ local wire_memory=
+ [ grub-bhyve = grub-bhyve ]
+ echo 'GRUB Process does not run in background....'
GRUB Process does not run in background....
+ echo 'If your terminal appears to be hanging, check iohyve console ubuntu in second terminal to complete GRUB process...'
If your terminal appears to be hanging, check iohyve console ubuntu in second terminal to complete GRUB process...
+ [ no = yes ]
+ [ no = no ]
+ [ debian = openbsd60 ]
+ [ debian = openbsd59 ]
+ [ debian = openbsd58 ]
+ [ debian = openbsd57 ]
+ [ debian = netbsd ]
+ [ debian = debian ]
+ printf '\(hd0\)\ /dev/zvol/Storage/iohyve/ubuntu/disk0\n'
+ printf '\(cd0\)\ /dev/zvol/Storage/iohyve/ubuntu/disk0\n'
+ grub-bhyve -m /iohyve/ubuntu/device.map -r hd0,msdos1 -c /dev/nmdm0A -M 16G ioh-ubuntu
+ local vmpid=59894
+ wait 59894
+ bhyve -c 4 -A -H -P -m 16G -s 0,hostbridge -s 1,lpc -s 2,ahci-hd,/dev/zvol/Storage/iohyve/ubuntu/disk0 -s 3,ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1 -s 4,virtio-net,tap0 -lcom1,/dev/nmdm0A ioh-ubuntu
rdmsr to register 0x34 on vcpu 1
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88The network interface slot is being created (-s 4,virtio-net,tap0), is it up and open on your host? It should be in ifconfig, and make sure sysctl net.link.tap.up_on_open is on, with both the if_bridge and if_tap kernel modules loaded.
sokool101
commented
7 years ago sysctl net.link.tap.up_on_open Comes back with a value of 1
How do I check the if_bridge and if_tap?
I have spun this up 3 different times now. Each time it fails after I run the add command. I must wipe all firmware and start over to get it to work. Here is my workflow. Maybe I'm in the wrong. I left the console open in another window the whole time and verified networking everytime you see me start, stop then destroy it. The final start is where I had no networking.
[root@SERVER] ~# iohyve setup pool=Storage kmod=1 net=lagg0
Setting up iohyve pool...
On FreeNAS installation.
Checking for symbolic link to /iohyve from /mnt/iohyve...
Symbolic link to /iohyve already exists.
Loading kernel modules...
bridge0 is already enabled on this machine...
sysctl already setup properly as well...
[root@SERVER] ~# iohyve fetchiso http://releases.ubuntu.com/trusty/ubuntu-14.04.5-server-amd64.iso
Fetching http://releases.ubuntu.com/trusty/ubuntu-14.04.5-server-amd64.iso...
/iohyve/ISO/ubuntu-14.04.5-server-amd64.iso/ub100% of 619 MB 5777 kBps 01m49s
[root@SERVER] ~# iohyve create ubuntu 128G
Creating ubuntu...
[root@SERVER] ~# iohyve set ubuntu loader=grub-bhyve ram=16G cpu=4 os=debian
Setting ubuntu loader=grub-bhyve...
Setting ubuntu ram=16G...
Setting ubuntu cpu=4...
Setting ubuntu os=debian...
[root@SERVER] ~# iohyve isolist
Listing ISO's...
ubuntu-14.04.5-server-amd64.iso
[root@SERVER] ~# iohyve install ubuntu ubuntu-14.04.5-server-amd64.iso
Installing ubuntu...
GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ubuntu in second terminal to complete GRUB process...
[root@SERVER] ~# rdmsr to register 0x34 on vcpu 0
[root@SERVER] ~# Unhandled ps2 keyboard command 0xf6
[root@SERVER] ~# iohyve start ubuntu
Starting ubuntu... (Takes 15 seconds for FreeBSD guests)
[root@SERVER] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ubuntu in second terminal to complete GRUB process...
rdmsr to register 0x34 on vcpu 3
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88
[root@SERVER] ~# iohyve stop ubuntu
Stopping ubuntu...
[root@SERVER] ~# Unhandled ps2 keyboard command 0xf6
[root@SERVER] ~# iohyve destroy ubuntu
Destroying ubuntu...
VM:ioh-ubuntu is not created.
VM:ioh-ubuntu is not created.
[root@SERVER] ~# iohyve start ubuntu
Starting ubuntu... (Takes 15 seconds for FreeBSD guests)
[root@SERVER] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ubuntu in second terminal to complete GRUB process...
rdmsr to register 0x34 on vcpu 0
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88
[root@SERVER] ~# iohyve stop ubuntu
Stopping ubuntu...
[root@SERVER] ~# Unhandled ps2 keyboard command 0xf6
iohyve destroy ubuntu
Destroying ubuntu...
VM:ioh-ubuntu is not created.
VM:ioh-ubuntu is not created.
[root@SERVER] ~# iohyve setup pool=Recordings kmod=1 net=lagg0
Setting up iohyve pool...
Secondary pool set up...
On FreeNAS installation.
Checking for symbolic link to /iohyve from /mnt/iohyve...
Symbolic link to /iohyve already exists.
Loading kernel modules...
bridge0 is already enabled on this machine...
sysctl already setup properly as well...
[root@SERVER] ~# iohyve start ubuntu
Starting ubuntu... (Takes 15 seconds for FreeBSD guests)
[root@SERVER] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ubuntu in second terminal to complete GRUB process...
rdmsr to register 0x34 on vcpu 0
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88
[root@SERVER] ~# iohyve stop ubuntu
Stopping ubuntu...
[root@SERVER] ~# Unhandled ps2 keyboard command 0xf6
iohyve destroy ubuntu
Destroying ubuntu...
VM:ioh-ubuntu is not created.
VM:ioh-ubuntu is not created.
[root@SERVER] ~# iohyve add ubuntu 2600G Recordings
Creating new zvol for ubuntu...
[root@SERVER] ~# iohyve start ubuntu
Starting ubuntu... (Takes 15 seconds for FreeBSD guests)
[root@SERVER] ~# GRUB Process does not run in background....
If your terminal appears to be hanging, check iohyve console ubuntu in second terminal to complete GRUB process...
rdmsr to register 0x34 on vcpu 2
Unhandled ps2 mouse command 0xe1
Unhandled ps2 mouse command 0x0a
Unhandled ps2 mouse command 0x01
Unhandled ps2 mouse command 0x41
Unhandled ps2 mouse command 0x88I can't seem to get this to happen. I'm going to try a few more things. The output from earlier helps out a lot.
You most likely have the kernel modules loaded if the sysctl is in place. You can list kernel modules by running kldstat. Without them, you would have no networking in the guest at all, not just after adding a disk.
sokool101
commented
7 years ago For what it's worth the newly added disk works just fine in the guest.
pr1ntf
commented
7 years ago I can reproduce this now. Stay tuned.
pr1ntf
commented
7 years ago [Note to self, unrelated to issue]:
pr1ntf
commented
7 years ago This is iohyve's fault, but in a rather unique way.
I bet before you add the disk, your network interface on Ubuntu is enp0s3, and then your interface is enp0s4 after you add the new disk. The problem here is that your network settings are from enp0s3 when you were installing the guest. So, of course, your /etc/network/interfaces file is set up for enp0s3 not enp0s4.
I know it may be a pain to do this every time you create a guest in this fashion, but on your guest, find your network interface in ifconfig -a then edit your /etc/network/interfaces file to reflect the current slot it's in. Simply issue sudo reboot and your guest should come back up with proper connectivity.
pr1ntf
commented
7 years ago I'm going to close this ticket out. If you still have issues after fixing your /etc/network/interfaces file, make a comment here and I'll reopen the issue.
It's been a fun two weeks!
sokool101
commented
7 years ago Yep, that fixed it. Awesome!
My only problem now is set commands do not change the properties as desired. Now that it is working I would like to set boot=1. Strangely using get boot I get a return of 1 but using getall comes back 0. I have rebooted the host and it does not start the guest automatically. Is there a config file I can manually edit to fix this issue?
[root@SERVER] ~# iohyve set ubuntu boot=1
Setting ubuntu boot=1...
[root@SERVER] ~# iohyve get ubuntu boot
Getting ubuntu prop boot...
1
[root@SERVER] ~# iohyve getall ubuntu
Getting ubuntu iohyve properties...
bargs -A_-H_-P
boot 0
con nmdm0
cpu 4
description Mon Jan 30 22:22:47 EST 2017
install no
loader grub-bhyve
name ubuntu
os debian
pcidev:1 ahci-hd,/dev/zvol/Recordings/iohyve/ubuntu/disk1
persist 1
ram 16G
size 128G
tap tap0
template NO
vnc NO
vnc_h 600
vnc_ip 127.0.0.1
vnc_tablet NO
vnc_w 800
vnc_wait NOAll the properties are stored in ZFS user properties, not config files.
What's happening is this line is getting confused picking up the wrong pool.
I have opened a separate issue for this.
idahoakl
commented
6 years ago Posting this here for anyone searching from Google and finds this issue.
I was testing this on Ubuntu Zesty (17.10) using the virtio network driver. It appears that when adding a disk the pci device id for the network adapter is changed (the disks are indexed first and then network adapters). Thus network interface enp0s3 becomes enp0s4.
The version of Ubuntu I'm using uses systemd for networking. During initial installation of the system a systemd unit for configuring the network interface is placed in /run/systemd/network. This file (called 10-netplan-enp0s3.network in my case) is hard coded to the full interface name and can't tolerate the change in pci device id when a new disk is added. To solve this I copied the file to /etc/systemd/network and edited the Match line with the glob pattern enp0s* to tolerate adding additional disks in the future. Find more about this systemd unit at http://manpages.ubuntu.com/manpages/zesty/man5/systemd.network.5.html
FreeNAS 9.10.2 U1
After installing an Ubuntu guest, everything works as expected. Boots fine and can access via SSH ect. I then destroy the guest and run a setup command to create a secondary pool. Then I run an add command to add disk using secondary pool to guest. Start guest and I no longer have networking. Despite everything in ifconfig showing proper with all members present in bridge0, just no IP is set. I am using a lagg interface if that makes any difference. I can format and mount the new disk fine as expected using the internal console.
I have tried a few different times in different orders of operation with odd results. For example, if I setup a secondary pool before creating and installing the guest, set commands will not stick. So I can't change the loader or other parameters to even install the guest at all.
Here is some info I'm sure you will want as well as the shortened version of the output of ifconfig with the pertinent interfaces shown.