Closed xuchenhao001 closed 6 years ago
Now that I know more about the kernel
from hyperhq/hyperstart
, I built my s390x kernel from torvalds/linux in s390x/ubuntu
docker container:
make defconfig
make -j 8
and then i integrated kernel
, kernel_config
and initrd.img
modules.tar
with hyperhq/hyperstart
.
After that, i got a hyper-initrd.img
.
Then, i tried to run ubuntu image by: hyperctl run -t s390x/ubuntu
but it didn't work:
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Firewalld running: false"
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=warning msg="Your kernel does not support swap memory limit."
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Loading containers: start."
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Loading containers: done."
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.258757 9436 json.go:370] read tty data failed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259145 9436 json.go:427] SB[vm-NwUOEtsyii] tty socket closed, quit the reading goroutine: read unix @->/var/run/hyper/vm-NwUOEtsyii/tty.sock: read: connection reset by peer
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259182 9436 json.go:570] SB[vm-NwUOEtsyii] get hyperstart API version error: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259217 9436 vm_states.go:288] SB[vm-NwUOEtsyii] hyperstart failed: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259225 9436 vm_states.go:246] SB[vm-NwUOEtsyii] Shutting down because of an exception: %!(EXTRA string=connection to vm broken)
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259268 9436 vm_states.go:226] SB[vm-NwUOEtsyii] Start POD failed: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259288 9436 run.go:38] ubuntu-0536492129: failed to add pod: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259302 9436 server.go:170] Handler for POST /v0.8.1/pod/create returned error: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259515 9436 json.go:137] read init data failed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259530 9436 json.go:171] SB[vm-NwUOEtsyii] error when readVmMessage() for ready message: read unix @->/var/run/hyper/vm-NwUOEtsyii/hyper.sock: read: connection reset by peer
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259561 9436 qmp_handler.go:183] get qmp welcome failed: read unix @->/var/run/hyper/vm-NwUOEtsyii/qmp.sock: read: connection reset by peer
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259601 9436 qmp_handler.go:363] QMP initialize failed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.261501 9436 qemu_process.go:157] exit status 1
Looks the qemu didn't start correctly. Could you run hyperd with -v=4
and see what's in the logs.
I am sorry I couldn't reproduce it because I don't have a s390x machine
@gnawux Thanks for your attention! Here is the logs after set -v=4
:
$ hyperctl run -t s390x/ubuntu
...
-- The start-up result is done.
Nov 01 12:11:40 marist02 hyperd[9839]: time="2017-11-01T12:11:40Z" level=debug msg="Using default logging driver none"
Nov 01 12:11:40 marist02 hyperd[9839]: time="2017-11-01T12:11:40Z" level=debug msg="[graphdriver] trying provided driver "overlay""
Nov 01 12:11:40 marist02 hyperd[9839]: time="2017-11-01T12:11:40Z" level=debug msg="Using graph driver overlay"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Option DefaultDriver: bridge"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Option DefaultNetwork: bridge"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=info msg="Firewalld running: false"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Registering ipam driver: "default""
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=warning msg="Your kernel does not support swap memory limit."
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Cleaning up old shm/mqueue mounts: start."
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Cleaning up old shm/mqueue mounts: done."
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699033 9839 qmp_handler.go:183] get qmp welcome failed: read unix @->/var/run/hyper/vm-oOMYrhTfNz/qmp.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700238 9839 qmp_handler.go:363] QMP initialize failed
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699061 9839 json.go:137] read init data failed
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699110 9839 json.go:370] read tty data failed
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699797 9839 qemu_process.go:157] exit status 1
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700669 9839 vm_states.go:288] SB[vm-oOMYrhTfNz] read unix @->/var/run/hyper/vm-oOMYrhTfNz/qmp.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700759 9839 vm_states.go:246] SB[vm-oOMYrhTfNz] Shutting down because of an exception: %!(EXTRA string=connection to vm broken)
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700452 9839 json.go:171] SB[vm-oOMYrhTfNz] error when readVmMessage() for ready message: read unix @->/var/run/hyper/vm-oOMYrhTfNz/hyper.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700562 9839 json.go:427] SB[vm-oOMYrhTfNz] tty socket closed, quit the reading goroutine: read unix @->/var/run/hyper/vm-oOMYrhTfNz/tty.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: 2017/11/01 12:13:45 http: panic serving @: send on closed channel
Nov 01 12:13:45 marist02 hyperd[9839]: goroutine 51 [running]:
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.(*conn).serve.func1(0xc4205b80a0)
Nov 01 12:13:45 marist02 hyperd[9839]: /usr/local/go/src/net/http/server.go:1697 +0xc2
Nov 01 12:13:45 marist02 hyperd[9839]: panic(0x80ee2b40, 0x811cec40)
Nov 01 12:13:45 marist02 hyperd[9839]: /usr/local/go/src/runtime/panic.go:491 +0x26e
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor.(*VmContext).reportVmFault(...)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor/report.go:66
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor.(*VmContext).startPod(0xc42066c6e0, 0xc420191260, 0x805c6652)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor/vm_states.go:225 +0x2fa
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor.(*Vm).InitSandbox(0xc420064320, 0xc420191260, 0xc4200ba8f0, 0x80)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor/vm.go:231 +0x4e
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon/pod.(*XPod).createSandbox(0xc4200ba8f0, 0xc4211232c0, 0xc4210036b0, 0xc4200ba8f0)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/daemon/pod/provision.go:272 +0x33a
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon/pod.CreateXPod(0xc420190ae0, 0xc4211232c0, 0x0, 0x0, 0x0)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/daemon/pod/provision.go:36 +0x100
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon.(*Daemon).CreatePod(0xc420228070, 0x0, 0x0, 0xc4211232c0, 0xc4211232c0, 0x0, 0x0)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/daemon/run.go:36 +0x27e
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon.(*Daemon).CmdCreatePod(0xc420228070, 0xc4205f00b0, 0xac, 0x200, 0xc4205f00b0, 0xac)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/daemon/server.go:260 +0xc8
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server/router/pod.(*podRouter).postPodCreate(0xc42108f890, 0x3ffa085dfc0, 0xc4205b22d0, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x818efaa0, 0xc4205b22d0)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/server/router/pod/pod_routes.go:70 +0x20a
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server/router/pod.(*podRouter).(github.com/hyperhq/hyperd/server/router/pod.postPodCreate)-fm(0x3ffa085dfc0, 0xc4205b22d0, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x3ffa085dfc0, 0xc4205b22d0)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/server/router/pod/pod.go:26 +0x5a
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.versionMiddleware.func1(0x3ffa085ac08, 0xc420012018, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x810c71ac, 0xe)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/server/middleware.go:156 +0x166
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*Server).corsMiddleware.func1(0x3ffa085ac08, 0xc420012018, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x80e4e9e0, 0x811c86c8)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/server/middleware.go:134 +0xa2
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*Server).userAgentMiddleware.func1(0x3ffa085ac08, 0xc420012018, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x8103f7a0, 0x100000080fca580)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/server/middleware.go:117 +0xd4
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*Server).makeHTTPHandler.func1(0x818ee4a0, 0xc42114a000, 0xc42112c600)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/server/server.go:169 +0x1cc
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.HandlerFunc.ServeHTTP(0xc421127560, 0x818ee4a0, 0xc42114a000, 0xc42112c600)
Nov 01 12:13:45 marist02 hyperd[9839]: /usr/local/go/src/net/http/server.go:1918 +0x42
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc42101e050, 0x818ee4a0, 0xc42114a000, 0xc42112c600)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/gorilla/mux/mux.go:114 +0xde
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*routerSwapper).ServeHTTP(0xc42114dc50, 0x818ee4a0, 0xc42114a000, 0xc42007c300)
Nov 01 12:13:45 marist02 hyperd[9839]: /root/go/src/github.com/hyperhq/hyperd/server/router_swapper.go:29 +0x78
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.serverHandler.ServeHTTP(0xc4210e9380, 0x818ee4a0, 0xc42114a000, 0xc42007c300)
Nov 01 12:13:45 marist02 hyperd[9839]: /usr/local/go/src/net/http/server.go:2619 +0x96
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.(*conn).serve(0xc4205b80a0, 0x818ef9e0, 0xc42004e440)
Nov 01 12:13:45 marist02 hyperd[9839]: /usr/local/go/src/net/http/server.go:1801 +0x6f8
Nov 01 12:13:45 marist02 hyperd[9839]: created by net/http.(*Server).Serve
Nov 01 12:13:45 marist02 hyperd[9839]: /usr/local/go/src/net/http/server.go:2720 +0x264
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.701139 9839 json.go:570] SB[vm-oOMYrhTfNz] get hyperstart API version error: hyperstart closed
At /var/log/hyper/qemu/vm-oOMYrhTfN.log
I found logs about vm:
qemu-system-s390x: -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir: 'virtio-9p-ccw' is not a valid device model name
looks like your qemu did not build with VirtFs
enabled?
I rebuild qemu with:
git clone git://git.qemu.org/qemu.git
cd qemu
git checkout v2.10.1
git submodule init
git submodule update --recursive
./configure --prefix=/usr --enable-virtfs
make
make install
but it seems not working with the same error:
-- The start-up result is done.
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Using default logging driver none"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="[graphdriver] trying provided driver "overlay""
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Using graph driver overlay"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Option DefaultDriver: bridge"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Option DefaultNetwork: bridge"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=info msg="Firewalld running: false"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Registering ipam driver: "default""
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=warning msg="Your kernel does not support swap memory limit."
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Cleaning up old shm/mqueue mounts: start."
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Cleaning up old shm/mqueue mounts: done."
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.179870 27626 hypervisor.go:37] SB[vm-WLrdFtGHVq] watch hyperstart timeout
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.180992 27626 json.go:570] SB[vm-WLrdFtGHVq] get hyperstart API version error: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181026 27626 vm_states.go:288] SB[vm-WLrdFtGHVq] watch hyperstart timeout
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181035 27626 vm_states.go:246] SB[vm-WLrdFtGHVq] Shutting down because of an exception: %!(EXTRA string=connection to vm broken)
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181133 27626 json.go:370] read tty data failed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181148 27626 json.go:427] SB[vm-WLrdFtGHVq] tty socket closed, quit the reading goroutine: read unix @->/var/run/hyper/vm-WLrdFtGHVq/tty.sock: use of closed network connection
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181160 27626 vm_states.go:226] SB[vm-WLrdFtGHVq] Start POD failed: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181178 27626 run.go:38] ubuntu-6203458613: failed to add pod: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181194 27626 server.go:170] Handler for POST /v0.8.1/pod/create returned error: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.185606 27626 json.go:137] read init data failed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.185730 27626 json.go:171] SB[vm-WLrdFtGHVq] error when readVmMessage() for ready message: EOF
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.185891 27626 qmp_handler.go:141] QMP exit as got error: EOF
:disappointed:
could you find the same message
qemu-system-s390x: -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir: 'virtio-9p-ccw' is not a valid device model name
No, I found there is a new log file at /var/log/hyper/qemu
, but it has no content:
$ root@marist02:/var/log/hyper/qemu# ls -ltr
total 4
-rw-r--r-- 1 root root 126 Nov 2 03:18 vm-doHDBOZdv.log
-rw-r--r-- 1 root root 0 Nov 2 09:38 vm-WLrdFtGHV.log
(the first one is created before rebuild my qemu with --enable-virtfs
)
could you find the qemu command line and run it directly? I think there are still some qemu issue, but could not get any useful info here. It's too hard to debug without an environment to reproduce it...
@gnawux
After a long period of working on qemu-system-s390x
, now i have something found:
When I use libvirt to start a guest vm (Ubuntu) on s390x, i got qemu running parameters like this:
/usr/bin/qemu-system-s390x -name ubuntu16.04 -S -machine s390-ccw-virtio-xenial,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,maxcpus=2,sockets=2,cores=1,threads=1 -object iothread,id=iothread1 -uuid 4db4b6b7-145b-4942-8833-509da364e32b -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-ubuntu16.04/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /var/lib/libvirt/images/kernel.ubuntu -initrd /var/lib/libvirt/images/initrd.ubuntu -device virtio-scsi-ccw,id=scsi0,devno=fe.0.0002 -drive file=/home/ubuntu/zhengxil/xchxubj/kvm/ubuntu.qcow2,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/home/ubuntu/zhengxil/xchxubj/kvm/xenial-server-s390x.iso,format=raw,if=none,id=drive-scsi0-0-0-1,readonly=on -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:9c:58:88,devno=fe.0.0001 -chardev pty,id=charconsole0 -device sclpconsole,chardev=charconsole0,id=console0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 -msg timestamp=on
On the other hand, when I type hyperctl run -t s390x/ubuntu:16.04
, it hangs for a while. And at that time, I start an another ssh and check the qemu process with ps -ef | grep qemu
:
/usr/bin/qemu-system-s390x -machine s390-ccw-virtio,accel=kvm,usb=off -cpu host -kernel /var/lib/hyper/kernel -initrd /var/lib/hyper/hyper-initrd.img -append "console=ttyS1 panic=1 no_timer_check" -realtime mlock=off -no-user-config -nodefaults -enable-kvm -rtc base=utc,clock=vm,driftfix=slew -no-reboot -display none -boot strict=on -m 128 -smp 1 -qmp unix:/var/run/hyper/vm-ESajskTODD/qmp.sock,server,nowait -chardev socket,id=charconsole0,path=/var/run/hyper/vm-ESajskTODD/console.sock,server,nowait -device sclpconsole,chardev=charconsole0 -device virtio-serial-ccw,id=virtio-serial0 -device virtio-scsi-ccw,id=scsi0 -chardev socket,id=charch0,path=/var/run/hyper/vm-ESajskTODD/hyper.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch1,path=/var/run/hyper/vm-ESajskTODD/tty.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -fsdev local,id=virtio9p,path=/var/run/hyper/vm-ESajskTODD/share_dir,security_model=none -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir -daemonize -pidfile /var/run/hyper/vm-ESajskTODD/pidfile -D /var/log/hyper/qemu/vm-ESajskTOD.log
Roughly speaking, there are a lot of differences between this two.
However, as I know, s390x do not support tty
, but pty
instead.
So if I can configure the default parameters with hyperctl to start qemu-system-s390x
?
Here is my xml file for libvirt:
<domain type = 'kvm'>
<name>ubuntu</name>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>1048576</currentMemory>
<vcpu current='1'>2</vcpu>
<os>
<type arch = 's390x' machine = 's390-ccw-virtio'>hvm</type>
</os>
<devices>
<emulator>/usr/bin/qemu-system-s390x</emulator>
<disk type = 'file' device = 'disk'>
<source file = '/home/ubuntu/kvm/ubuntu16.04.raw'/>
<target dev='vda' bus='virtio'/>
</disk>
<interface type = 'bridge'>
<source bridge = 'virbr0'/>
</interface>
<console type='pty'>
<target type='sclp' port='0'/>
</console>
</devices>
</domain>
Thanks!
@xuchenhao001
hyper uses two unix sockets to communicate with hyperstart in vm, for x86/arm64, these two unix socket on host are represented as virtio-port in vm. the path are /var/run/hyper/vm-oOMYrhTfNz/hyper.sock and /var/run/hyper/vm-oOMYrhTfNz/tty.sock on host.
so you should launch qemu with virtserialport device. https://github.com/hyperhq/runv/blob/master/hypervisor/qemu/qemu_amd64.go#L87
@gao-feng Thanks! But as i know, there is no ttyS device on s390x machine:
$ ls /dev
autofs chsc cpu_dma_latency dasda1 disk fd hwrng lightnvm loop1 loop5 mem network_latency prandom sclp stderr tty vhost-net
block clp cuse dasda2 dm-0 full initctl log loop2 loop6 memory_bandwidth network_throughput ptmx sclp_line0 stdin ttysclp0 zero
btrfs-control console dasd_eer dasdb dm-1 fuse kmsg loop-control loop3 loop7 mqueue null pts shm stdout urandom
char core dasda dasdb1 ecryptfs hugepages kvm loop0 loop4 mapper net port random snapshot sys-vg vfio
So which one should I use?
The console device is ttysclp0. Here is an old example of qemu parameters from of a working docker/ hyperstart/runv setup that includes the two unix sockets:
-machine s390-ccw-virtio,accel=kvm,usb=off \
-cpu host \
-kernel /var/lib/runv/kernel \
-initrd /var/lib/runv/initrd \
-append "console=ttyS1 panic=1 no_timer_check" \
-realtime mlock=off \
-no-user-config \
-nodefaults \
-enable-kvm \
-no-reboot \
-display none \
-boot strict=on \
-m 512 \
-smp 4 \
-qmp unix:/var/run/hyper/vm-rnqlfpzLLe/qmp.sock,server,nowait \
-device virtio-scsi-ccw,id=scsi0 \
-chardev socket,id=charconsole0,path=/var/run/hyper/vm-rnqlfpzLLe/console.sock,server,nowait \
-device sclpconsole,chardev=charconsole0 \
-device virtio-serial-ccw,id=virtio-serial0 \
-chardev socket,id=charch0,path=/var/run/hyper/vm-rnqlfpzLLe/hyper.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
-chardev socket,id=charch1,path=/var/run/hyper/vm-rnqlfpzLLe/tty.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
-fsdev local,id=virtio9p,path=/var/run/hyper/vm-rnqlfpzLLe/share_dir,security_model=none \
-device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir \
-daemonize \
-pidfile /var/run/hyper/vm-rnqlfpzLLe/pidfile \
-D /var/log/hyper/qemu/vm-rnqlfpzLLe.log
thank you @pmorjan nice to see you again here.
@pmorjan Thank you for your help!
I have read your code about this old example. It's working for runv to start or remove an vm.
However, when I start a vm from command hyperctl run -t s390x/ubuntu:16.04
, it seems that hyperd cannot read any data from qemu. Just like the error log before:
E1102 09:39:02.181133 27626 json.go:370] read tty data failed
I think if there is something wrong with these lines in your code:
"-append", "\"console=ttyS1 panic=1 no_timer_check\"",
"-chardev", fmt.Sprintf("socket,id=charch0,path=%s,server,nowait", ctx.HyperSockName),
"-device", "virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0",
"-chardev", fmt.Sprintf("socket,id=charch1,path=%s,server,nowait", ctx.TtySockName),
"-device", "virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1",
Because there is no ttyS device on s390x at all.
@xuchenhao001 do you have this parameter -device sclpconsole,chardev=charconsole0
provided by @pmorjan
@gnawux Of course I do. here is the qemu process detail:
/usr/bin/qemu-system-s390x \
-machine s390-ccw-virtio,accel=kvm,usb=off \
-cpu host \
-kernel /var/lib/hyper/kernel \
-initrd /var/lib/hyper/hyper-initrd.img \
-append "console=ttyS1 panic=1 no_timer_check" \
-realtime mlock=off \
-no-user-config \
-nodefaults \
-enable-kvm \
-rtc base=utc,clock=vm,driftfix=slew \
-no-reboot \
-display none \
-boot strict=on \
-m 128 \
-smp 1 \
-qmp unix:/var/run/hyper/vm-ESajskTODD/qmp.sock,server,nowait \
-chardev socket,id=charconsole0,path=/var/run/hyper/vm-ESajskTODD/console.sock,server,nowait \
-device sclpconsole,chardev=charconsole0 \
-device virtio-serial-ccw,id=virtio-serial0 \
-device virtio-scsi-ccw,id=scsi0 \
-chardev socket,id=charch0,path=/var/run/hyper/vm-ESajskTODD/hyper.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
-chardev socket,id=charch1,path=/var/run/hyper/vm-ESajskTODD/tty.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
-fsdev local,id=virtio9p,path=/var/run/hyper/vm-ESajskTODD/share_dir,security_model=none \
-device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir \
-daemonize \
-pidfile /var/run/hyper/vm-ESajskTODD/pidfile \
-D /var/log/hyper/qemu/vm-ESajskTOD.log
I was just query the validity of some parameters:
-append "console=ttyS1 panic=1 no_timer_check" \
# and
-chardev socket,id=charch0,path=/var/run/hyper/vm-ESajskTODD/hyper.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
-chardev socket,id=charch1,path=/var/run/hyper/vm-ESajskTODD/tty.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
@xuchenhao001 It restored my old build environment and can confirm that the qemu config works as expected. I successfully ran a container via hyperctl and hyperd(v0.8.1) on s390x. This is the qemu command line of a running hyperd container:
/usr/bin/qemu-system-s390x \
-machine s390-ccw-virtio,accel=kvm,usb=off \
-cpu host \
-kernel /data/runv/dist/kernel \
-initrd /data/runv/dist/initrd \
-append "console=ttyS1 panic=1 no_timer_check" \
-realtime mlock=off \
-no-user-config \
-nodefaults \
-enable-kvm \
-rtc base=utc,clock=vm,driftfix=slew \
-no-reboot \
-display none \
-boot strict=on \
-m 128 \
-smp 1 \
-qmp unix:/var/run/hyper/vm-FDpmPVFtfA/qmp.sock,server,nowait \
-chardev socket,id=charconsole0,path=/var/run/hyper/vm-FDpmPVFtfA/console.sock,server,nowait \
-device sclpconsole,chardev=charconsole0 \
-device virtio-serial-ccw,id=virtio-serial0 \
-device virtio-scsi-ccw,id=scsi0 \
-chardev socket,id=charch0,path=/var/run/hyper/vm-FDpmPVFtfA/hyper.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
-chardev socket,id=charch1,path=/var/run/hyper/vm-FDpmPVFtfA/tty.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
-fsdev local,id=virtio9p,path=/var/run/hyper/vm-FDpmPVFtfA/share_dir,security_model=none \
-device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir \
-daemonize \
-pidfile /var/run/hyper/vm-FDpmPVFtfA/pidfile \
-D /var/log/hyper/qemu/vm-FDpmPVFtf.log
I guess there is something wrong in your kernel / initrd setup.
But does it make sense not to follow upstream development of hyperd and stay on the backlevel 0.8.1 version? I did a quick look and it seems hyperd now depends on the lightweight kvmtool which is not available on s390x (as far as I know). On the other hand the overhead of Qemu on s390x is smaller compared to x86 (no BIOS) and it's well tested. Making upstream hyperd work on s390x again could be a challenge.
@pmorjan Yes, I'm on version v0.8.1. And my qemu running parameters look the same as yours.
My host is on LPAR. For kernel and initrd, I simply download an iso from ubuntu offical website, and extract boot/kernel.ubuntu
& boot/initrd.ubuntu
from iso to /var/lib/hyper/kernel
& /var/lib/hyper/hyper-initrd.img
. Is that right?
Since that I successfully run ubuntu installer on s390x with this two files, I take them for granted as kernel
& hyper-initrd.img
.
@xuchenhao001 I'm not sure if your kernel will work at all. Does it have the 9p/virtio modules builtin? If not then you have to include them into your initrd and patch hyperstart/src/init to load them. 9pfs is the filesytem type used for the rootfs.
@pmorjan Ok, thank you! I will check it up.
What's more, how did you build your kernel
& hyper-initrd.img
on s390x? If you can provide your kernel_config
file at hyperstart
project?
Thank you!
@xuchenhao001 The important config options are as follows:
CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_FSCACHE=y
CONFIG_9P_FS=y
See the attached config for vanilla 4.8.0 that worked for me: config-4.8.0-s390x.gz
$ ./hyperctl run -t alpine sh
/ # uname -a
Linux alpine-2987341563 4.8.0 #6 SMP Wed Nov 22 12:38:13 CET 2017 s390x Linux
/ # lsmod
Module Size Used by Not tainted
/ #
@pmorjan Thank you!!! It works for me!! :smile: What's more, I tried to build a linux kernel v4.12.4 based on your config, luckily, it still works! Also, thank you @gnawux for your help!
Hello, I tried to transplant hyperd to IBM Z (s390x), so that i can run kubernetes/frakti. It's ok when I build hyperd on s390x, but there are some errors when I creat a pod:
I built hyperstart with:
and hyperd with:
hyperd status:
Thanks!