search

hyperhq / hyperd

HyperContainer Daemon
http://www.hypercontainer.io
Apache License 2.0
1.98k stars 194 forks source link

Not compatible with s390x architecture #674

Closed xuchenhao001 closed 6 years ago

xuchenhao001 commented 7 years ago

Hello, I tried to transplant hyperd to IBM Z (s390x), so that i can run kubernetes/frakti. It's ok when I build hyperd on s390x, but there are some errors when I creat a pod:

$ hyperctl pull gcr.io/google_containers/kubernetes-dashboard-s390x:v1.7.1
v1.7.1: Pulling from google_containers/kubernetes-dashboard-s390x
f33e1e4438ca: Pull complete 
Digest: sha256:c29bb3d0de84c6221bc7ad9c92cdb7b0ef7d28164ceb6a13577095608ecd9e68
Status: Downloaded newer image for gcr.io/google_containers/kubernetes-dashboard-s390x:v1.7.1
sha256:c29bb3d0de84c6221bc7ad9c92cdb7b0ef7d28164ceb6a13577095608ecd9e68: Pulling from google_containers/kubernetes-dashboard-s390x
Digest: sha256:c29bb3d0de84c6221bc7ad9c92cdb7b0ef7d28164ceb6a13577095608ecd9e68
Status: Downloaded newer image for gcr.io/google_containers/kubernetes-dashboard-s390x@sha256:c29bb3d0de84c6221bc7ad9c92cdb7b0ef7d28164ceb6a13577095608ecd9e68
$ hyperctl images
REPOSITORY                                               TAG                 IMAGE ID            CREATED               VIRTUAL SIZE
gcr.io/google_containers/kubernetes-dashboard-s390x      v1.7.1              2ace3956d7fe        2017-10-02 14:11:24   129.3 MB
$ hyperctl start gcr.io/google_containers/kubernetes-dashboard-s390x:v1.7.1
hyperctl ERROR: Error from daemon's response: The pod(gcr.io/google_containers/kubernetes-dashboard-s390x:v1.7.1) can not be found, please create it first
$ hyperctl create gcr.io/google_containers/kubernetes-dashboard-s390x:v1.7.1
hyperctl ERROR: Error from daemon's response: hyperstart closed

I built hyperstart with:

$ git clone https://github.com/hyperhq/hyperstart.git hyperstart
$ cd hyperstart
$ git checkout v1.0.0
$ ./autogen.sh
$ ./configure
$ make

and hyperd with:

$ mkdir -p ${GOPATH}/src/github.com/hyperhq
$ cd ${GOPATH}/src/github.com/hyperhq
$ git clone https://github.com/hyperhq/hyperd.git hyperd
$ cd hyperd
$ git checkout v0.8.1
$ ./autogen.sh
$ ./configure
$ make

hyperd status:

$ systemctl status hyperd
● hyperd.service - hyperd
   Loaded: loaded (/lib/systemd/system/hyperd.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2017-10-31 02:06:47 UTC; 2h 26min ago
     Docs: http://docs.hypercontainer.io
 Main PID: 18445 (hyperd)
    Tasks: 12
   Memory: 183.8M
      CPU: 2h 19min 39.110s
   CGroup: /system.slice/hyperd.service
           ├─18445 /usr/bin/hyperd --log_dir=/var/log/hyper
           └─19244 /usr/bin/qemu-system-s390x -machine s390-ccw-virtio,accel=kvm,usb=off -cpu host -kernel /var/lib/hyper/kernel -initrd /var/lib/hyper/hyper-initrd.img -append "console=ttyS1 panic=1 no_timer_check" -realtime mlock=off -no-user-config -nodefaults -enable-kvm -rtc base=utc,clock=vm,driftfix=slew -no-reboot -display none -boot strict=on -m 64 -smp 1 -qmp unix:/var/run/hyper/vm-MFpZfrrvfB/qmp.sock,server,nowait -chardev socket,id=charconsole0,path=/var/run/hyper/vm-MFpZfrrvfB/console.so ck,server,nowait -device sclpconsole,chardev=charconsole0 -device virtio-serial-ccw,id=virtio-serial0 -device virtio-scsi-ccw,id=scsi0 -chardev socket,id=charch0,path=/var/run/hyper/vm-MFpZfrrvfB/hyper.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch1,path=/var/run/hyper/vm-MFpZfrrvfB/tty.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -fsdev local,id=virtio9p,path=/var/run/hyper/vm-MFpZfrrvfB/share_dir,security_model=none -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir -daemonize -pidfile /var/run/hyper/vm-MFpZfrrvfB/pidfile -D /var/log/hyper/qemu/vm-MFpZfrrvf.log
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.141016   18445 vm_states.go:288] SB[vm-mPBTQEmNPS] watch hyperstart timeout
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.141034   18445 vm_states.go:246] SB[vm-mPBTQEmNPS] Shutting down because of an exception: %!(EXTRA string=connection to vm broken)
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.141187   18445 json.go:370] read tty data failed
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.141217   18445 json.go:427] SB[vm-mPBTQEmNPS] tty socket closed, quit the reading goroutine: read unix @->/var/run/hyper/vm-mPBTQEmNPS/tty.sock: use of closed network connection
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.141237   18445 vm_states.go:226] SB[vm-mPBTQEmNPS] Start POD failed: hyperstart closed
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.141262   18445 run.go:38] k8s_POD.0_kubernetes-dashboard-987cf49dd-qvlgh_kube-system_c37d43c6-bde0-11e7-89a7-fa163ee9b03d_cfd96c23: failed to add pod: hyperstart closed
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.141271   18445 pod.go:18] CreatePod failed: hyperstart closed
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.149643   18445 qmp_handler.go:141] QMP exit as got error: EOF
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.149700   18445 json.go:137] read init data failed
Oct 31 04:32:58 marist02 hyperd[18445]: E1031 04:32:58.149957   18445 json.go:171] SB[vm-mPBTQEmNPS] error when readVmMessage() for ready message: EOF

Thanks!

xuchenhao001 commented 7 years ago

Now that I know more about the kernel from hyperhq/hyperstart, I built my s390x kernel from torvalds/linux in s390x/ubuntu docker container:

make defconfig
make -j 8

and then i integrated kernel, kernel_config and initrd.img modules.tar with hyperhq/hyperstart. After that, i got a hyper-initrd.img. Then, i tried to run ubuntu image by: hyperctl run -t s390x/ubuntu but it didn't work:

Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Firewalld running: false"
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=warning msg="Your kernel does not support swap memory limit."
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Loading containers: start."
Nov 01 09:23:55 marist02 hyperd[9436]: time="2017-11-01T09:23:55Z" level=info msg="Loading containers: done."
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.258757    9436 json.go:370] read tty data failed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259145    9436 json.go:427] SB[vm-NwUOEtsyii] tty socket closed, quit the reading goroutine: read unix @->/var/run/hyper/vm-NwUOEtsyii/tty.sock: read: connection reset by peer
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259182    9436 json.go:570] SB[vm-NwUOEtsyii] get hyperstart API version error: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259217    9436 vm_states.go:288] SB[vm-NwUOEtsyii] hyperstart failed: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259225    9436 vm_states.go:246] SB[vm-NwUOEtsyii] Shutting down because of an exception: %!(EXTRA string=connection to vm broken)
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259268    9436 vm_states.go:226] SB[vm-NwUOEtsyii] Start POD failed: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259288    9436 run.go:38] ubuntu-0536492129: failed to add pod: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259302    9436 server.go:170] Handler for POST /v0.8.1/pod/create returned error: hyperstart closed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259515    9436 json.go:137] read init data failed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259530    9436 json.go:171] SB[vm-NwUOEtsyii] error when readVmMessage() for ready message: read unix @->/var/run/hyper/vm-NwUOEtsyii/hyper.sock: read: connection reset by peer
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259561    9436 qmp_handler.go:183] get qmp welcome failed: read unix @->/var/run/hyper/vm-NwUOEtsyii/qmp.sock: read: connection reset by peer
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.259601    9436 qmp_handler.go:363] QMP initialize failed
Nov 01 09:24:29 marist02 hyperd[9436]: E1101 09:24:29.261501    9436 qemu_process.go:157] exit status 1
gnawux commented 7 years ago

Looks the qemu didn't start correctly. Could you run hyperd with -v=4 and see what's in the logs.

I am sorry I couldn't reproduce it because I don't have a s390x machine

xuchenhao001 commented 7 years ago

@gnawux Thanks for your attention! Here is the logs after set -v=4:

$ hyperctl run -t s390x/ubuntu

...

-- The start-up result is done.
Nov 01 12:11:40 marist02 hyperd[9839]: time="2017-11-01T12:11:40Z" level=debug msg="Using default logging driver none"
Nov 01 12:11:40 marist02 hyperd[9839]: time="2017-11-01T12:11:40Z" level=debug msg="[graphdriver] trying provided driver "overlay""
Nov 01 12:11:40 marist02 hyperd[9839]: time="2017-11-01T12:11:40Z" level=debug msg="Using graph driver overlay"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Option DefaultDriver: bridge"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Option DefaultNetwork: bridge"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=info msg="Firewalld running: false"
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Registering ipam driver: "default""
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=warning msg="Your kernel does not support swap memory limit."
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Cleaning up old shm/mqueue mounts: start."
Nov 01 12:11:41 marist02 hyperd[9839]: time="2017-11-01T12:11:41Z" level=debug msg="Cleaning up old shm/mqueue mounts: done."
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699033    9839 qmp_handler.go:183] get qmp welcome failed: read unix @->/var/run/hyper/vm-oOMYrhTfNz/qmp.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700238    9839 qmp_handler.go:363] QMP initialize failed
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699061    9839 json.go:137] read init data failed
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699110    9839 json.go:370] read tty data failed
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.699797    9839 qemu_process.go:157] exit status 1
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700669    9839 vm_states.go:288] SB[vm-oOMYrhTfNz] read unix @->/var/run/hyper/vm-oOMYrhTfNz/qmp.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700759    9839 vm_states.go:246] SB[vm-oOMYrhTfNz] Shutting down because of an exception: %!(EXTRA string=connection to vm broken)
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700452    9839 json.go:171] SB[vm-oOMYrhTfNz] error when readVmMessage() for ready message: read unix @->/var/run/hyper/vm-oOMYrhTfNz/hyper.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.700562    9839 json.go:427] SB[vm-oOMYrhTfNz] tty socket closed, quit the reading goroutine: read unix @->/var/run/hyper/vm-oOMYrhTfNz/tty.sock: read: connection reset by peer
Nov 01 12:13:45 marist02 hyperd[9839]: 2017/11/01 12:13:45 http: panic serving @: send on closed channel
Nov 01 12:13:45 marist02 hyperd[9839]: goroutine 51 [running]:
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.(*conn).serve.func1(0xc4205b80a0)
Nov 01 12:13:45 marist02 hyperd[9839]:         /usr/local/go/src/net/http/server.go:1697 +0xc2
Nov 01 12:13:45 marist02 hyperd[9839]: panic(0x80ee2b40, 0x811cec40)
Nov 01 12:13:45 marist02 hyperd[9839]:         /usr/local/go/src/runtime/panic.go:491 +0x26e
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor.(*VmContext).reportVmFault(...)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor/report.go:66
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor.(*VmContext).startPod(0xc42066c6e0, 0xc420191260, 0x805c6652)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor/vm_states.go:225 +0x2fa
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor.(*Vm).InitSandbox(0xc420064320, 0xc420191260, 0xc4200ba8f0, 0x80)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/hyperhq/runv/hypervisor/vm.go:231 +0x4e
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon/pod.(*XPod).createSandbox(0xc4200ba8f0, 0xc4211232c0, 0xc4210036b0, 0xc4200ba8f0)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/daemon/pod/provision.go:272 +0x33a
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon/pod.CreateXPod(0xc420190ae0, 0xc4211232c0, 0x0, 0x0, 0x0)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/daemon/pod/provision.go:36 +0x100
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon.(*Daemon).CreatePod(0xc420228070, 0x0, 0x0, 0xc4211232c0, 0xc4211232c0, 0x0, 0x0)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/daemon/run.go:36 +0x27e
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/daemon.(*Daemon).CmdCreatePod(0xc420228070, 0xc4205f00b0, 0xac, 0x200, 0xc4205f00b0, 0xac)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/daemon/server.go:260 +0xc8
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server/router/pod.(*podRouter).postPodCreate(0xc42108f890, 0x3ffa085dfc0, 0xc4205b22d0, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x818efaa0, 0xc4205b22d0)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/server/router/pod/pod_routes.go:70 +0x20a
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server/router/pod.(*podRouter).(github.com/hyperhq/hyperd/server/router/pod.postPodCreate)-fm(0x3ffa085dfc0, 0xc4205b22d0, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x3ffa085dfc0, 0xc4205b22d0)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/server/router/pod/pod.go:26 +0x5a
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.versionMiddleware.func1(0x3ffa085ac08, 0xc420012018, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x810c71ac, 0xe)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/server/middleware.go:156 +0x166
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*Server).corsMiddleware.func1(0x3ffa085ac08, 0xc420012018, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x80e4e9e0, 0x811c86c8)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/server/middleware.go:134 +0xa2
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*Server).userAgentMiddleware.func1(0x3ffa085ac08, 0xc420012018, 0x818ee4a0, 0xc42114a000, 0xc42112c600, 0xc4205b2150, 0x8103f7a0, 0x100000080fca580)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/server/middleware.go:117 +0xd4
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*Server).makeHTTPHandler.func1(0x818ee4a0, 0xc42114a000, 0xc42112c600)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/server/server.go:169 +0x1cc
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.HandlerFunc.ServeHTTP(0xc421127560, 0x818ee4a0, 0xc42114a000, 0xc42112c600)
Nov 01 12:13:45 marist02 hyperd[9839]:         /usr/local/go/src/net/http/server.go:1918 +0x42
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/vendor/github.com/gorilla/mux.(*Router).ServeHTTP(0xc42101e050, 0x818ee4a0, 0xc42114a000, 0xc42112c600)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/vendor/github.com/gorilla/mux/mux.go:114 +0xde
Nov 01 12:13:45 marist02 hyperd[9839]: github.com/hyperhq/hyperd/server.(*routerSwapper).ServeHTTP(0xc42114dc50, 0x818ee4a0, 0xc42114a000, 0xc42007c300)
Nov 01 12:13:45 marist02 hyperd[9839]:         /root/go/src/github.com/hyperhq/hyperd/server/router_swapper.go:29 +0x78
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.serverHandler.ServeHTTP(0xc4210e9380, 0x818ee4a0, 0xc42114a000, 0xc42007c300)
Nov 01 12:13:45 marist02 hyperd[9839]:         /usr/local/go/src/net/http/server.go:2619 +0x96
Nov 01 12:13:45 marist02 hyperd[9839]: net/http.(*conn).serve(0xc4205b80a0, 0x818ef9e0, 0xc42004e440)
Nov 01 12:13:45 marist02 hyperd[9839]:         /usr/local/go/src/net/http/server.go:1801 +0x6f8
Nov 01 12:13:45 marist02 hyperd[9839]: created by net/http.(*Server).Serve
Nov 01 12:13:45 marist02 hyperd[9839]:         /usr/local/go/src/net/http/server.go:2720 +0x264
Nov 01 12:13:45 marist02 hyperd[9839]: E1101 12:13:45.701139    9839 json.go:570] SB[vm-oOMYrhTfNz] get hyperstart API version error: hyperstart closed
xuchenhao001 commented 7 years ago

At /var/log/hyper/qemu/vm-oOMYrhTfN.log I found logs about vm:

qemu-system-s390x: -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir: 'virtio-9p-ccw' is not a valid device model name
gnawux commented 7 years ago

looks like your qemu did not build with VirtFs enabled?

xuchenhao001 commented 7 years ago

I rebuild qemu with:

git clone git://git.qemu.org/qemu.git
cd qemu
git checkout v2.10.1
git submodule init
git submodule update --recursive
./configure --prefix=/usr --enable-virtfs
make
make install

but it seems not working with the same error:

-- The start-up result is done.
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Using default logging driver none"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="[graphdriver] trying provided driver "overlay""
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Using graph driver overlay"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Option DefaultDriver: bridge"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Option DefaultNetwork: bridge"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=info msg="Firewalld running: false"
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Registering ipam driver: "default""
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=warning msg="Your kernel does not support swap memory limit."
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Cleaning up old shm/mqueue mounts: start."
Nov 02 09:37:39 marist02 hyperd[27626]: time="2017-11-02T09:37:39Z" level=debug msg="Cleaning up old shm/mqueue mounts: done."
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.179870   27626 hypervisor.go:37] SB[vm-WLrdFtGHVq] watch hyperstart timeout
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.180992   27626 json.go:570] SB[vm-WLrdFtGHVq] get hyperstart API version error: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181026   27626 vm_states.go:288] SB[vm-WLrdFtGHVq] watch hyperstart timeout
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181035   27626 vm_states.go:246] SB[vm-WLrdFtGHVq] Shutting down because of an exception: %!(EXTRA string=connection to vm broken)
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181133   27626 json.go:370] read tty data failed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181148   27626 json.go:427] SB[vm-WLrdFtGHVq] tty socket closed, quit the reading goroutine: read unix @->/var/run/hyper/vm-WLrdFtGHVq/tty.sock: use of closed network connection
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181160   27626 vm_states.go:226] SB[vm-WLrdFtGHVq] Start POD failed: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181178   27626 run.go:38] ubuntu-6203458613: failed to add pod: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.181194   27626 server.go:170] Handler for POST /v0.8.1/pod/create returned error: hyperstart closed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.185606   27626 json.go:137] read init data failed
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.185730   27626 json.go:171] SB[vm-WLrdFtGHVq] error when readVmMessage() for ready message: EOF
Nov 02 09:39:02 marist02 hyperd[27626]: E1102 09:39:02.185891   27626 qmp_handler.go:141] QMP exit as got error: EOF

:disappointed:

gnawux commented 7 years ago

could you find the same message

qemu-system-s390x: -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir: 'virtio-9p-ccw' is not a valid device model name
xuchenhao001 commented 7 years ago

No, I found there is a new log file at /var/log/hyper/qemu, but it has no content:

$ root@marist02:/var/log/hyper/qemu# ls -ltr
total 4
-rw-r--r-- 1 root root 126 Nov  2 03:18 vm-doHDBOZdv.log
-rw-r--r-- 1 root root   0 Nov  2 09:38 vm-WLrdFtGHV.log

(the first one is created before rebuild my qemu with --enable-virtfs)

gnawux commented 7 years ago

could you find the qemu command line and run it directly? I think there are still some qemu issue, but could not get any useful info here. It's too hard to debug without an environment to reproduce it...

xuchenhao001 commented 6 years ago

@gnawux After a long period of working on qemu-system-s390x, now i have something found: When I use libvirt to start a guest vm (Ubuntu) on s390x, i got qemu running parameters like this:

/usr/bin/qemu-system-s390x -name ubuntu16.04 -S -machine s390-ccw-virtio-xenial,accel=kvm,usb=off -m 1024 -realtime mlock=off -smp 1,maxcpus=2,sockets=2,cores=1,threads=1 -object iothread,id=iothread1 -uuid 4db4b6b7-145b-4942-8833-509da364e32b -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-ubuntu16.04/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc -no-shutdown -boot strict=on -kernel /var/lib/libvirt/images/kernel.ubuntu -initrd /var/lib/libvirt/images/initrd.ubuntu -device virtio-scsi-ccw,id=scsi0,devno=fe.0.0002 -drive file=/home/ubuntu/zhengxil/xchxubj/kvm/ubuntu.qcow2,format=raw,if=none,id=drive-virtio-disk0 -device virtio-blk-ccw,scsi=off,devno=fe.0.0000,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=2 -drive file=/home/ubuntu/zhengxil/xchxubj/kvm/xenial-server-s390x.iso,format=raw,if=none,id=drive-scsi0-0-0-1,readonly=on -device scsi-cd,bus=scsi0.0,channel=0,scsi-id=0,lun=1,drive=drive-scsi0-0-0-1,id=scsi0-0-0-1,bootindex=1 -netdev tap,fd=25,id=hostnet0,vhost=on,vhostfd=27 -device virtio-net-ccw,netdev=hostnet0,id=net0,mac=52:54:00:9c:58:88,devno=fe.0.0001 -chardev pty,id=charconsole0 -device sclpconsole,chardev=charconsole0,id=console0 -device virtio-balloon-ccw,id=balloon0,devno=fe.0.0003 -msg timestamp=on

On the other hand, when I type hyperctl run -t s390x/ubuntu:16.04, it hangs for a while. And at that time, I start an another ssh and check the qemu process with ps -ef | grep qemu:

/usr/bin/qemu-system-s390x -machine s390-ccw-virtio,accel=kvm,usb=off -cpu host -kernel /var/lib/hyper/kernel -initrd /var/lib/hyper/hyper-initrd.img -append "console=ttyS1 panic=1 no_timer_check" -realtime mlock=off -no-user-config -nodefaults -enable-kvm -rtc base=utc,clock=vm,driftfix=slew -no-reboot -display none -boot strict=on -m 128 -smp 1 -qmp unix:/var/run/hyper/vm-ESajskTODD/qmp.sock,server,nowait -chardev socket,id=charconsole0,path=/var/run/hyper/vm-ESajskTODD/console.sock,server,nowait -device sclpconsole,chardev=charconsole0 -device virtio-serial-ccw,id=virtio-serial0 -device virtio-scsi-ccw,id=scsi0 -chardev socket,id=charch0,path=/var/run/hyper/vm-ESajskTODD/hyper.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 -chardev socket,id=charch1,path=/var/run/hyper/vm-ESajskTODD/tty.sock,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 -fsdev local,id=virtio9p,path=/var/run/hyper/vm-ESajskTODD/share_dir,security_model=none -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir -daemonize -pidfile /var/run/hyper/vm-ESajskTODD/pidfile -D /var/log/hyper/qemu/vm-ESajskTOD.log

Roughly speaking, there are a lot of differences between this two. However, as I know, s390x do not support tty, but pty instead. So if I can configure the default parameters with hyperctl to start qemu-system-s390x? Here is my xml file for libvirt:

<domain type = 'kvm'>
    <name>ubuntu</name>
    <memory unit='KiB'>2097152</memory>
    <currentMemory unit='KiB'>1048576</currentMemory>
    <vcpu current='1'>2</vcpu>
    <os>
        <type arch = 's390x' machine = 's390-ccw-virtio'>hvm</type>
    </os>
    <devices>
        <emulator>/usr/bin/qemu-system-s390x</emulator>
        <disk type = 'file' device = 'disk'>
            <source file = '/home/ubuntu/kvm/ubuntu16.04.raw'/>
            <target dev='vda' bus='virtio'/>
        </disk>
        <interface type = 'bridge'>
            <source bridge = 'virbr0'/>
        </interface>
        <console type='pty'>
            <target type='sclp' port='0'/>
        </console>
    </devices>
</domain>

Thanks!

gao-feng commented 6 years ago

@xuchenhao001

hyper uses two unix sockets to communicate with hyperstart in vm, for x86/arm64, these two unix socket on host are represented as virtio-port in vm. the path are /var/run/hyper/vm-oOMYrhTfNz/hyper.sock and /var/run/hyper/vm-oOMYrhTfNz/tty.sock on host.

so you should launch qemu with virtserialport device. https://github.com/hyperhq/runv/blob/master/hypervisor/qemu/qemu_amd64.go#L87

xuchenhao001 commented 6 years ago

@gao-feng Thanks! But as i know, there is no ttyS device on s390x machine:

$ ls /dev
autofs         chsc     cpu_dma_latency  dasda1  disk      fd         hwrng    lightnvm      loop1  loop5   mem               network_latency     prandom  sclp        stderr  tty       vhost-net
block          clp      cuse             dasda2  dm-0      full       initctl  log           loop2  loop6   memory_bandwidth  network_throughput  ptmx     sclp_line0  stdin   ttysclp0  zero
btrfs-control  console  dasd_eer         dasdb   dm-1      fuse       kmsg     loop-control  loop3  loop7   mqueue            null                pts      shm         stdout  urandom
char           core     dasda            dasdb1  ecryptfs  hugepages  kvm      loop0         loop4  mapper  net               port                random   snapshot    sys-vg  vfio

So which one should I use?

pmorjan commented 6 years ago

The console device is ttysclp0. Here is an old example of qemu parameters from of a working docker/ hyperstart/runv setup that includes the two unix sockets:

-machine s390-ccw-virtio,accel=kvm,usb=off \
    -cpu host \
    -kernel /var/lib/runv/kernel \
    -initrd /var/lib/runv/initrd \
    -append "console=ttyS1 panic=1 no_timer_check" \
    -realtime mlock=off \
    -no-user-config \
    -nodefaults \
    -enable-kvm \
    -no-reboot \
    -display none \
    -boot strict=on \
    -m 512 \
    -smp 4 \
    -qmp unix:/var/run/hyper/vm-rnqlfpzLLe/qmp.sock,server,nowait \
    -device virtio-scsi-ccw,id=scsi0 \
    -chardev socket,id=charconsole0,path=/var/run/hyper/vm-rnqlfpzLLe/console.sock,server,nowait \
    -device sclpconsole,chardev=charconsole0 \
    -device virtio-serial-ccw,id=virtio-serial0 \
    -chardev socket,id=charch0,path=/var/run/hyper/vm-rnqlfpzLLe/hyper.sock,server,nowait \
    -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
    -chardev socket,id=charch1,path=/var/run/hyper/vm-rnqlfpzLLe/tty.sock,server,nowait \
    -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
    -fsdev local,id=virtio9p,path=/var/run/hyper/vm-rnqlfpzLLe/share_dir,security_model=none \
    -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir \
    -daemonize \
    -pidfile /var/run/hyper/vm-rnqlfpzLLe/pidfile \
    -D /var/log/hyper/qemu/vm-rnqlfpzLLe.log
gnawux commented 6 years ago

thank you @pmorjan nice to see you again here.

xuchenhao001 commented 6 years ago

@pmorjan Thank you for your help! I have read your code about this old example. It's working for runv to start or remove an vm. However, when I start a vm from command hyperctl run -t s390x/ubuntu:16.04, it seems that hyperd cannot read any data from qemu. Just like the error log before:

E1102 09:39:02.181133   27626 json.go:370] read tty data failed

I think if there is something wrong with these lines in your code:

"-append", "\"console=ttyS1 panic=1 no_timer_check\"",

"-chardev", fmt.Sprintf("socket,id=charch0,path=%s,server,nowait", ctx.HyperSockName),
"-device", "virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0",
"-chardev", fmt.Sprintf("socket,id=charch1,path=%s,server,nowait", ctx.TtySockName),
"-device", "virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1",

Because there is no ttyS device on s390x at all.

gnawux commented 6 years ago

@xuchenhao001 do you have this parameter -device sclpconsole,chardev=charconsole0 provided by @pmorjan

xuchenhao001 commented 6 years ago

@gnawux Of course I do. here is the qemu process detail:

/usr/bin/qemu-system-s390x \
-machine s390-ccw-virtio,accel=kvm,usb=off \
-cpu host \
-kernel /var/lib/hyper/kernel \
-initrd /var/lib/hyper/hyper-initrd.img \
-append "console=ttyS1 panic=1 no_timer_check" \
-realtime mlock=off \
-no-user-config \
-nodefaults \
-enable-kvm \
-rtc base=utc,clock=vm,driftfix=slew \
-no-reboot \
-display none \
-boot strict=on \
-m 128 \
-smp 1 \
-qmp unix:/var/run/hyper/vm-ESajskTODD/qmp.sock,server,nowait \
-chardev socket,id=charconsole0,path=/var/run/hyper/vm-ESajskTODD/console.sock,server,nowait \
-device sclpconsole,chardev=charconsole0 \
-device virtio-serial-ccw,id=virtio-serial0 \
-device virtio-scsi-ccw,id=scsi0 \
-chardev socket,id=charch0,path=/var/run/hyper/vm-ESajskTODD/hyper.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
-chardev socket,id=charch1,path=/var/run/hyper/vm-ESajskTODD/tty.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
-fsdev local,id=virtio9p,path=/var/run/hyper/vm-ESajskTODD/share_dir,security_model=none \
-device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir \
-daemonize \
-pidfile /var/run/hyper/vm-ESajskTODD/pidfile \
-D /var/log/hyper/qemu/vm-ESajskTOD.log

I was just query the validity of some parameters:

-append "console=ttyS1 panic=1 no_timer_check" \
# and
-chardev socket,id=charch0,path=/var/run/hyper/vm-ESajskTODD/hyper.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
-chardev socket,id=charch1,path=/var/run/hyper/vm-ESajskTODD/tty.sock,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
pmorjan commented 6 years ago

@xuchenhao001 It restored my old build environment and can confirm that the qemu config works as expected. I successfully ran a container via hyperctl and hyperd(v0.8.1) on s390x. This is the qemu command line of a running hyperd container:

/usr/bin/qemu-system-s390x \
    -machine s390-ccw-virtio,accel=kvm,usb=off \
    -cpu host \
    -kernel /data/runv/dist/kernel \
    -initrd /data/runv/dist/initrd \
    -append "console=ttyS1 panic=1 no_timer_check" \
    -realtime mlock=off \
    -no-user-config \
    -nodefaults \
    -enable-kvm \
    -rtc base=utc,clock=vm,driftfix=slew \
    -no-reboot \
    -display none \
    -boot strict=on \
    -m 128 \
    -smp 1 \
    -qmp unix:/var/run/hyper/vm-FDpmPVFtfA/qmp.sock,server,nowait \
    -chardev socket,id=charconsole0,path=/var/run/hyper/vm-FDpmPVFtfA/console.sock,server,nowait \
    -device sclpconsole,chardev=charconsole0 \
    -device virtio-serial-ccw,id=virtio-serial0 \
    -device virtio-scsi-ccw,id=scsi0 \
    -chardev socket,id=charch0,path=/var/run/hyper/vm-FDpmPVFtfA/hyper.sock,server,nowait \
    -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charch0,id=channel0,name=sh.hyper.channel.0 \
    -chardev socket,id=charch1,path=/var/run/hyper/vm-FDpmPVFtfA/tty.sock,server,nowait \
    -device virtserialport,bus=virtio-serial0.0,nr=2,chardev=charch1,id=channel1,name=sh.hyper.channel.1 \
    -fsdev local,id=virtio9p,path=/var/run/hyper/vm-FDpmPVFtfA/share_dir,security_model=none \
    -device virtio-9p-ccw,fsdev=virtio9p,mount_tag=share_dir \
    -daemonize \
    -pidfile /var/run/hyper/vm-FDpmPVFtfA/pidfile \
    -D /var/log/hyper/qemu/vm-FDpmPVFtf.log

I guess there is something wrong in your kernel / initrd setup.

But does it make sense not to follow upstream development of hyperd and stay on the backlevel 0.8.1 version? I did a quick look and it seems hyperd now depends on the lightweight kvmtool which is not available on s390x (as far as I know). On the other hand the overhead of Qemu on s390x is smaller compared to x86 (no BIOS) and it's well tested. Making upstream hyperd work on s390x again could be a challenge.

xuchenhao001 commented 6 years ago

@pmorjan Yes, I'm on version v0.8.1. And my qemu running parameters look the same as yours.

My host is on LPAR. For kernel and initrd, I simply download an iso from ubuntu offical website, and extract boot/kernel.ubuntu & boot/initrd.ubuntu from iso to /var/lib/hyper/kernel & /var/lib/hyper/hyper-initrd.img. Is that right? Since that I successfully run ubuntu installer on s390x with this two files, I take them for granted as kernel & hyper-initrd.img.

pmorjan commented 6 years ago

@xuchenhao001 I'm not sure if your kernel will work at all. Does it have the 9p/virtio modules builtin? If not then you have to include them into your initrd and patch hyperstart/src/init to load them. 9pfs is the filesytem type used for the rootfs.

xuchenhao001 commented 6 years ago

@pmorjan Ok, thank you! I will check it up.

What's more, how did you build your kernel & hyper-initrd.img on s390x? If you can provide your kernel_config file at hyperstart project? Thank you!

pmorjan commented 6 years ago

@xuchenhao001 The important config options are as follows:

CONFIG_NET_9P=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_FSCACHE=y
CONFIG_9P_FS=y

See the attached config for vanilla 4.8.0 that worked for me: config-4.8.0-s390x.gz

$ ./hyperctl   run -t alpine sh
/ # uname -a
Linux alpine-2987341563 4.8.0 #6 SMP Wed Nov 22 12:38:13 CET 2017 s390x Linux
/ # lsmod
Module                  Size  Used by    Not tainted
/ #
xuchenhao001 commented 6 years ago

@pmorjan Thank you!!! It works for me!! :smile: What's more, I tried to build a linux kernel v4.12.4 based on your config, luckily, it still works! Also, thank you @gnawux for your help!