nabla-containers / runnc

OCI-interfacing Container runtime for Nabla Containers
https://nabla-containers.github.io/
ISC License
257 stars 22 forks source link

Unable to run nabla containers on VM #55

Closed qvicksilver closed 5 years ago

qvicksilver commented 5 years ago

Trying to follow the blog post here: https://nabla-containers.github.io/2018/06/28/nabla-setup/

However instead of node-express-nabla starting I get this:

EDITED for formating

# docker run --rm -p 8080:8080 --runtime=runnc nablact/node-express-nabla &
[1] 9112
Unable to find image 'nablact/node-express-nabla:latest' locally
latest: Pulling from nablact/node-express-nabla
97f317d44c0e: Pull complete 
e16adb3ca835: Pull complete 
Digest: sha256:d5eacd917462da6b65899fa12d5de555378edfd5dd29d0558434a535a7e4b56c
Status: Downloaded newer image for nablact/node-express-nabla:latest
Running with args: [/opt/runnc/bin/runnc-cont -k8s -nabla-run /opt/runnc/bin/nabla-run -tap tape19b6c36d8bb -cwd / -volume /var/run/docker/runtime-runnc/moby/e19b6c36d8bbcbbe1445e8de11185e2676178c72a4523e14eef74585dbc3daa5/rootfs.iso:/ -mem 512 -unikernel /var/lib/docker/overlay2/6e7204d5f1daef45219f3d6fc48d276e4b8e26bd9acbf72065278870fbbc8e51/merged/node.nabla -env PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin -env HOSTNAME=e19b6c36d8bb -- /home/node/app/app.js][/opt/runnc/bin/nabla-run --mem=512 --net-mac=02:42:ac:11:00:02 --net=tape19b6c36d8bb --disk=/var/run/docker/runtime-runnc/moby/e19b6c36d8bbcbbe1445e8de11185e2676178c72a4523e14eef74585dbc3daa5/rootfs.iso /var/lib/docker/overlay2/6e7204d5f1daef45219f3d6fc48d276e4b8e26bd9acbf72065278870fbbc8e51/merged/node.nabla {"env":"PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin","env":"HOSTNAME=e19b6c36d8bb","cmdline":"/var/lib/docker/overlay2/6e7204d5f1daef45219f3d6fc48d276e4b8e26bd9acbf72065278870fbbc8e51/merged/node.nabla /home/node/app/app.js","net":{"if":"ukvmif0","cloner":"True","type":"inet","method":"static","addr":"172.17.0.2","mask":"16","gw":"172.17.0.1"},"blk":{"source":"etfs","path":"/dev/ld0a","fstype":"blk","mountpoint":"/"},"cwd":"/"}]
            |      ___|
  __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Memory map: 512 MB addressable:
Solo5:     unused @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0xaed6f7)
Solo5:     rodata @ (0xaed6f8 - 0xda55b7)
Solo5:       data @ (0xda55b8 - 0xfdbea7)
Solo5:       heap >= 0xfdc000 < stack < 0x20000000
[1]+  Exit 136                docker run --rm -p 8080:8080 --runtime=runnc nablact/node-express-nabla

I'm trying to run this on an Ubuntu 18.04.1 VM using KVM as hypervisor. Any ideas?

InTheCloudDan commented 5 years ago

Hi @qvicksilver feel free to join our Slack Team! https://slack.nabla-containers.com I'm still working on understanding the full code base myself so I'm not exactly sure how to enable more debugging for this issue but I'm looking into it.

Can you also provide the docker version installed on your system?

lumjjb commented 5 years ago

He is in our slack, we've resolved the issue :), just putting it here for searchable reference :).

InTheCloudDan commented 5 years ago

@djwillia I looked at the channel, I think you provided the definitive answer, mind answering it here so it's clear without the slack context?

lumjjb commented 5 years ago

tl; dr; This has to do with the way the CPU is emulated in the VM. When launching the VM, try using -cpu host instead of -cpu Haswell-IBRS

Long debug output:

Try:

sudo /opt/runnc/bin/runnc-cont -nabla-run /opt/runnc/bin/nabla-run -unikernel tests/integration/test_hello.nabla

^ assumes you are in the runnc github directory for tests/integration/test_hello.nabla

Get:

root@ubuntu:~/go/src/github.com/nabla-containers/runnc# /opt/runnc/bin/runnc-cont -nabla-run /opt/runnc/bin/nabla-run  -unikernel tests/integration/test_hello.nabla
[/opt/runnc/bin/nabla-run --mem=512 --net=tap100 --disk=/tmp/nabla146319861 tests/integration/test_hello.nabla {“cmdline”:“tests/integration/test_hello.nabla “,”net”:{“if”:“ukvmif0",“cloner”:“True”,“type”:“inet”,“method”:“static”,“addr”:“10.0.0.2",“mask”:“24",“gw”:“10.0.0.1"},“cwd”:“/”}]
           |      ___|
 __|  _ \  |  _ \ __ \
\__ \ (   | | (   |  ) |
____/\___/ _|\___/____/
Solo5: Memory map: 512 MB addressable:
Solo5:     unused @ (0x0 - 0xfffff)
Solo5:       text @ (0x100000 - 0x103fff)
Solo5:     rodata @ (0x104000 - 0x104fff)
Solo5:       data @ (0x105000 - 0x109fff)
Solo5:       heap >= 0x10a000 < stack < 0x20000000
Floating point exception

Try:

Solo5 tests run

root@ubuntu:~/solo5/tests# ./run-tests.sh -v
--------------------------------------------------------------------------------
Starting tests at Fri Nov 30 16:26:17 CET 2018
test_hello.ukvm                 : ERROR
/tmp/tmp.qiOAxdtK0U/test_hello.ukvm.log.0:
> -------- test_hello.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_hello/ukvm-bin -- /home/ubuntu/solo5/tests/test_hello/test_hello.ukvm Hello_Solo5
>             |      ___|
>   __|  _ \  |  _ \ __ \
> \__ \ (   | | (   |  ) |
> ____/\___/ _|\___/____/
> Solo5: Memory map: 1024 MB addressable:
> Solo5:     unused @ (0x0 - 0xfffff)
> Solo5:       text @ (0x100000 - 0x103fff)
> Solo5:     rodata @ (0x104000 - 0x104fff)
> Solo5:       data @ (0x105000 - 0x109fff)
> Solo5:       heap >= 0x10a000 < stack < 0x40000000
> Floating point exception
> -------- test_hello.ukvm END (status 136) --------
test_quiet.ukvm                 : ERROR
/tmp/tmp.qiOAxdtK0U/test_quiet.ukvm.log.0:
> -------- test_quiet.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_quiet/ukvm-bin -- /home/ubuntu/solo5/tests/test_quiet/test_quiet.ukvm --solo5:quiet
> Floating point exception
> -------- test_quiet.ukvm END (status 136) --------
test_globals.ukvm               : ERROR
/tmp/tmp.qiOAxdtK0U/test_globals.ukvm.log.0:
> -------- test_globals.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_globals/ukvm-bin -- /home/ubuntu/solo5/tests/test_globals/test_globals.ukvm
>             |      ___|
>   __|  _ \  |  _ \ __ \
> \__ \ (   | | (   |  ) |
> ____/\___/ _|\___/____/
> Solo5: Memory map: 1024 MB addressable:
> Solo5:     unused @ (0x0 - 0xfffff)
> Solo5:       text @ (0x100000 - 0x103fff)
> Solo5:     rodata @ (0x104000 - 0x104fff)
> Solo5:       data @ (0x105000 - 0x109fff)
> Solo5:       heap >= 0x10a000 < stack < 0x40000000
> Floating point exception
> -------- test_globals.ukvm END (status 136) --------
test_exception.ukvm             : ERROR
/tmp/tmp.qiOAxdtK0U/test_exception.ukvm.log.0:
> -------- test_exception.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_exception/ukvm-bin -- /home/ubuntu/solo5/tests/test_exception/test_exception.ukvm
>             |      ___|
>   __|  _ \  |  _ \ __ \
> \__ \ (   | | (   |  ) |
> ____/\___/ _|\___/____/
> Solo5: Memory map: 1024 MB addressable:
> Solo5:     unused @ (0x0 - 0xfffff)
> Solo5:       text @ (0x100000 - 0x103fff)
> Solo5:     rodata @ (0x104000 - 0x104fff)
> Solo5:       data @ (0x105000 - 0x109fff)
> Solo5:       heap >= 0x10a000 < stack < 0x40000000
> Floating point exception
> -------- test_exception.ukvm END (status 136) --------
test_fpu.ukvm                   : ERROR
/tmp/tmp.qiOAxdtK0U/test_fpu.ukvm.log.0:
> -------- test_fpu.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_fpu/ukvm-bin -- /home/ubuntu/solo5/tests/test_fpu/test_fpu.ukvm
>             |      ___|
>   __|  _ \  |  _ \ __ \
> \__ \ (   | | (   |  ) |
> ____/\___/ _|\___/____/
> Solo5: Memory map: 1024 MB addressable:
> Solo5:     unused @ (0x0 - 0xfffff)
> Solo5:       text @ (0x100000 - 0x103fff)
> Solo5:     rodata @ (0x104000 - 0x104fff)
> Solo5:       data @ (0x105000 - 0x109fff)
> Solo5:       heap >= 0x10a000 < stack < 0x40000000
> Floating point exception
> -------- test_fpu.ukvm END (status 136) --------
test_time.ukvm                  : ERROR
/tmp/tmp.qiOAxdtK0U/test_time.ukvm.log.0:
> -------- test_time.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_time/ukvm-bin -- /home/ubuntu/solo5/tests/test_time/test_time.ukvm
>             |      ___|
>   __|  _ \  |  _ \ __ \
> \__ \ (   | | (   |  ) |
> ____/\___/ _|\___/____/
> Solo5: Memory map: 1024 MB addressable:
> Solo5:     unused @ (0x0 - 0xfffff)
> Solo5:       text @ (0x100000 - 0x103fff)
> Solo5:     rodata @ (0x104000 - 0x104fff)
> Solo5:       data @ (0x105000 - 0x109fff)
> Solo5:       heap >= 0x10a000 < stack < 0x40000000
> Floating point exception
> -------- test_time.ukvm END (status 136) --------
test_blk.ukvm                   : ERROR
/tmp/tmp.qiOAxdtK0U/test_blk.ukvm.log.1:
> + dd if=/dev/zero of=/tmp/tmp.qiOAxdtK0U/disk.img bs=4k count=1024
> 1024+0 records in
> 1024+0 records out
> 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.00231656 s, 1.8 GB/s
/tmp/tmp.qiOAxdtK0U/test_blk.ukvm.log.0:
> -------- test_blk.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_blk/ukvm-bin --disk=/tmp/tmp.qiOAxdtK0U/disk.img -- /home/ubuntu/solo5/tests/test_blk/test_blk.ukvm
>             |      ___|
>   __|  _ \  |  _ \ __ \
> \__ \ (   | | (   |  ) |
> ____/\___/ _|\___/____/
> Solo5: Memory map: 1024 MB addressable:
> Solo5:     unused @ (0x0 - 0xfffff)
> Solo5:       text @ (0x100000 - 0x103fff)
> Solo5:     rodata @ (0x104000 - 0x104fff)
> Solo5:       data @ (0x105000 - 0x109fff)
> Solo5:       heap >= 0x10a000 < stack < 0x40000000
> Floating point exception
> -------- test_blk.ukvm END (status 136) --------
test_ping_serve.ukvm            : ERROR
/tmp/tmp.qiOAxdtK0U/test_ping_serve.ukvm.log.0:
> -------- test_ping_serve.ukvm START --------
> + timeout 30s /home/ubuntu/solo5/tests/test_ping_serve/ukvm-bin --net=tap100 -- /home/ubuntu/solo5/tests/test_ping_serve/test_ping_serve.ukvm limit
>             |      ___|
>   __|  _ \  |  _ \ __ \
> \__ \ (   | | (   |  ) |
> ____/\___/ _|\___/____/
> Solo5: Memory map: 1024 MB addressable:
> Solo5:     unused @ (0x0 - 0xfffff)
> Solo5:       text @ (0x100000 - 0x103fff)
> Solo5:     rodata @ (0x104000 - 0x104fff)
> Solo5:       data @ (0x105000 - 0x109fff)
> Solo5:       heap >= 0x10a000 < stack < 0x40000000
> Floating point exception
> -------- test_ping_serve.ukvm END (status 136) --------
/tmp/tmp.qiOAxdtK0U/test_ping_serve.ukvm.log.2:
> + timeout 30s ping -fq -c 100000 10.0.0.2
> PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
test_hello.virtio               : ERROR
/tmp/tmp.qiOAxdtK0U/test_hello.virtio.log.0:
> -------- test_hello.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -- /home/ubuntu/solo5/tests/test_hello/test_hello.virtio Hello_Solo5
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_hello/test_hello.virtio -append Hello_Solo5
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 176: exec: qemu-system-x86_64: not found
> -------- test_hello.virtio END (status 127) --------
test_quiet.virtio               : ERROR
/tmp/tmp.qiOAxdtK0U/test_quiet.virtio.log.0:
> -------- test_quiet.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -- /home/ubuntu/solo5/tests/test_quiet/test_quiet.virtio --solo5:quiet
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_quiet/test_quiet.virtio -append --solo5:quiet
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 176: exec: qemu-system-x86_64: not found
> -------- test_quiet.virtio END (status 127) --------
test_globals.virtio             : ERROR
/tmp/tmp.qiOAxdtK0U/test_globals.virtio.log.0:
> -------- test_globals.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -- /home/ubuntu/solo5/tests/test_globals/test_globals.virtio
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_globals/test_globals.virtio
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 179: exec: qemu-system-x86_64: not found
> -------- test_globals.virtio END (status 127) --------
test_exception.virtio           : ERROR
/tmp/tmp.qiOAxdtK0U/test_exception.virtio.log.0:
> -------- test_exception.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -- /home/ubuntu/solo5/tests/test_exception/test_exception.virtio
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_exception/test_exception.virtio
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 179: exec: qemu-system-x86_64: not found
> -------- test_exception.virtio END (status 127) --------
test_fpu.virtio                 : ERROR
/tmp/tmp.qiOAxdtK0U/test_fpu.virtio.log.0:
> -------- test_fpu.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -- /home/ubuntu/solo5/tests/test_fpu/test_fpu.virtio
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_fpu/test_fpu.virtio
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 179: exec: qemu-system-x86_64: not found
> -------- test_fpu.virtio END (status 127) --------
test_time.virtio                : ERROR
/tmp/tmp.qiOAxdtK0U/test_time.virtio.log.0:
> -------- test_time.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -- /home/ubuntu/solo5/tests/test_time/test_time.virtio
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_time/test_time.virtio
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 179: exec: qemu-system-x86_64: not found
> -------- test_time.virtio END (status 127) --------
test_blk.virtio                 : ERROR
/tmp/tmp.qiOAxdtK0U/test_blk.virtio.log.0:
> -------- test_blk.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -d /tmp/tmp.qiOAxdtK0U/disk.img -- /home/ubuntu/solo5/tests/test_blk/test_blk.virtio
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -drive file=/tmp/tmp.qiOAxdtK0U/disk.img,if=virtio,format=raw -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_blk/test_blk.virtio
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 179: exec: qemu-system-x86_64: not found
> -------- test_blk.virtio END (status 127) --------
/tmp/tmp.qiOAxdtK0U/test_blk.virtio.log.1:
> + dd if=/dev/zero of=/tmp/tmp.qiOAxdtK0U/disk.img bs=4k count=1024
> 1024+0 records in
> 1024+0 records out
> 4194304 bytes (4.2 MB, 4.0 MiB) copied, 0.00878381 s, 478 MB/s
test_ping_serve.virtio          : ERROR
/tmp/tmp.qiOAxdtK0U/test_ping_serve.virtio.log.0:
> -------- test_ping_serve.virtio START --------
> + timeout 30s /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh -n tap100 -- /home/ubuntu/solo5/tests/test_ping_serve/test_ping_serve.virtio limit
> + exec qemu-system-x86_64 -cpu Westmere -m 128 -nodefaults -no-acpi -display none -serial stdio -device virtio-net,netdev=n0 -netdev tap,id=n0,ifname=tap100,script=no,downscript=no -device isa-debug-exit -kernel /home/ubuntu/solo5/tests/test_ping_serve/test_ping_serve.virtio -append limit
> /home/ubuntu/solo5/tests/../tools/run/solo5-run-virtio.sh: 176: exec: qemu-system-x86_64: not found
> -------- test_ping_serve.virtio END (status 127) --------
/tmp/tmp.qiOAxdtK0U/test_ping_serve.virtio.log.2:
> + timeout 30s ping -fq -c 100000 10.0.0.2
> PING 10.0.0.2 (10.0.0.2) 56(84) bytes of data.
Finished tests at Fri Nov 30 16:27:19 CET 2018
--------------------------------------------------------------------------------
Result                          : FAILURE

QEMU Launched arguments

/usr/libexec/qemu-kvm -name nabla -S -machine pc-i440fx-rhel7.0.0,accel=kvm,usb=off,dump-guest-core=off -cpu Haswell-IBRS,-hle,-rtm -m 2048 -realtime mlock=off -smp 1,sockets=1,cores=1,threads=1 -uuid 4f4a2889-4063-4a7c-9db9-e2011ad66511 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-224-nabla/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x4.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x4 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x4.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x4.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive file=/dev/virt_pool/nabla,format=raw,if=none,id=drive-virtio-disk0,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x6,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -netdev tap,fd=33,id=hostnet0,vhost=on,vhostfd=35 -device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:33:d1:cc,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/channel/target/domain-224-nabla/org.qemu.guest_agent.0,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

Issue resolved :). Use -cpu host instead of -cpu Haswell-IBRS