ARM: number of SMP CPUs requested (96) exceeds max CPUs supported

[devimc]

Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=debug msg="loaded configuration" format=TOML source=runtime
Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=info arguments="\"create --bundle /var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d --pid-file /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d/init.pid --console-socket /tmp/pty392961911/pty.sock bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d\"" commit=b9eddecc2c0f1c148fb61f885d87f6e88159f517 name=cc-runtime source=runtime version=3.0.17
Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=debug msg="converting /run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d/config.json" source=virtcontainers/oci
Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=debug default-kernel-parameters="init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d::off::" source=runtime
Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=debug msg="container rootfs: /var/lib/docker/overlay2/bacda3e3960f9a97550148f9b177db546a5305d599e7806fd614f16fcc50a17b/merged" source=virtcontainers/oci
Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=debug msg="Creating bridges" source=virtcontainers subsystem=qemu
Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=debug msg="Creating UUID" source=virtcontainers subsystem=qemu
Feb 09 17:07:03 parsnip cc-runtime[67336]: time="2018-02-09T17:07:03Z" level=debug msg="Could not retrieve anything from storage" source=virtcontainers subsystem=hyper
Feb 09 17:07:04 parsnip cc-runtime[67336]: time="2018-02-09T17:07:04Z" level=info msg="Attaching virtual endpoint" source=virtcontainers subsystem=network
Feb 09 17:07:04 parsnip cc-runtime[67336]: time="2018-02-09T17:07:04Z" level=info msg="Starting VM" pod-id=bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d source=virtcontainers subsystem=pod
Feb 09 17:07:04 parsnip cc-runtime[67336]: time="2018-02-09T17:07:04Z" level=debug default-kernel-parameters="root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug systemd.show_status=true systemd.log_level=debug" source=virtcontainers subsystem=qemu
Feb 09 17:07:04 parsnip cc-runtime[67336]: time="2018-02-09T17:07:04Z" level=debug msg="#### {/usr/local/bin/qemu-system-arm context.Background pod-bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d 18b2e84d-6c09-46ed-aa82-c42f09a6c33c host {virt } [{unix /run/virtcontainers/pods/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d/mon-18b2e84d-6c09 true true} {unix /run/virtcontainers/pods/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d/ctl-18b2e84d-6c09 true true}] [{virtio-serial-pci serial0 false} {socket virtconsole  console0 charconsole0 /run/virtcontainers/pods/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d/console.sock  false} {nvdimm memory-backend-file mem0 nv0 /usr/share/clear-containers/clear-containers-arm.img 536870912} {socket virtserialport  channel0 charch0 /run/virtcontainers/pods/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d/hyper.sock sh.hyper.channel.0 false} {socket virtserialport  channel1 charch1 /run/virtcontainers/pods/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d/tty.sock sh.hyper.channel.1 false} {virtio-9p-pci local extra-9p-hyperShared /run/hyper/shared/pods/bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d hyperShared none false}] {utc  slew} none {/usr/share/clear-containers/Image root=/dev/pmem0p1 rootflags=dax,data=ordered,errors=remount-ro rw rootfstype=ext4 tsc=reliable no_timer_check rcupdate.rcu_expedited=1 i8042.direct=1 i8042.dumbkbd=1 i8042.nopnp=1 i8042.noaux=1 noreplace-smp reboot=k panic=1 console=hvc0 console=hvc1 initcall_debug iommu=off cryptomgr.notests net.ifnames=0 pci=lastbus=0 debug systemd.show_status=true systemd.log_level=debug init=/usr/lib/systemd/systemd systemd.unit=clear-containers.target systemd.mask=systemd-networkd.service systemd.mask=systemd-networkd.socket ip=::::::bb4c7362ab4e938fcb8966a889b501cc5776245a9e6cddafac9877f36c7c672d::off::} {2048M 2 129896M} {96 96 1 1 0} kvm-pit.lost_tick_policy=discard {true true true true false false true false}  [] []}\n" source=virtcontainers subsystem=qemu
Feb 09 17:07:04 parsnip cc-runtime[67336]: time="2018-02-09T17:07:04Z" level=error msg="Unable to launch qemu: exit status 1" source=virtcontainers subsystem=qmp
Feb 09 17:07:04 parsnip cc-runtime[67336]: time="2018-02-09T17:07:04Z" level=error msg="qemu-system-arm: Number of SMP CPUs requested (96) exceeds max CPUs supported by machine 'mach-virt' (8)\n" source=virtcontainers subsystem=qmp
Feb 09 17:07:04 parsnip cc-runtime[67336]: time="2018-02-09T17:07:04Z" level=error msg="qemu-system-arm: Number of SMP CPUs requested (96) exceeds max CPUs supported by machine 'mach-virt' (8)\n" source=runtime

[devimc]

workaround: set default_vcpus=2 in configuration file

[jodh-intel]

I think the 96 is probably coming from:

• https://github.com/clearcontainers/runtime/blob/master/config.go#L188

It's a shame that:

• qemu-system-arm has this limitation.

• qemu doesn't provide a way to dump out such limits.

  If it did, we could query it at build time and then take the lower of the build-time and runtime values.

[grahamwhaley]

yes, I guess that 96 is the number of physical 'cpu's on the host, and the qemu can't handle it (as you say, sadly). The workaround of modifying the toml is valid, but a little limiting. It will do for now I think though.

[sboeuf]

Well in case of ARM, we can find the maximum of CPUs that can be used by Qemu, and put that into a constant into virtcontainers qemu.go. And in case we use qemu for ARM, we check the number of CPUs requested, that we cast to this max.

[jodh-intel]

Yes, we could do that. But what happens when qemu raises the limit on ARM? We might not notice and the user will be artificially limited to a smaller number of cpus.

@devimc - could you paste in the full qemu command-line? I've done a bit of digging and it appears that to overcome this limitation, we need to be running qemu with:

-machine virt,gic-version=$value,accel=kvm

The default $value of gic-version is 2, which seems to be limited to 8 cpus. Since your system appears to have 96 cpus, we need to set either gic-version=3 or gic-version=host (better).

[jamesodhunt]

Ah - I see you already have that fix in https://github.com/containers/virtcontainers/pull/614 ;)