Firecracker panicked at 'Error creating the Kvm object: Error(13)', src/vmm/src/vstate/system.rs:53:30

[bootrino]

I'm following this tutorial but I am also trying to do networking: https://aws.amazon.com/blogs/aws/firecracker-lightweight-virtualization-for-serverless-computing/

To do networking, I added the script below: 3_config_guest_network.sh

When I follow the tutorial without script 3_config_guest_network.sh the virtual machine starts OK.

When I run the script below 4_start_guest_machine.sh firecracker crashes with:

@homecloud:~$ ./firecracker --api-sock /tmp/firecracker.socket
2022-02-19T16:10:21.775601118 [anonymous-instance:main:ERROR:src/firecracker/src/main.rs:91] Firecracker panicked at 'Error creating the Kvm object: Error(13)', src/vmm/src/vstate/system.rs:53:30
Aborted
@homecloud:~$

1_set_guest_kernel.sh

arch=`uname -m`
kernel_path=$(pwd)"/hello-vmlinux.bin"

if [ ${arch} = "x86_64" ]; then
    curl --unix-socket /tmp/firecracker.socket -i \
      -X PUT 'http://localhost/boot-source'   \
      -H 'Accept: application/json'           \
      -H 'Content-Type: application/json'     \
      -d "{
            \"kernel_image_path\": \"${kernel_path}\",
            \"boot_args\": \"console=ttyS0 reboot=k panic=1 pci=off\"
       }"
elif [ ${arch} = "aarch64" ]; then
    curl --unix-socket /tmp/firecracker.socket -i \
      -X PUT 'http://localhost/boot-source'   \
      -H 'Accept: application/json'           \
      -H 'Content-Type: application/json'     \
      -d "{
            \"kernel_image_path\": \"${kernel_path}\",
            \"boot_args\": \"keep_bootcon console=ttyS0 reboot=k panic=1 pci=off\"
       }"
else
    echo "Cannot run firecracker on $arch architecture!"
    exit 1
fi

2_set_root_guestfs.sh

rootfs_path=$(pwd)"/hello-rootfs.ext4"
curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/drives/rootfs' \
  -H 'Accept: application/json'           \
  -H 'Content-Type: application/json'     \
  -d "{
        \"drive_id\": \"rootfs\",
        \"path_on_host\": \"${rootfs_path}\",
        \"is_root_device\": true,
        \"is_read_only\": false
   }"

3_config_guest_network.sh

curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/network-interfaces/eth0'       \
  -H  'Accept: application/json'          \
  -H  'Content-Type: application/json'    \
  -d '{
      "iface_id": "eth0",
      "guest_mac": "02:FC:00:00:00:00",
      "host_dev_name": "tap1"
   }'

4_start_guest_machine.sh

curl --unix-socket /tmp/firecracker.socket -i \
  -X PUT 'http://localhost/actions'       \
  -H  'Accept: application/json'          \
  -H  'Content-Type: application/json'    \
  -d '{"action_type": "InstanceStart"}'

### This is my network configuration:

/etc/systemd/network/31-tap1.netdev

[NetDev]
Description="This is a tap device for a VM or a container"
Name=tap1
Kind=tap

[Tap]

/etc/systemd/network/31-tap1.network

[Match]
Name=tap1

[Link]
RequiredForOnline=no

[Network]
Description="This is a tap device for a VM or a container"
LinkLocalAddressing=no
LLMNR=no
IPv6AcceptRA=no
Bridge=br0

/etc/systemd/network/br0.netdev

[NetDev]
Name=br0
Kind=bridge

/etc/systemd/network/br0.network

[Match]
Name=br0

[Network]
DHCP=no
DNS=8.8.8.8
DNS=8.8.4.4
Address=192.168.1.87/24
Gateway=192.168.1.1

/etc/systemd/network/eth0.network

[Match]
Name=enp38s0

[Network]
Bridge=br0

andrewstuart@homecloud:~$ networkctl status -a

● 1: lo
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: n/a
                          Type: loopback
                         State: carrier (unmanaged)
                    HW Address: 00:00:00:00:00:00
                           MTU: 65536
                         QDisc: noqueue
  IPv6 Address Generation Mode: eui64
          Queue Length (Tx/Rx): 1/1
                       Address: 127.0.0.1
                                ::1

● 2: enp38s0
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: /etc/systemd/network/eth0.network
                          Type: ether
                         State: enslaved (configured)
                          Path: pci-0000:26:00.0
                        Driver: r8169
                        Vendor: Realtek Semiconductor Co., Ltd.
                         Model: RTL8125 2.5GbE Controller
                    HW Address: d8:bb:c1:8d:7b:82
                           MTU: 1500 (min: 68, max: 9194)
                         QDisc: pfifo_fast
                        Master: br0
  IPv6 Address Generation Mode: none
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: yes
                         Speed: 1Gbps
                        Duplex: full
                          Port: tp

● 3: br0
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: /etc/systemd/network/br0.network
                          Type: bridge
                         State: degraded-carrier (configured)
                        Driver: bridge
                    HW Address: ea:57:6f:56:5c:88
                           MTU: 1500 (min: 68, max: 65535)
                         QDisc: noqueue
  IPv6 Address Generation Mode: eui64
                 Forward Delay: 15s
                    Hello Time: 2s
                       Max Age: 20s
                   Ageing Time: 5min
                      Priority: 32768
                           STP: no
        Multicast IGMP Version: 2
                          Cost: 2000
                    Port State: disabled
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 1Gbps
                       Address: 192.168.1.87
                                fe80::e857:6fff:fe56:5c88
                       Gateway: 192.168.1.1 (TP-LINK TECHNOLOGIES CO.,LTD.)
                           DNS: 8.8.8.8
                                8.8.4.4
             DHCP6 Client DUID: DUID-EN/Vendor:0000ab119d8d0bc303cf56130000

● 5: tap1
                     Link File: /usr/lib/systemd/network/99-default.link
                  Network File: /etc/systemd/network/31-tap1.network
                          Type: ether
                         State: no-carrier (configured)
                        Driver: tun
                    HW Address: da:e2:6d:4d:c3:73
                           MTU: 1500 (min: 68, max: 65521)
                         QDisc: pfifo_fast
                        Master: br0
  IPv6 Address Generation Mode: none
          Queue Length (Tx/Rx): 1/1
              Auto negotiation: no
                         Speed: 10Mbps
                        Duplex: full
                          Port: tp

[bootrino]

This happened because I had not run:

sudo setfacl -m u:${USER}:rw /dev/kvm

Suggestion to project - a good error message suggesting this might be the cause instead of crashing.

[mattschlebusch]

More detail in the associated pull request. The TL;DR is the issue was easily reproducible. Instead of terminating the FC process, FC will log a more precise log message if the InstanceStart API call fails to bootstrap the KVM device. The process will be left running to try the API call again after the /dev/kvm ACL has been updated.