firecracker-microvm / firecracker-demo

A demo running 4000 Firecracker microVMs.
http://firecracker-microvm.io
Apache License 2.0
247 stars 56 forks source link

Unable to connect to microvm from host via tap #47

Closed anthonycorletti closed 1 year ago

anthonycorletti commented 1 year ago

Hi! Thanks for creating this demo.

I'm experimenting with firecracker and when running this demo I ran into an issue that occurs after I've created the microvms. I'm unable to log in to a microvm and run iperf on it directly. Additionally I don't see the micro vm heat map light up so I think I've messed something up in my networking configuration.

I created an a1.metal instance with Amazon Linux because my account didn't have enough quota for an i3.metal and was able to ssh into it.

Here are the relevant security group rules (inbound at the top and outbound at the bottom):

Screenshot 2023-02-12 at 1 53 14 PM

Here's what I ran in terminal window 2:

[ec2-user@ip-172-31-31-137 firecracker-demo]$ sudo tee -a /etc/security/limits.conf <<EOL
> $USER soft nproc 16384
> $USER hard nproc 16384
> EOL
ec2-user soft nproc 16384
ec2-user hard nproc 16384
[ec2-user@ip-172-31-31-137 firecracker-demo]$ sudo chmod 777 /dev/kvm
[ec2-user@ip-172-31-31-137 firecracker-demo]$ which python3 iperf3
/usr/bin/python3
/usr/bin/iperf3
[ec2-user@ip-172-31-31-137 firecracker-demo]$ sudo ./0.initial-setup.sh 100
Saved firecracker at /home/ec2-user/firecracker-demo/scripts/../resources/firecracker
Saved kernel at /home/ec2-user/firecracker-demo/scripts/../resources/vmlinux...
Saved rootfs and ssh key at /home/ec2-user/firecracker-demo/scripts/../resources/rootfs.ext4 and /home/ec2-user/firecracker-demo/scripts/../resources/rootfs.id_rsa...
[ec2-user@ip-172-31-31-137 firecracker-demo]$ ./parallel-start-many.sh 0 100 6
Start @ Sun Feb 12 19:02:06 UTC 2023.
i is 0
i is 1
i is 2
i is 3
i is 4
i is 5
FC 69 still not ready...
FC 53 still not ready...
FC 86 still not ready...
FC 54 still not ready...
FC 70 still not ready...
FC 37 still not ready...
FC 87 still not ready...
FC 55 still not ready...
FC 39 still not ready...
FC 23 still not ready...
FC 90 still not ready...
FC 57 still not ready...
FC 24 still not ready...
FC 8 still not ready...
FC 75 still not ready...
FC 60 still not ready...
FC 27 still not ready...
FC 44 still not ready...
FC 61 still not ready...
FC 94 still not ready...
FC 77 still not ready...
FC 95 still not ready...
FC 78 still not ready...
FC 14 still not ready...
FC 14 still not ready...
FC 79 still not ready...
FC 96 still not ready...
FC 80 still not ready...
FC 99 still not ready...
FC 99 still not ready...
FC 82 still not ready...
Done @ Sun Feb 12 19:02:13 UTC 2023.
Started 100 microVMs in 6948 milliseconds.
MicroVM mutation rate was 16.66666666666666666666 microVMs per second.
[ec2-user@ip-172-31-31-137 firecracker-demo]$ ID="42"
[ec2-user@ip-172-31-31-137 firecracker-demo]$ ip addr show fc-$ID-tap0 | grep "inet "
    inet 169.254.0.170/30 scope global fc-42-tap0
[ec2-user@ip-172-31-31-137 firecracker-demo]$ ssh -i resources/rootfs.id_rsa root@169.254.0.169
ssh: connect to host 169.254.0.169 port 22: No route to host

Could you please let me know what I'm missing here? Because this is an arm instance I did comment out sudo modprobe kvm_intel here.

Thanks!

dianpopa commented 1 year ago

Hi @anthonycorletti !

Sorry this did not go smooth for you.

Can you also check that the microVM you are trying to connect to has actually started up? You can check that by looking at output/fc-sb$i-log.

anthonycorletti commented 1 year ago

Hi @dianpopa, thanks for getting back to me. So it looks like the microvm started but exited, and I'm not sure why.

[ec2-user@ip-172-31-29-236 firecracker-demo]$ cat output/fc-sb42-log
Running Firecracker v1.1.2
2023-03-04T23:46:57.959165656 [42:fc_api] The request was executed successfully. Status code: 204 No Content.
2023-03-04T23:46:57.969704637 [42:fc_api] The API server received a Put request on "/metrics" with body "{  \"metrics_path\": \"/dev/null\"}".
2023-03-04T23:46:57.969780621 [42:fc_api] The request was executed successfully. Status code: 204 No Content.
2023-03-04T23:46:58.038609978 [42:fc_api] The API server received a Put request on "/boot-source" with body "{  \"kernel_image_path\": \"/home/ec2-user/firecracker-demo/resources/vmlinux\",  \"boot_args\": \"init=/sbin/boottime_init panic=1 pci=off nomodules reboot=k tsc=reliable quiet i8042.nokbd i8042.noaux 8250.nr_uarts=0 ipv6.disable=1 ip=169.254.0.169::169.254.0.170:255.255.255.252::eth0:off\"}".
2023-03-04T23:46:58.038710250 [42:fc_api] The request was executed successfully. Status code: 204 No Content.
2023-03-04T23:46:58.068538664 [42:fc_api] The API server received a Put request on "/drives/1" with body "{  \"drive_id\": \"1\",  \"path_on_host\": \"/home/ec2-user/firecracker-demo/resources/rootfs.ext4\",  \"is_root_device\": true,  \"is_read_only\": true}".
2023-03-04T23:46:58.068659407 [42:fc_api] The request was executed successfully. Status code: 204 No Content.
2023-03-04T23:46:58.130778285 [42:fc_api] The API server received a Put request on "/network-interfaces/1" with body "{  \"iface_id\": \"1\",  \"guest_mac\": \"02:FC:00:00:00:2A\",  \"host_dev_name\": \"fc-42-tap0\"}".
2023-03-04T23:46:58.131725781 [42:fc_api] The request was executed successfully. Status code: 204 No Content.
2023-03-04T23:46:58.168533654 [42:fc_api] The API server received a Put request on "/actions" with body "{  \"action_type\": \"InstanceStart\"}".
2023-03-04T23:46:58.177196351 [42:main] Could not read cache level for index 3: Failed to read cache information: No such file or directory (os error 2)
2023-03-04T23:46:58.190800117 [42:main] Artificially kick devices.
2023-03-04T23:46:58.191006637 [42:fc_api] The request was executed successfully. Status code: 204 No Content.
2023-03-04T23:46:59.894391181 [42:fc_vcpu 0] Received KVM_SYSTEM_EVENT: type: 2, event: 0
2023-03-04T23:46:59.894470657 [42:main] Vmm is stopping.
2023-03-04T23:46:59.901157880 [42:fc_api] The API server received a Put request on "/shutdown-internal".
2023-03-04T23:46:59.901313796 [42:main] Vmm is stopping.

Any ideas on what I should try next?

dianpopa commented 1 year ago

You are receiving the KVM_SYSTEM_EVENT_RESET that signal the guest has requested a reset of the VM (i.e the guest was not able to boot with the KVM setup provided by the hypervisor). It might be that the host you are trying to run on does not recognize some of the components Firecracker sets up for the microVM. What is the ARM arch you are running on? (Firecracker supports armv8). Also does your host have support for GICv3? What host kernel you are using? Were you able to start a microVM outside the demo?

anthonycorletti commented 1 year ago

Hey! Yes, I was able to start a microvm outside the demo's instructions by following the getting started guide in firecracker's main repo.

My test was running on an arm (aarch64), a1.metal, ec2 instance running ubuntu 20.4. I didn't change any of the kernel settings in the demo.

anthonycorletti commented 1 year ago

@dianpopa I think my error was due to me flipping around the gateway IP and the tap IP address by mistake 🤦 I've gotten that resolved and in testing out this demo, decided to build hotbox – a simple way to run your code in the cloud on firecracker microvms! It would be awesome if you could check it out and give some feedback. thanks again for your help and for making this demo!