[linux_distributions][chromium_os] How to use crosvm

[LukeShortCloud]

Create custom virtual machines with crosvm.

https://www.reddit.com/r/PixelBook/comments/7zxz57/howto_boot_a_ubuntu_vm_using_crosvm/?utm_source=amp&utm_medium=&utm_content=post_body

[LukeShortCloud]

https://www.collabora.com/news-and-blog/blog/2017/11/09/quick-hack-experiments-with-crosvm/

[LukeShortCloud]

Here is a full example of how to use crosvm with Vulkan support enabled:

$ sudo LD_LIBRARY_PATH=<...> VK_ICD_FILENAMES=<...> ./target/debug/crosvm run \
      --gpu vulkan=true \
      --display-window-keyboard \
      --display-window-mouse \
      --host_ip 192.168.0.1 \
      --netmask 255.255.255.0 \
      --mac 12:34:56:78:9a:bc \
      --rwdisk disk.qcow2 \
      -p root=/dev/vda1 \
      <path-to-bzImage>

https://docs.mesa3d.org/drivers/venus.html

[LukeShortCloud]

crosvm requires (1) a Linux kernel and (2) a virtual disk with the root file system.

$ git clone --depth 1 -b chromeos-5.15 https://chromium.googlesource.com/chromiumos/third_party/kernel
$ cd ./kernel/
$ make chromiumos-container-vm-x86_64_defconfig
$ make -j $(nproc) bzImage

""" This kernel does not build any modules, nor does it support loading them, so there is no need to worry about an initramfs, although they are supported in crosvm. """

https://google.github.io/crosvm/running_crosvm/basic_usage.html

[LukeShortCloud]

Example of starting Termina with a custom built kernel:

chronos@localhost / $ sudo vmc start --kernel /mnt/stateful_partition/home/.shadow/${USER_ID}/mount/user/Downloads/bzImage termina

[LukeShortCloud]

Other kernel configurations available:

$ ls -1 arch/x86/configs/
chromiumos-borealis-vm-x86_64_defconfig
chromiumos-container-vm-x86_64_defconfig
chromiumos-jail-vm-x86_64_defconfig
i386_defconfig
tiny.config
x86_64_defconfig
xen.config

[LukeShortCloud]

Here are tested and working steps to build a crosvm virtual machine: https://github.com/LukeShortCloud/rootpages/issues/624#issuecomment-1012792080

[LukeShortCloud]

Re-opening to address more advanced crosvm usage including: (1) network and (2) graphics support.

[LukeShortCloud]

For networking, based off official documentation and real-world examples (as shown below), I've tested it and it's not working using an Arch Linux root file system and a chromeos-5.15 kernel.

     printf "crosvm/target/debug/crosvm run -s /run/user/{{.UID}}/crosvm.sock --host_ip=192.168.30.1 --netmask=255.255.255.0 --mac "AA:BB:CC:00:00:12" -r crosvm/rootfs.ext4 ----disable-sandbox -p 'ip=192.168.30.2::192.168.30.1:255.255.255.0:crosvm:eth0' ~/linux/arch/x86/boot/compressed/vmlinux.bin"

Once the virtual machine is booted, I have to manually set the IP address and default gateway (but it still does not work).

https://gist.github.com/markdryan/537eb922e31c6e47fbf75bf99831ec05

I also found this post that says that crosvm does not support ping so I've been using curl to test with.

https://spectrum-os.org/lists/archives/spectrum-devel/878s4i3ixs.fsf@alyssa.is/T/

UPDATE: Networking works with Debian 11. At least, I'm able to ping the default gateway that is automatically created (the --host_ip). I still cannot seem to reach out to the Internet.

UPDATE 2: I was able to get networking working on my Arch Linux hypervisor (have not tried on Chrome OS yet). I added the tap device to my bridge, set the bridge as the master, and then assigned an IP address on my home network along with the default gateway of my router. I can now reach out to the Internet! This is still only with a Debian 11 virtual machine.

$ sudo brctl addif br0 vmtap0
$ sudo ip link set vmtap0 master br0

Related documentation regarding tap and bridge devices: https://github.com/LukeShortCloud/rootpages/issues/637

UPDATE 3: I got Arch Linux working! In the virtual machine, I had to restart the network device for some reason. That removed the default gateway so I had to add that back, too.

$ ip link set enp0s4 down
$ ip link set enp0s4 up
$ ip route add default via 192.168.1.1

UPDATE 4: It is important to note that networking requires running with (1) root privileges or, at least, (2) the CAP_NET_ADMIN capability.

https://google.github.io/crosvm/running_crosvm/requirements.html

UPDATE 5: For Chrome OS, these commands need to be run. For wired ethernet, use eth0 instead of wlan0. Be sure to also change vmtap0 to match the tap device number.

$ sudo sysctl net.ipv4.ip_forward=1
$ sudo iptables -t nat  -A POSTROUTING -o wlan0 -j MASQUERADE
$ sudo iptables -A FORWARD -i wlan0 -o vmtap0 -m state --state RELATED,ESTABLISHED -j ACCEPT
$ sudo iptables -A FORWARD -o wlan0 -i vmtap0 -j ACCEPT

Then you can setup an IP on that vmtap0 device, add an IP in the same subnet mask inside the virtual machine, then use the vmtap0 IP address as the default gateway.

Ping works but other traffic (DNS, HTTP, HTTPS, etc.) does not.

https://google.github.io/crosvm/appendix/example_usage.html

UPDATE 6:

As an alternative to 5 for Chrome OS, this will accomplish the same results:

$ sudo iptables -t nat -A POSTROUTING -o wlan0 -j MASQUERADE
$ sudo iptables -A FORWARD -i vmtap0 -j ACCEPT

https://askubuntu.com/questions/885260/mapping-tap-interface-with-wifi-interface

UPDATE 7:

I got all the traffic working for a crosvm on Chrome OS! I had to remove any NAT rules in place for the vmtap# device.

View rules:

$ iptables -t nat -S | grep tap6

Delete any existing rules:

# Ingress rules.
$ sudo iptables -t nat -D ingress_default_forwarding -i vmtap6 -m socket --nowildcard -j ACCEPT
$ sudo iptables -t nat -D ingress_default_forwarding -i vmtap6 -p tcp -j DNAT --to-destination 100.115.92.6
$ sudo iptables -t nat -D ingress_default_forwarding -i vmtap6 -p udp -j DNAT --to-destination 100.115.92.6
# DNS rules.
$ sudo iptables -t nat -D redirect_default_dns -i vmtap6 -p tcp -m tcp --dport 53 -j DNAT --to-destination 100.115.92
$ sudo iptables -t nat -D redirect_default_dns -i vmtap6 -p udp -m udp --dport 53 -j DNAT --to-destination 100.115.92.13

[LukeShortCloud]

You can stop a virtual machine using the socket.

$ crosvm run -s /tmp/crosvm.sock ...

$ crosvm stop /tmp/crosvm.sock

https://www.reddit.com/r/PixelBook/comments/7zxz57/howto_boot_a_ubuntu_vm_using_crosvm/