not a bug - attempt to document notes on running the entire BlueOS stack as a VIRTUAL Raspberry Pi

[davidbuzz]

Bug description

host OS is ubuntu laptop 64bit

goal: to make a VM that runs BlueOS in its entirely good enough that it doesn't know its running on a VM...

mkdir ~/BlueOS
cd BlueOS
wget https://github.com/bluerobotics/BlueOS-docker/releases/download/1.0.1/BlueOS-raspberry.zip
unzip BlueOS-raspberry.zip
ls -l blueos.img
# we'll re-size it larger , so keep the original incase that doesn't work.
cp blueos.imb blueos2.img
qemu-img resize blueos2.img +16G

we need these..

sudo apt install -y qemu-system-arm
sudo apt install -y virt-manager

sudo virsh --connect=qemu:///system net-start default

start qemu/kvm gui with root privs or it didnt work for me..

sudo virt-manager

get qemu-ready kernel and dtb - see url at bottom..

git clone https://github.com/dhruvvyas90/qemu-rpi-kernel

look at the current 'default' network..

sudo virsh net-dumpxml default

make a new file netdump.xml containing tweaks[so the VM boots as 192.168.2.2 later]:

<network>
  <name>default</name>
  <uuid>0b43d6e7-26b8-4547-a6f5-1eb1ec2bc3bf</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:50:e0:9d'/>
  <ip address='192.168.2.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.2.2' end='192.168.2.2'/>
    </dhcp>
  </ip>
</network>

destroy stock network:

sudo virsh net-destroy default

re-create from xml file:

sudo virsh net-create netdump.xml

make a new VM from the command-line [ close-and-re-open virt-manager afterwards to see it in gui]

sudo virt-install --name rpios --arch armv6l --machine versatilepb --cpu arm1176 --vcpus 1 --memory 256 --import --disk blueos2.img,format=raw,bus=virtio --network bridge,source=virbr0,model=virtio --video vga --graphics spice --boot 'dtb=qemu-rpi-kernel/versatile-pb-buster.dtb,kernel=qemu-rpi-kernel/kernel-qemu-4.19.50-buster,kernel_args=root=/dev/vda2 panic=1' --events on_reboot=destroy --osinfo detect=on,require=off

re-start gui and hope it worked..

sudo virt-manager

use GUI to start the VM labeled 'rpios' we made above. did your VM boot, mine did.

in the GUI, if you 'click' the VM's console and get your keystrokes locked into it, press left-ctrl and left-alt together.

[in the VM, you can login as user 'pi' and password 'raspberry'], and you'll see its hostname is 'blueos']

resize VM's filesystem to match the enlarged disk..

sudo raspi-config 
-> advanced options -> expand filesystem -> enter -> .... ->ok -> finish. reboot now ? -> yes. 

use GUI to re-start the VM labeled 'rpios' we made above. did your VM boot, mine did.

at this point it might be pingable at 192.168.2.2 if the above xml worked... if it didn't, then login to console again [pi/raspberry], then type sudo ifconfig to get the VM's current ip address.

You can now hopefully ping its IP address from another terminal window on your host computer... if you CAN ping it, then you can also 'ssh pi@192.168.2.2' and ssh into it that way.. so now you have a console AND ssh access working. SSH access is better because u can cut-n-base into it more easily.

into your 'ssh' window that goes to the VM cut-n-paste these one at a time.. and hope they work... pi@blueos:~ $ ls

# get blueos install script...
wget https://github.com/bluerobotics/BlueOS-docker/blob/master/install/install.sh
chmod 755 ./install.sh

#since this VM is 'armv6l' architecture, but the install script doesn't list that as supported, we need to edit line 63 of the install.sh that we just downloaded before we use it...  [ change "armv7l" to "armv6l" ] 
vim ./install.sh

# this file is expected by rfkill, which the above script runs.
sudo touch /dev/rfkill
# this allows docker to work..
sudo update-alternatives --set iptables /usr/sbin/iptables-legacy
sudo update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy

# install latest dockerOS -
sudo ./install.sh --skip-board-config

if it doesn't complete, try rebooting and running it a second time.

sudo touch /dev/rfkill
sudo ./install.sh --skip-board-config

... sometime later here's the last bit of the log u should see from the above command....
...
Configure hostname to blueos
Installation finished successfully.
You can access after the reboot:
- The computer webpage: http://blueos.local
- The ssh client: pi@blueos.local
System will reboot in 10 seconds.
Connection to 192.168.2.2 closed by remote host.

waitf or it to reboot, again, and be pingable... and then try visit.. http://192.168.2.2/

references: https://linuxconfig.org/how-to-run-the-raspberry-pi-os-in-a-virtual-machine-with-qemu-and-kvm

Steps to reproduce

see above.

Primary pain point(s)

No response

Prerequisites

• [X] I have checked to make sure that a similar request has not already been filed or fixed.

[patrickelectric]

@davidbuzz We currently don't have x86 binaries docker versions available at the moment, but should be easy to modify the project to provide it. From what I could read, it could make it running in a VM much more simpler.

With your permission, can we add this in our wiki ?

[davidbuzz]

add this in our wiki ? sure, but its highly experimental, and i may not have got everything right ,so its worth someone else trying it. [maybe flag it as 'experimental' or something in the wiki?]

[davidbuzz]

for the record... I have the .img booting in qemu, and i have the vm starting, and I have the bueos scriot installing, but I don't actually have blueos working after that. i had docker issues to do with the architecture differences between arm v6 and v7 architecture/s .. i tried 'armv7l' instead of v6l as v6l was an unknown architecture in the blueos install script, but then docker gave weird warnings, so i think the way its documented above is closer to accurate, but the install.sh needs to be adjusted so v6l is one of the "supported architechtures" ?

eg:

...
docker.io/bluerobotics/blueos-core:master
WARNING: The requested image's platform (linux/arm/v7) does not match the detected host platform (linux/arm/v6) and no specific platform was requested

[davidbuzz]

current issue: pi@blueos:~ $ sudo docker start blueos-bootstrap Error response from daemon: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: bpf_prog_query(BPF_CGROUP_DEVICE) failed: function not implemented: unknown Error: failed to start containers: blueos-bootstrap

... which i believe the internet says would be fixed by upgrading from a 4.x kernel to a 5.x kernel.

[Williangalvani]

@davidbuzz it seems like docker as a whole is failing to run in the vm?

perhaps we should start building x86_64 images again, they should build much faster than arm, too...

[davidbuzz]

better version of QEMU setup completely revised/alternative method that uses -aarch64 variant of qemu, AND uses the dedicated 'raspi3b' machine type, AND uses vanilla/stock 'bullseye' VM , AND most importantly, it uses a 5.x linux kernel . I think this is a much cleaner setup, but i didn't start with the 'blueos' SD-card image, and it doesn't use virt-manager

mostly from... https://stackoverflow.com/questions/61562014/qemu-kernel-for-raspberry-pi-3-with-networking-and-virtio-support

IMPORTANT do this all as root, sudo isn't enuf for some bits.

wget https://downloads.raspberrypi.org/raspios_lite_arm64/images/raspios_lite_arm64-2022-04-07/2022-04-04-raspios-bullseye-arm64-lite.img.xz
unxz 2022-04-04-raspios-bullseye-arm64-lite.img.xz

temporarily mount the disk image at the ./boot folder, so we can copy the kernel and dtb out of it, set the 'pi' password as 'raspberry', and ensure usb-ethernet drivers are loaded..

mkdir boot
mount -o loop,offset=4194304 2022-04-04-raspios-bullseye-arm64-lite.img boot
cp boot/bcm2710-rpi-3-b-plus.dtb .
cp boot/kernel8.img .

see https://raspberrypi.stackexchange.com/questions/137915/standard-passwort-not-accepted-headless-setup echo 'pi:$6$6jHfJHU59JxxUfOS$k9natRNnu0AaeS/S9/IeVgSkwkYAjwJfGuYfnwsUoBxlNocOn.5yIdLRdSeHRiw8EWbbfwNSgx9/vUhu0NqF50' > boot/userconf

edit boot/cmdline.txt to load the usb ethernet drivers: modules-load=dwc2,g_ether from console=serial0,115200 console=tty1 root=PARTUUID=0ee3e8a8-02 rootfstype=ext4 fsck.repair=yes rootwait quiet init=/usr/lib/raspi-config/init_resize.sh thus: echo "console=serial0,115200 console=tty1 root=PARTUUID=0ee3e8a8-02 rootfstype=ext4 fsck.repair=yes rootwait modules-load=dwc2,g_ether quiet init=/usr/lib/raspi-config/init_resize.sh " > boot/cmdline.txt

umount boot

resize and adjust format of disk image

qemu-img convert -f raw -O qcow2 2022-04-04-raspios-bullseye-arm64-lite.img  2022-04-04-raspios-bullseye-arm64-lite.qcow2
qemu-img resize 2022-04-04-raspios-bullseye-arm64-lite.qcow2 16g

boot it, noting that the qemu 'console' can be exited by typing 'ctrl-a' [then releasing it], then pressing 'x'.

qemu-system-aarch64 -m 1024 -M raspi3b -kernel kernel8.img \
  -dtb bcm2710-rpi-3-b-plus.dtb -sd 2022-04-04-raspios-bullseye-arm64-lite.qcow2 \
  -append "console=ttyAMA0 root=/dev/mmcblk0p2 rw rootwait rootfstype=ext4" \
  -nographic -device usb-net,netdev=net0 -netdev user,id=net0,hostfwd=tcp::5555-:22

if not done above, edit your /boot/cmdline.txt file to add modules-load=dwc2,g_ether to /boot/cmdline.txt after rootwait.

login to VM as user->'pi' and password -> 'raspberry' [as set above].

inside vm, run sudo raspi-config or as root and enable ssh. also do the 'Expand Filesystem' and reboot the vm, or there won't be enough room on the virtual disk.

get in wth: ssh pi@localhost -p 5555

inside the vm.. you can execute whatever commands you want now, such as trying to install blueos etc.

[cut-n-paste to terminal, no need to do it over ssh, but you can too]

sudo su -c 'curl -fsSL https://raw.githubusercontent.com/bluerobotics/blueos-docker/master/install/install.sh | bash'

get out with... [ this tests that networking in the vm is working enough to get to 'the internet'] ping google.com

[davidbuzz]

... this latest setup, i was able to get BlueOS to install, and docker to startup, but after the qemu VM is running, i was unable to ssh into it any more, as it was so busy keeping all these services running , that ssh didn't work after that. i was hoping to be able able to either add port 443 to the list of forwarded ports like ssh, or tunnel through the ssh session to the internal port 443, allowing me to get to the web interface that way.... neither of these have i achieved at this time. I've given up for now, but the principal is sound.

[davidbuzz]

I won't be taking this work any further. if u feel it needs to go in the wiki, you can, otherwise its fine to close this

[patrickelectric]

Hi @davidbuzz thanks for checking this out! To update you, we just created amd64 and armv8 dockers (can run on apple silicon)