bottlerocket-os / bottlerocket

An operating system designed for hosting containers
https://bottlerocket.dev
Other
8.68k stars 512 forks source link

Add virtio_scsi disk controller for QEMU virtual machines #3045

Closed tommie closed 1 year ago

tommie commented 1 year ago

What I'd like: To run Bottlerocket OS on Hetzner Cloud.

Device type (e.g. network interface, disk controller): Disk controller

Device vendor: QEMU

Device model: N/A

Driver used on other Linux distribition: virtio_scsi / CONFIG_SCSI_VIRTIO

Any alternatives you've considered:

Looked at https://github.com/bottlerocket-os/bottlerocket/issues/842#issuecomment-628762310

I can reproduce the error in VirtualBox by using a LsiLogic, BusLogic or virtio-scsi controller instead of the default IDE/ATA. I haven't tested a patched Bottlerocket with virtio_scsi, so I can't say if it's enough, but given the list in the linked comment, it's the only piece missing.

tommie commented 1 year ago

Having had a second look...

The driver is actually being built, but only as a module. Since there is no initrd, I don't think there is a way for me to use the module before verity has brought up the root filesystem. Would it be possible to extend bootconfig to a simple initrd that loads modules?

Another option: if the kernel were to use multiboot, GRUB could be made to preload modules from the private partition and load it that way. AFAIK, this is not possible with the linux loader. This would require optionally reading a GRUB config snippet from the private partition, similar to the kind of hook bootconfig.data is for the kernel.

stmcginnis commented 1 year ago

Hi @tommie. Thanks for digging in to this.

This is typically handled by a bootstrap container. From there you would be able to load additional kernel modules.

An example of loading something similar is discussed here: https://github.com/bottlerocket-os/bottlerocket/issues/2409#issuecomment-1440875158

Hopefully that gives enough context to see what needs to be done.

tommie commented 1 year ago

The problem is the root file system can't be mounted (/dev/mapper/dm-0 never becomes ready), so a boot container won't help.

I see the same kernel messages as https://github.com/bottlerocket-os/bottlerocket/issues/842#issuecomment-597837874

bcressey commented 1 year ago

Are you using one of the metal-* variants? We'd need to add this to config-bottlerocket-metal there if so.

@foersleo what do you think?

When Secure Boot support arrives after #2501, we wouldn't want to extend the GRUB config, since that could be used to bypass the intended protections. #2729 already disabled any unexpected use of an initrd. So I'd prefer that the kernel just have the required modules built in.

tommie commented 1 year ago

@bcressey Yes, sorry, I'm using the metal version. I had a look at "vmware" too, but it doesn't have any kernel config overrides, so I ignored that.

Including virtio_scsi in the kernel makes sense to me (but I might be biased in this thread ;).

Since I like modularity, I was also experimenting with an initramfs (up until I realized the CONFIG_INITRAMFS_FORCE was in effect...) I made a firstdog init that simply loads modules from a directory and does switch_root. Combining SB with forced initramfs and the modularity of modules seems tricky, even if module signing is mandatory.

However, we could

Both of these require boot FS to be writeable, which isn't very nice.

foersleo commented 1 year ago

I think the case for building in the module is solid, considering the options seem to be a lot more headache for comparably low advantages.

I will prepare a PR and have a look if that pulls in any unforeseen dependencies. Going by the config definition I would not expect anything big that would be pulled in though.

tommie commented 1 year ago

Thank you!