Open marmarek opened 3 years ago
It was discussed briefly off issue that a debug ISO would miss the point, where having a different xen-debug source tree, clearly differentiable from the Xen standard target, would make it easier to maintain and deploy from testing repository when needed.
That Xen + linux emergency kernel to be kexec'ed + kexec dependencies for dom0 should make its way into a grub clearly separated entree for users to select and as wished end result, be able to extract needed information from the memory dump (saved under /boot? which was why I talked about a debug ISO at the first place, so that /boot is big enough to dump memory content? Saved under swap? As Marek stated, having the emergency kernel dump into root would require LUKS key to be passed, which in most user cases is not possible.)
Thoughts on better approaches?
I'm inclined to help into this and maybe a specific ISO_FLAVOR could be weekly built.
Dumping into /boot
has the obvious problem that it is unencrypted. Could some form of asymmetric encryption be used?
I'd rather embed LUKS key into the initramfs that is loaded to be kexec'ed. Such initramfs would be sensitive then, so storing it in /boot
is out of the question. Not sure what's better:
/root
? /var/lib/something
?)Since Argon used by LUKS2 needs more RAM than its useful to assign for the crash kernel (*), I consider injecting the actual LUKS master key. This may need some initramfs adjustments.
Alternative option is to dump into some USB stick. But that's way less reliable (especially if kexec'ed Linux won't undo IOMMU setup done by Xen).
(*) This memory needs to be reserved at the boot time, and cannot be used for anything else - if you need 1.5GB for the crash kernel, you'd have that much less for the actual system.
- inject LUKS key at the load time
This definitely seems like the better option. Having the key be in a file on disk risks it leaking, which would be bad. I would much prefer for it to remain entirely in-memory.
If neither the thunderbolt nor expresscard are available and the machine is recent enough to have one or more M.2 slots, and you don't mind sessions with the laptop case off (or potentially ribbon cables dangling out of the case), there's this clever solution. After all, an M.2 device is just a pcie device:
Excellent idea @brendanhoar! After few tweaks and hitting unrelated bug, I got this working :) Debugging suspend issues should be much easier (for me, at least) now.
I consider injecting the actual LUKS master key.
I wanted to extract the master key from the running kernel, but this is actually impossible on R4.1, because cryptsetup loads it into kernel keyring as a "logon" key - which can be read only by the kernel. I mean, technically there surely is some hack to extract it, but not a proper API. This means, we either need to store a keyfile somewhere on the disk, or ask the user for disk passphrase the second time during boot (or more precisely: when loading crash kernel).
I consider injecting the actual LUKS master key.
I wanted to extract the master key from the running kernel, but this is actually impossible on R4.1, because cryptsetup loads it into kernel keyring as a "logon" key - which can be read only by the kernel. I mean, technically there surely is some hack to extract it, but not a proper API. This means, we either need to store a keyfile somewhere on the disk, or ask the user for disk passphrase the second time during boot (or more precisely: when loading crash kernel).
Patch cryptsetup?
Patch cryptsetup?
No, I don't want to weaken security feature for everyone, just to make debugging easier for few. I'd rather consider saving keyfile somewhere on the root fs - which would affect only those who opt-in for debug setup.
Patch cryptsetup?
No, I don't want to weaken security feature for everyone, just to make debugging easier for few. I'd rather consider saving keyfile somewhere on the root fs - which would affect only those who opt-in for debug setup.
Serious question: how is this security feature useful in Qubes OS? Once someone has code exec in dom0 it is game over anyway.
Since https://github.com/QubesOS/qubes-vmm-xen/pull/138 is merged, here is how to use it to get Xen's console:
/sys/bus/pci/devices/*/dbc
file in sys-usb, if it's there and says "disabled", it's okay.dbgp=xhci,share=yes console=vga,xhci
to Xen's cmdline (either in grub menu directly - the line with multiboot2 /xen-*.gz
, or in /etc/default/grub
in GRUB_CMDLINE_XEN_DEFAULT
option). If your system have multiple USB3 controllers, you may need to specify it explicitly - for example dbgp=xhci@pci00:14.0,share=yes console=vga,xhci
- see lspci
command./dev/ttyUSB0
on the second computer - it should appear when Xen starts.picocom -b 115200 /dev/ttyUSB0
(or similar program - minicom
, cutecom
or even cat
works).So if I understand share=yes
correctly, the device can even be assigned to sys-usb
at the same time? That's awesome!
Too bad that my T530 apparently has no dbc/debug port... -_-
So if I understand
share=yes
correctly, the device can even be assigned tosys-usb
at the same time? That's awesome!
It can be, but doing so allows sys-usb
to take control of the system.
On 10/31/22 09:07, Demi Marie Obenour wrote:
So if I understand
share=yes
correctly, the device can even be assigned tosys-usb
at the same time? That's awesome!It can be, but doing so allows
sys-usb
to take control of the system.
Ok, so a dedicated dom0 USB device/port is more optimal after all...
Btw is serial port debugging still supported and will it remain supported?
I'm asking because I don't see CONFIG_SERIAL_8250_CONSOLE
here or anywhere else and those Express Card UART adapters for Thinkpads aren't too cheap to try either... (40-60 EUR)
So if I understand
share=yes
correctly, the device can even be assigned tosys-usb
at the same time? That's awesome!
As Demi noted, this has some risks associated. If you want, you can use share=no
, in which case only Xen will have access to the controller (and sys-usb will likely fail to start). Or you can use share=hwdom
(or no share
option at all), in which case dom0 will have access to the controller and not sys-usb (but if you use sys-usb, then dom0 avoids touching the controller anyway).
Btw is serial port debugging still supported and will it remain supported?
Yes. But it's increasingly hard to have an actual serial port in a laptop.
I'm asking because I don't see
CONFIG_SERIAL_8250_CONSOLE
here or anywhere else and those Express Card UART adapters for Thinkpads aren't too cheap to try either... (40-60 EUR)
I think you're confusing Linux config with Xen config.
I got dom0 kernel logs via UART running by adding GRUB_CMDLINE_LINUX="$GRUB_CMDLINE_LINUX console=ttyUSB0,115200"
to /etc/default/grub
.
Do these include the Xen logs?
The Xen instructions didn't work for me (I tried GRUB_CMDLINE_XEN_DEFAULT="$GRUB_CMDLINE_XEN_DEFAULT console=com1 com1=115200,8n1 loglvl=all guest_loglvl=all"
). I fear it's because the PCI Express Card apparently is a USB hub which just runs the RS232 port as USB client which Xen doesn't support, but the Linux kernel does. Of course such details aren't documented by the Express Card manufacturer...
Yeah the pci card/expresscard needs to be a real UART serial port, not a usb port + usb serial device.
Even then I think @marmarek ran into some issues getting Xen to use a real one, IIRC.
B
On laptops lacking physical serial console, collecting logs of a crashed Xen (or dom0 kernel) is hard. One method is to use kexec to extract them from the RAM. Currently this requires several non-trivial changes to the system. This issue is to ease preparing system for such debugging.
Details from the original issue:
@marmarek @fepitre From this comment:
This exact QubesOS 4.1 ISO deployment (debug iso with all above dependencies preconfigured) would help to debug so many corner cases for QubesOS project, including this one? Instruct a willing tester to install QubesOS on spare drive and report results without having to go through the burden of customizing such testing testbed? Should that be discussed in a separate issue (with more details on doing this manual and then automate the process, maybe?)
@marmarek / @fepitre : Random thought, but if there is enough space affected to /boot in the default partitition scheme of that debug iso, dumping said logs from kexec'ed linux kernel to dump memory under /boot would work around a lot of the complexity added dumping needed logs and go faster into having what logs we need, easily, from an additioanl boot of said debug ISO in Read Only rescue mode from boot options.
Originally posted by @tlaurion in https://github.com/QubesOS/qubes-issues/issues/6066#issuecomment-895447248