Consider supporting Xen's Virtual Machine Introspection / DRAKVUF

[andrewdavidwong]

On 2016-11-04 02:35, Zrubi wrote:

Intrusion/virus detection inside the affected VM not really makes sense.

However newer Xen versions has a nice feature: https://wiki.xenproject.org/wiki/Virtual_Machine_Introspection

And already a real project using this feature: https://drakvuf.com/

That feature wound really make sense and would fit in Qubes philosophy pretty nicely.

[rootkovska]

There are lots of problems that VM introspection presents to security of a system like Qubes OS. Architecture-wise.

The biggest one is that even if we allow read-only access to memory and contexts of the VMs, we should still assume the Introspection VM would be able to establish a communication (covert) channel with any of the VMs in the system, including the least trusted on. This means the read-only Introspection VM is hard (or even impossible) to realize currently, especially on Xen. Now, because we need to assume that the (complex) logic inside such an Introspection VM will be exploitable, this means we're negating VM isolation in the system.

It's worth thinking if the above problem could be satisfactory solved by introducing many Introspection VMs and enforcing that each of them has access to only one AppVM?

[tklengyel]

Indeed, granting an introspection VM blank VMI access to all domains would by definition negate hard isolation between them. Also, as you say, with XSM it is very much possible to restrict the introspection access to specific domain(s) and also to run the VMI logic from dedicated VM(s). I already require users of DRAKVUF to deploy Xen with a custom XSM policy like that (see https://github.com/tklengyel/xen/commit/bfdf2068b1ddf4eb1cd8338375f6f76afe862f63).

[DemiMarie]

@rootkovska What if the inspection VM was a disposable VM, and the inspection code was written in a statically-typed, memory-safe language? (I know that memory safety does not prevent all bugs. That said, if I write a program in safe Rust, and it does not call (directly or indirectly) any functions that access the filesystem or spawn processes (which can be determined by a simple static inspection), then I can be fairly confident that it won’t do that at runtime, either).

[DrWhax]

Bitdefender released their HVMI library as open-source, which has Xen support from what I can see.

https://github.com/hvmi/hvmi

[mfc]

would anyone be interested in being a mentor on this and including it in the 2021 GSoC Project Ideas list?

[tlaurion]

Crosslink to user attempted discussion on the forum: https://forum.qubes-os.org/t/memory-introspection-thread/10020

[3hhh]

Somewhat related: #7831

[scrubbedha]

I wrote a guide on how to use DRAKVUF with Qubes r4.2.0-rc5 here: https://github.com/scrubbedha/Qubes_Drakvuf (requires patching qubes-vmm-xen)

[tklengyel]

Fantastic!

Couple comments: dom0_mem=2048M,max:2048M dom0_max_vcpus=2 dom0_vcpus_pin=1 is indeed optional but generally speaking disabling memory ballooning in dom0 is good practice, as well as pinning vcpus. Setting max vcpu to something reasonable will also make dom0 not use CPUs it doesn't need, giving more performance to the other cores. But it will work just fine without any of these set.

force-ept=1 was needed on some hardware to enable EPT, people ran into issues where it wasn't getting enabled, so having this in the default command line is just easier even if its unnecessary on most platforms where EPT gets enabled just fine.

ept=ad=0 is required but not because security reasons but because there is a bug with altp2m + PML that crashes Xen. Disabling AD disables PML as well. Neither AD nor PML is needed for anything other then live migration so turning them off is no issue and may actually give a little performance benefit.

hpet=legacy-replacement is due some newer hardware having quirks with their timing and Xen wouldn't boot unless this setting is used. I don't quite understand the issue but it's easier to have this as the default then help everyone figure out what option they need when they run into Xen not booting.

sched=credit is optional, you can actually use any scheduler you want

[tklengyel]

Also, Scenario 4: Abuse of altp2m views doesn't apply. There are different types of HVMPARAMs, and we specifically use one that's inaccessible to the guest (ie. external), see https://xenbits.xen.org/gitweb/?p=xen.git;a=blob;f=xen/include/public/hvm/params.h;h=a22b4ed45d2eb2dcaf61db9bef1f83b80d426a40;hb=HEAD#l252

[tklengyel]

Also, if you are building DRAKVUF with meson you don't need to run autoreconf -vi.

BTW if you have the altp2m setting configured in the cfg your skips the get|set HVM_PARAM_ALTP2M check patch is fine, what DRAKVUF is doing is redundant, we just had many users forgot to set the altp2m param in the domain config so we double check it when starting. I would recommend just adding a command line option to DRAKVUF, --skip-altp2m-check to not bother if you know the config already set it.

[scrubbedha]

Awesome feedback! Thanks heaps for elucidating on the --skip-altp2m-check argument as I was previously unaware of it!

I've removed scenario 4, the superfluous autoreconf step, and did my best incorporate all of your other comments on the Xen cmdline options into the guide citing this issue.

Really appreciate your work on the DRAKVUF project (as I use it daily for work) and I earnestly hope it gains traction within the Qubes community.

[DemiMarie]

Worth noting that nested HVM in Xen is exploitably broken: my understanding is that an L2 guest can break out to (host) dom0 (so very bad). It really should be build-time disabled in Qubes OS, but I don’t think Xen supports that yet. PV-in-HVM should work fine, though.

@tklengyel: Does the XSM policy you use allow the introspection domain to use any hypercalls that are covered by XSA-77? If so, then these hypercalls would need to first be exempted from XSA-77, which means fixing the existing holes and persuading the Xen Security Team that they indeed have been fixed.

[tklengyel]

Sorry, I meant you could add --skip-altp2m-check, then you wouldn't have to patch out-of-tree ;)

As for XSA-77, for the most part the stuff we need is monitor_op, memory_op, hvm_op and opening event channels. Domctl is required to pause the whole domain, control vmtrace, etc. but it could conceivably be moved into a helper tool in dom0 that could restrict the requested target of the hypercall to intended domains. I've done that in the past through xenstore so the domain would be created from dom0 but then controlled from the drakvuf control domain. Of course the better solution would be to check with XSM enabled if the hypercalls actually can cause issues and fix it. It's been a while but with the XSM policy linked to in the setup notes I couldn't run drakvuf/introspection on VMs not having the right label but of course I didn't do an exhaustive check of all hypercalls to see if they could cause issues.

[DemiMarie]

@tklengyel Do you think you can fix the needed APIs and convince upstream to exempt them from XSA-77? One could proxy them but that’s an ugly hack I’d prefer to avoid.

[tklengyel]

@DemiMarie Unlikely I'll have time to do that any time in the foreseeable future