No virtualization is available in a HVM qube

SurinameClubcard commented 6 years ago

Qubes OS version:

R4.0

Affected component(s):

Depending on the outcome of a risk and threat assessment, it might be allowable to enable nested virtualization in HVM mode. E.g., I'd like to run GNS3 in a HVM based Linux distribution. Without proper support for nested virtualization, that won't work / perform.

Steps to reproduce the behavior:

Install your favorite Linux distribution in a HVM-based qube.
Install VirtualBox in that HVM.
VirtualBox will complain that VT-x/AMD-V hardware acceleration is missing.

Expected behavior:

Hardware acceleration should be availabe in HVM.

Actual behavior:

Currently I'm unable to run GNS3 in a Linux based HVM with acceptable performance.

General notes:

I will not participate in a flame war on nested virtualization. ;-)

I'm aware of risks introduced by nested virtualization. In my situation, I've done the assessment and I accept these risks.

Related issues:

I'm more than happy to patch Qubes myself to make this possible. E.g., there is https://groups.google.com/forum/#!msg/qubes-devel/UzO0BsIfIow/wWjpd3IPAgAJ which already contains a patch. Is it really that simple to enable it? Or did things change massively between then and now.

There is also https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen#Current_status, which I don't understand, because it looks to me that newer versions of Xen won't work but older versions do?

Maybe I greatly underestimate the complexity of this matter ...

marmarek commented 6 years ago

There is also https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen#Current_status, which I don't understand, because it looks to me that newer versions of Xen won't work but older versions do?

It is exactly as you read it - nested virtualization in Xen is in "preview" state for a long time, and it doesn't seems to be really maintained.

This is also the reason why we don't provide easy way to enable it - we choose to spend time on feature that actually improve security and stability, not do the opposite.

If you still want to enable, see /usr/share/qubes/templates/libvirt/xen.xml and https://dev.qubes-os.org/projects/core-admin/en/latest/libvirt.html

andrewdavidwong commented 6 years ago

Closing as "won't do." If anyone has a new reason for why this should be done, please leave a comment, and we'll be happy to take another look. Thank you.

SurinameClubcard commented 6 years ago

I understand the "won't do" closure and I fully appreciate the fact that you're spending time on features that improve security and stability. To me, Qubes is about stability and security, but also about freedom and flexibility. That said, nested virtualization does serve a purpose. The possibility to run ESXi in ESXi is awesome for some use cases, especially for teaching purposes. Nesting Qubes in Qubes also seems so logical, for example, to test-drive a new version. Using Vagrant for getting OWASP WebGoat up-and-running springs to my mind. My original GNS3 example. Docker. I'm more-or-less seeing Qubes as an operating system for operation systems. If Qubes would be able to support nested virtualization, it would be a more complete product. And I currently don't see any threats that will make the outcome of my risk assessment a negative one. But YMMV ofc.

If I understand marmarek's reply correctly, there is no need to patch? The functionality is already in Xen and the only thing to do is enabling it to the tools? I'd like to give it try anyway... ;-)

misterzed88 commented 6 years ago

I actually tried this feature by enabling VMX support in a template for a HVM VM (based on an Ubuntu template). When the VM is started the system crashes and reboots without any further notice (no kernel panic, etc). The same behaviour was experienced on three different machines with different hardware (all based on fairly new Intel CPUs).

As a reference, I tried the same scenario (nested HVM with Xen), on Ubuntu 16, where it worked fine. Even though this case used a different Xen version (4.6.5 instead of 4.8.4) I still suspect that the Qubes packaging/use of Xen may be the culprit (perhaps any of the custom patches?).

I know that this feature is not a priority for you, but still, considering that there are some relevant use cases for it, and a crash may be a symptom of a serious bug or security vulnerability (potentially triggable without nested hvm support) would this be something worth investigating further? If you think so, I can try to help pinpointing the problem further, even setting up Xen debug output over serial console, if necessary.

marmarek commented 6 years ago

We have specifically compiled out some Xen features, but I don't think nested HVM is one of them, so it is probably a bug somewhere in Xen (either upstream, or our patches). As you can see in Xen wiki, nested HVM support is rotting, so it may be that it worked on 4.6, but not on 4.8.

Anyway, if you can get serial console output of that crash, it may be useful to find what is going on.

jpouellet commented 5 years ago

Can confirm that my xen 4.8.5 host resets when trying to boot a VM with nested HVM enabled.

marmarek commented 5 years ago

I haven't tried but it may be different in R4.1 with Xen 4.12. Early testing images can be downloaded from OpenQA, for example at https://openqa.qubes-os.org/tests/2777#downloads

ahimta commented 5 years ago

I need this for Android development because the Android Emulator without nested virtualization is:

Too slow that it's almost unusable and you can spend 10+ minutes on something that would take <2 minutes with virtualization.
Google is dropping support for this and only images for older Android versions (7.1.1-) are supported for non-virtualization emulator.

I can use a physical phone instead of an emulator but there are some major drawbacks:

Can't test on as many Android versions. It's easy to test on all Android versions with the emulator (typically 5.0.1+).
It can make the feedback loop much longer because the emulator is faster to test and closer to the IDE, etc...

It would be very valuable if this could be resolved because a developer may be forced to switch to another OS if the Android Emulator makes a big difference for them.

I see some people mentioning Docker but I can attest to both Docker and Docker Compose working flawlessly on a fedora-29 AppVM.

Thanks for all your great work and please tell me if you need to know more about this:smiley:.

brendanhoar commented 5 years ago

I went ahead and installed the 20190701 R4.1 early build @marmarek linked to (using XEN 4.12). Seems to boot up and work fine in light testing on thinkpad x230 and x230 tablet. If I have a chance Friday, I’ll plan to install an HVM or two and virtual box in the HVMs and see what happens.

andrewdavidwong commented 5 years ago

Since this was closed as a won't do for security reasons (see https://github.com/QubesOS/qubes-issues/issues/4104#issuecomment-406080934), I will refrain from reopening unless I hear from @marmarek.

SurinameClubcard commented 5 years ago

Since this was closed as a won't do for security reasons (see #4104 (comment)), I will refrain from reopening unless I hear from @marmarek.

Understood, yet, there seems to be more and more demand every day... ;-)

afk11 commented 5 years ago

I'm also here because I have a need to virtualize an Android 7+ build which all require VT-x / SVM..

justintaft commented 5 years ago

I'm also here because I have a need to virtualize an Android 7+ build which all require VT-x / SVM..

Haven't tested, but you can try running android x86. https://www.android-x86.org/

ahimta commented 5 years ago

Just had an issue where snapcraft fails because it uses multipass which uses KVM:sweat_smile: but luckily they have a Docker guide that can workaround this and probably without any serious limitations:smiley:.

I think nested virtualization is probably far off in the future for Qubes OS and for good reasons. One thing that would be very helpful and may be even necessary is to document the nested virtualization and some workarounds for some usecases (e.g: snapcraft/snap):sweat_smile:.

sentient6 commented 5 years ago

Following the steps @marmarek documented in the first place, nested virtualization works for me in a HVM on R4.0 (xen 4.8.5-10, libvirt 3.3.0-9): after booting the HVM (no crash, no hickups, no nothing), vmx is available => "works for me". I wonder which future update of Xen will crash it; therefore, I appreciate Qubes R4.x was designed to allow for switching to a different hypervisor.

Giszmo commented 4 years ago

The incapacity to out-of-the-box run an AVD (Android Virtual Device) next to my Android Studio to develop Android apps and test them is currently blocking me from jumping onto Qubes. Maybe the capacity to run an AVD could be provided as a VM on dom0 like all other VMs with the features to connect to it from the VM that runs my Android Studio?

(I'm not yet familiar with Qubes. Maybe that is already what is possible?)

LockedThread commented 3 years ago

I'm going to have to stop using Qubes, sadly, because it doesn't support Android Development.

DemiMarie commented 3 years ago

@LockedThread you might be able to fall back to software emulation using QEMU. It will be slow, but on the other hand, many Android phones are slow, so if your app has acceptable performance in the emulator, you can be confident it will perform well in practice.

LockedThread commented 3 years ago

@LockedThread you might be able to fall back to software emulation using QEMU. It will be slow, but on the other hand, many Android phones are slow, so if your app has acceptable performance in the emulator, you can be confident it will perform well in practice.

At that point I feel like buying an Android phone and passing the USB connection to my computer for app testing would be more worth it.

DemiMarie commented 3 years ago

@LockedThread That is also an option, and indeed one that has many advantages even if virtualization was available.

LockedThread commented 3 years ago

@DemiMarie Alright, I really love Qubes OS but there's some things that truly frustrate me lmao.

SurinameClubcard commented 3 years ago

I use(d) Qubes as an operating system for operating systems. Not having nested VMs was a deal breaker. I changed to LMDE4 and added Proxmox on top of that. Completely different in terms of architecture but great for my goals. FYI: LMDE is Linux Mint based on Debian Buster and Proxmox is a very rich virtualization environment, including Linux Containers (LXC). Installing Proxmox on top of LMDE is no different compared to installing Proxmox on top of Buster, which is very well documented. If you're looking for flexibility without the security constrains, you might want to look into this combination.

andrewdavidwong commented 3 years ago

FYI, there's also a separate Android-related issue: #2233

Giszmo commented 3 years ago

@andrewdavidwong thanks a lot. It's certainly the more hopeful issue for us Android devs, being still "open" :D

icequbes1 commented 3 years ago

Have not tried the Android emulator, but I've had some success with nested virtualization as of about a month or two ago - see my last post in this forum thread 0 - KVM+libvirt in a Fedora HVM qube was able to boot Windows, but no extensive testing was performed.

DemiMarie commented 3 years ago

@icequbes1 note that nested virtualization is not security supported by Xen upstream and has “known DoS and suspected privilege escalation vulnerabilities” (to quote an earlier XSA).

icequbes1 commented 3 years ago

It seems every suggestion I have in Qubes turns out to be a security vulnerability ;)

If nothing else, my comments were just to indicate "the machine didn't crash" compared to earlier reports, but yes, buyer beware due to the identified security concerns.

drewwiens commented 2 years ago

TL;DR don't be like me and try this, it's totally broken

Apologies for posting on an old thread but for anyone else who comes along and might otherwise waste a day on this, I want to report that I was able to start HVM qubes in R4.1.1 with vmx on and kvm-ok reporting that kvm is available, however it was still not enough for me to run any of these:

VMWare Workstation Player (crashed on VM boot)
VirtualBox (wouldn't install; reported that xen isn't supported)
Docker Desktop for Linux (wouldn't open after install; Docker Desktop requires a VM while Docker Engine does not)

I persisted because I had found evidence that someone claimed they got Qubes OS running in an HVM qube in R4.1.

Sorry I don't have qubes running right now to check the exact syntax but this is the gist of how I was able to start the HVMs with vmx on and kvm-ok:

Setting a fixed ram amount/disabling memory management on the hvm qube in its Qubes Settings for that qube -- this was mandatory, the qube crashed with it on but did start with it off -- and
Not modify the existing <cpu mode='host-passthrough'> in xen.xml, but instead turning its if-statement into an if-elif-statement where the first if block matched if hvm and set <feature name='vmx' policy='require' /> and then the elif was just the existing if statement. The machine wouldn't boot at all by modifying cpu block presumably because it matches other VM types not just HVM and <feature name='vmx' policy='require' /> made those other VM types crash.

DemiMarie commented 2 years ago

@marmarek: I wonder if this can be build-time disabled in Xen, so that any attempt to enable it is a no-op. Nested virtualization in Xen is just broken.

SurinameClubcard commented 2 years ago

If I recall correctly, Qubes was designed to allow for switching to different hypervisors?

drewwiens commented 2 years ago

If I recall correctly, Qubes was designed to allow for switching to different hypervisors?

I've heard that, but haven't seen any steps on how to do it

andrewdavidwong commented 2 years ago

If I recall correctly, Qubes was designed to allow for switching to different hypervisors?

Yes: https://blog.invisiblethings.org/2013/03/21/introducing-qubes-odyssey-framework.html

I've heard that, but haven't seen any steps on how to do it

Many aspects of the architecture have been generalized, but that doesn't mean that users can simply follow a tutorial to switch hypervisors right now. Much more development work is still needed.

Since the discussion is beginning to veer off-topic and this issue has been closed for a quite a while, I'm going to lock this issue and kindly ask everyone to direct any further questions and discussion to our forum or mailing lists.

tlaurion commented 11 months ago

@marmarek

I see this conversation is unlocked again now.

@DemiMarie said above

@marmarek: I wonder if this can be build-time disabled in Xen, so that any attempt to enable it is a no-op. Nested virtualization in Xen is just broken.

So people want somehow nested virt. Kvm could do it. People want it for android studio (emulation if developed app etc) and firmware testing and other advanced users-aware-of-negative-corner-case-side-effects they are willing to adapt their risk exposure for without hacking their way to make it work, simply by installing packages and be ready to go. On the other side, Qubes team says wanting to make it impossible at build time.... So. Let's build something that would fit both sides of the story?

Will try to propose iterations toward a working compromise so user needing feature can have it if they actually choose to boot into that mode for whatever workflow they need to in total awareness, from boot time option. Meaning a separate Xen-nested-kvm, kernel-nested-kvm and whatnot so that we do iterations to make this feature available outside of the deployed defaults, totally on-demand (contrib repo? Unstable repo? You tell me).

I would love to see if qubesos team would be interested if community was making the work, made by those needing the feature, in the goal of having those packages installed side by side and used when needed and boot in that mode when needed, reboot in normal mode and be happy.

If positive feedback, will prioritize

learning qubes-builderv2 and respect contribution guidelines from https://m.youtube.com/watch?v=xo2BVTn7ohs&t=1544s
deprioritize https://github.com/devcontainers/images/issues/884 so qubes can be used instead for my needed workflow

As of now I expected @DemiMarie comment to be unchallengeable :

<@insurgo:matrix.org> Ok, otherwise some info on the state of the situation and needed patches ?

The situation is that nested virt in Xen is exploitably broken, with suspected privilege escalation vulnerabilities exploitable by both L1 and L2 guests. It is also extremely unlikely for any L1 guest that is not Xen itself to boot at all. Pretend it did not exist. It ought to be guarded by a Xen Kconfig option under UNSUPPORTED that defaults to off. XenServer has enterprise customers asking for nested virt so support will happen eventually, but there are other changes that have even higher priority. In particular, address-space isolation (essentially a secrets-free hypervisor) is critical for mitigating future speculative execution vulnerabilities and so will happen sooner.

But it seems (unconfirmed) that doing the following would be enough to expose vmx (outside of kvm requirements)

Drop a file named consistently with the name of the HVM you want nestedvirt enabled in folder: /etc/qubes/templates/libvirt/xen/by-name

$ cat ubuntu-untrusted-forensics.xml 
{% extends 'libvirt/xen.xml' %}
{% block cpu %}
<cpu mode='host-passthrough'>
<feature name='vmx' policy='require'/>
<feature name='svm' policy='require'/>
<!-- disable SMAP inside VM, because of Linux bug -->
<feature name='smap' policy='disable'/>
</cpu>
{% endblock %}
{% block features %}
<pae/>
<acpi/>
<apic/>
<nestedhvm/>
<viridian/>
<hap state='on'/> <!-- enable Hardware Assisted Paging -->
{% endblock %}

Challenge/guide me more?

marmarek commented 11 months ago

Unfortunately it isn't just security support (which is already a blocker for having it enabled in Qubes). It simply doesn't work. The second you try to start a nested VM, at best the outer VM will crash but more often the whole host.

adrelanos commented 11 months ago

https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen implies that this is possible. Which implies to me it is possible without a crash except that Qubes broke it (which is ok because not a project goal). To make it work again the community now would "only have to unbreak that".

I would love to see if qubesos team would be interested if community was making the work, made by those needing the feature, in the goal of having those packages installed side by side and used when needed and boot in that mode when needed, reboot in normal mode and be happy.

This.

Okay if the community tries?

DemiMarie commented 11 months ago

https://wiki.xenproject.org/wiki/Nested_Virtualization_in_Xen implies that this is possible. Which implies to me it is possible without a crash except that Qubes broke it (which is ok because not a project goal). To make it work again the community now would "only have to unbreak that".

As you can see from the tables in the article, nested virtualization support in Xen has either regressed over time, failed to keep pace with changing requirements from L1 hypervisors, or both.

I would love to see if qubesos team would be interested if community was making the work, made by those needing the feature, in the goal of having those packages installed side by side and used when needed and boot in that mode when needed, reboot in normal mode and be happy.

This.

Okay if the community tries?

Patches to improve nested virtualization in upstream Xen are welcome, and should be sent to xen-devel@lists.xenproject.org. This is a very difficult task and requires detailed knowledge of not only the Xen hypervisor, but also of the x86-64 architecture and Intel and AMD’s virtualization extensions.

As Marek pointed out, this isn’t really something that can be solved on the Qubes OS side.

github-actions[bot] commented 11 months ago

This issue has been closed as an "upstream issue." This means that the issue pertains to software that does not belong to the Qubes OS Project and that we do not develop or control. We suggest that you file this issue in the appropriate project's issue tracker instead. For more information, see Why don't you fix upstream bugs that affect Qubes OS?

We respect the time and effort you have taken to file this issue, and we understand that this outcome may be unsatisfying. Please accept our sincere apologies and know that we greatly value your participation and membership in the Qubes community.

If anyone reading this believes that this issue was closed in error or that the resolution of "upstream issue" is not accurate, please leave a comment below saying so, and we will review this issue again. For more information, see How issues get closed.

tlaurion commented 11 months ago

https://forum.qubes-os.org/t/xen-kvm-nested-virtualization-for-testing/20759/11

QubesOS / qubes-issues