Enable nested virtualization

[felipecrs]

I don't know which is the hypervisor used to spin the VMs, but enabling nested virtualization would be a great enhancement. For example, this way we could build Vagrant boxes in GitHub Actions.

[felipecrs]

This feature is enabled by default in Travis CI. So I wonder why not here?

[dalehamel]

It looks like this is likely because azure only supports nested virtualization on dv3 or above https://azure.microsoft.com/en-ca/blog/nested-virtualization-in-azure/ but azure runs on ds2 instances: https://help.github.com/en/actions/automating-your-workflow-with-github-actions/virtual-environments-for-github-hosted-runners#cloud-hosts-for-github-hosted-runners

This seems to be older than dv3 from the catalogue: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dsv2-series vs https://docs.microsoft.com/en-us/azure/virtual-machines/windows/sizes-general#dv3-series-1

I haven't checked the exact CPU features, but i wonder if Hyper-V powering azure needs to work with a specific CPU family / featureset?

Has anyone tried just running KVM on azure? What happens? Is there even support for it in the stock azure / ubuntu kernel for bionic?

[dalehamel]

My use-case is I'd like to run an android emulator on Github Actions btw, like with https://github.com/google/android-emulator-container-scripts

[dalehamel]

When I run:

$egrep -c '(vmx|svm)' /proc/cpuinfo
0

This indicates there isn't CPU support, at least not passed through to the guest.

I installed kvm packages and was able to modprobe kvm, but there is no /dev/kvm, which is what I would expect.

Unless Azure adds support for this, or github actions switches / offers other machine classes, I don't think nested virtualization can be done.

[felipecrs]

Travis supports not only nested virtualization but also xvfb, that I guess you also need to run your Android tests.

https://github.com/felipecassiors/ubuntu1804-4dev/blob/master/.travis.yml

The support for nested virtualization is out-of-the-box there, where I use to build a Vagrant box using VirtualBox.

[AkihiroSuda]

https://github.com/actions/virtual-environments/blob/master/images/macos/macos-10.15-Readme.md

Intel x86 Emulator Accelerator (HAXM installer) | 7.5.1

macOS instances seem to support nested virtualization? (I couldn't install VirtualBox though: https://github.com/actions/virtual-environments/issues/429)

[hannes]

+1, I would also really like to see this. I wonder what went wrong, if I manually spin up an equivalent Azure VM, it has vmx set and VirtualBox works fine with hardware acceleration.

[felipecrs]

Seems like it is possible indeed: https://docs.microsoft.com/en-us/azure/virtual-machines/windows/nested-virtualization

[dalehamel]

Yes azure supports it but not the instance type GitHub actions runs on

[hannes]

Works fine on Travis ^^

[AjkayAlan]

+1 - Building vagrant boxes using packer is not currently possible with GitHub Actions, unless we self-host a runner. Same with Azure DevOps (which appears to use the same hosted images as GitHub Actions).

So, if this got prioritized, you would maybe solve this on both platforms.

[dalehamel]

I suspect that with self-hosted runners on azure you could circumvent this issue, as jobs could be dispatched to a host that supports nested virtualization.

The current worker that actions runs on in their free offering just doesn’t have a processor that supports this.

I suspect that if this feature is ever offered, it will be when a compatible processor is bumped into this low-tier instance class. Until then, self-hosted runners could be feasible but only really for corporate users.

[njsmith]

I wouldn't mind a self-hosted runner, but I need nested virtualization for a public project, and those aren't really usable with self-hosted runners: https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

[hannes]

I wouldn't mind a self-hosted runner, but I need nested virtualization for a public project, and those aren't really usable with self-hosted runners: https://help.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security-with-public-repositories

Yes, exactly, that's also the reason we can't use them.

[mvgijssel]

Would also love this to be able to use packer to build images.

[Andrei-Pozolotin]

+1

[alepauly]

Unfortunately we can't support this feature at this time, that said the feedback here is valuable and we will definitely keep it in mind . Thanks all.

[mvgijssel]

Folks might be interested in https://github.com/actions/virtual-environments/issues/433#issuecomment-610789144

[typetetris]

+1

[filipkowicz]

Android developers need it run emulator way faster. Needed for instrumentation tests

[mikehardy]

This won't be possible until GitHub hosted runners are updated to v3 or higher. They are on v2.

This likely has cost implications etc - it's impossible to say unless they official respond why it's not happening or where in the roadmap it may/will happen.

@alepauly Is there anywhere we can subscribe to changes in the runner build definition so we can just know when it's possible? Just in case the communication loop isn't closed?

There doesn't seem to be any declaration of Azure machine type target on the deployed runners in the virtual environments repo.

With a 10x actions minute multiplier charged for billing macOS runners, and those runners required for mobile testing (hardware emulator acceleration), most mobile test scenarios will probably need to farm the E2E testing out to Travis or Circle until this changes

[domenkozar]

This is on the roadmap, but it may not be possible for OSS: https://github.com/github/roadmap/issues/95

[mikehardy]

@domenkozar Ironically, if the intention is to limit expenses for OSS (a valid reason, no argument there) in the android emulator case at least the nested virtualization is so much more efficient that it might represent a savings due to faster execution / less CPU-hours burnt, but only if load doesn't migrate here from other systems I suppose

[domenkozar]

Note that I'm not a GitHub employee and this is pure speculation.

[felipecrs]

What I don't understand is why this Issue is closed.

[alepauly]

What I don't understand is why this Issue is closed.

@felipecrs - There are technical reasons why we can't support this at this time unfortunately.

This is on the roadmap, but it may not be possible for OSS: github/roadmap#95

@domenkozar - I don't think the multiple hosted runner size feature will necessarily lead to nested virtualization becoming available, I doubt it. Also, I'm not sure that I would infer that multiple sizes of runners will leave out OSS, I don't think it will.

@mikehardy (and all) - thanks for continuing to add thoughts in this issue (and sorry for my delayed response!). I closed this issue because we currently can't support it and there are no plans to do so soon. That said, I'll continue to pass this on to our team so they have the information they need to make decisions about the roadmap.

[elgalu]

Hi! is this issue up to date? is https://github.com/vagrant-libvirt/vagrant-libvirt not supported?

[joffrey-bion]

The inability to run linux containers on windows runners drastically reduces their value. I have a bunch of tests that rely on services running in docker and I cannot run those tests on windows because of that. I had to remove my windows pipelines which are kinda useless without those tests 😢

[jseguillon]

I can launch nested KVMs in my WSL2 windows 11... Why can't runners do the same ? Could this be a workaround if win11 runners exists ?

[mikehardy]

I can launch nested KVMs in my WSL2 windows 11... Why can't runners do the same ? Could this be a workaround if win11 runners exists ?

You'll have to test it and see. I do not see windows 11, I see various windows server versions: https://github.com/actions/virtual-environments/tree/main/images/win

They appear to have WSL1 on in the 2022 version and identical hardware to the linux runners: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources - so I am guessing they are running on the same "virtual metal" that linux is, and thus not capable of nested virtualization.

All assertions here are inferred, testing would obviously yield concrete results

[evertonlperes]

I can launch nested KVMs in my WSL2 windows 11... Why can't runners do the same ? Could this be a workaround if win11 runners exists ?

You'll have to test it and see. I do not see windows 11, I see various windows server versions: https://github.com/actions/virtual-environments/tree/main/images/win

They appear to have WSL1 on in the 2022 version and identical hardware to the linux runners: https://docs.github.com/en/actions/using-github-hosted-runners/about-github-hosted-runners#supported-runners-and-hardware-resources - so I am guessing they are running on the same "virtual metal" that linux is, and thus not capable of nested virtualization.

All assertions here are inferred, testing would obviously yield concrete results

That's interesting. I'm also struggling to run some E2E tests on the existing GitActions runners. The app need to run into a VM with nested-virtualization enabled (macOS, Windows and Linux). Also, for Windows the app requires WSL2 enabled instead WSL1.

Looking into the runners VM_Config, for Windows and Linux the VM has nested-virtualization embedded already.

• windows-2022 --> Standard_D8s_v4 (ref)
• ubuntu-20.04 --> Standard_D4s_v4 (ref)

I do have a Standard_D4s_v4 with Windows PRO 2H20 + WSL2 enabled and all the tests works fine. Not sure if the nested virtualization was disabled for GitActions runners though :/

[miketimofeev]

@evertonlperes these sizes are used only for image generation to speed up the process. The runners have different specs and don't have nested virtualization.

[evertonlperes]

@evertonlperes these sizes are used only for image generation to speed up the process. The runners have different specs and don't have nested virtualization.

Thanks for clarify @miketimofeev It could be a blocker to us, unfortunately :(

[alonbl]

Hi, Is this issue is what prevents the introduction of WSL2 in GitHub workflow slaves? Thanks!

[panekj]

@alonbl WSL2 is not available in Windows Server SKUs https://github.com/microsoft/WSL/issues/6301

[felipecrs]

And even if it were available, yes, this issue would also prevent GitHub Actions from using WSL2.

[domenkozar]

Has anyone tested the new larger runners if they have nested virt?

[mikehardy]

@domenkozar nice find! It looks like the DS4 machine type maps best to the configuration options available (https://docs.microsoft.com/en-us/azure/virtual-machines/dv4-dsv4-series) and they list it on azure as "nested virtualization supported". I wouldn't bet much, but I'd bet $1 it does work unless Github has specifically disabled it.

[maxkratz]

Has anyone tested the new larger runners if they have nested virt?

@domenkozar Unfortunately, they are "paid-only" for Teams and Enterprise users (from https://github.blog/changelog/2022-09-01-github-actions-larger-runners-are-now-in-public-beta/):

Larger runners are charged for in both private and public repos and do not consume included minutes.

[martijnarts]

It looks like large runners don't have KVM access enabled: https://github.com/ReactiveCircus/android-emulator-runner/issues/46#issuecomment-1284155863

[sbates130272]

I am another +1 on this but I do understand that GitHub prefers to run its runners (sic) on hardware that might not support nested virtualization out of the box. Of course anyone can do a self-hosted runner on hardware that does support nested-virtualization and do it that way. Any input from tyhe GitHub team on when their runner hardware will support nested?

[mattjohnsonpint]

Of course anyone can do a self-hosted runner on hardware that does support nested-virtualization and do it that way.

@sbates130272 - Unfortunately that's not quite true, because self-hosted runners are a security risk for public repos. https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners#self-hosted-runner-security

[nebuk89]

on the larger runners if you add yourself to the KVM user group on Linux, you should be good to go.

            sudo adduser $USER kvm
            sudo chown -R $USER /dev/kvm

Though this is not formally supported yet as we are still sorting the bits for Windows to work, though that should come in the next few weeks along with a change log documenting this as in support for runners >4 cores.

If anyone has any feedback on this or concerns or has issues I would <3 to hear.

[mrk-han]

on the larger runners if you add yourself to the KVM user group on Linux, you should be good to go.

            sudo adduser $USER kvm
            sudo chown -R $USER /dev/kvm

Though this is not formally supported yet as we are still sorting the bits for Windows to work, though that should come in the next few weeks along with a change log documenting this as in support for runners >4 cores.

If anyone has any feedback on this or concerns or has issues I would <3 to hear.

Thank you so much! Will experiment with this and get back to you. This is very exciting.

[mattjohnsonpint]

... for runners >4 cores.

Did you mean >= 4 cores ? Or will we need an 8 core runner minimum?

[maxkratz]

on the larger runners if you add yourself to the KVM user group on Linux, you should be good to go.

I first misread this and tried it on the normal (base) runners.

[...]
jobs:
  vagrant-provision:
    runs-on: ubuntu-22.04
    steps:
    - uses: actions/checkout@v3
    - name: set up KVM
      run: |
        sudo adduser $USER kvm
        sudo chown -R $USER /dev/kvm
[...]

resulting in:

Adding user `runner' to group `kvm' ...
Adding user runner to group kvm
Done.
chown: cannot access '/dev/kvm': No such file or directory
Error: Process completed with exit code 1.

Are there any plans on supporting KVM on the "normal" runner VMs @nebuk89?

[Gustl22]

Now Flutter removes OpenGL support in 3.7.0 which was testable through GH-CI software rendering at least on macOS. How developers are supposed to test their GUI applications in GH actions?

[nebuk89]

@mattjohnsonpint I did mean >= ! So 4-core will work.

@maxkratz not right now :( this is a test for us to see how people use it and to ensure our ability to detect bad actors abusing Actions when we have nested virt as another tool in the mix.

@Gustl22 a good question, @Steve-Glass is doing research into the future of what we offer on the Mac runners specifically :) I suspect we don't have an answer quite yet as 3.7 has only been out a week or so. It could be worth breaking that out into a separate issue?

[Gustl22]

@nebuk89 You mean a separate issue for GPU support on MacOS (maybe with Apple Silicon / M1) or for the specific Flutter 3.7 issue? ~I think it all comes down to put in some effort for GPU support in general from GH side. This is on the table for far too long and other CI providers provide much better conditions for GUI testing on all platforms. Idk, if a separation from this issue will solve the general problem.~

Edit: seems like there's a light at the end of the tunnel by supporting Linux and Windows nested virtualisation, let's hope it is usable on the free plan as well some day :) And yes, you're right, sorry: nested virtualization and GPU are two different things, although they are strongly connected as stated here.

[maxkratz]

@maxkratz not right now :( this is a test for us to see how people use it and to ensure our ability to detect bad actors abusing Actions when we have nested virt as another tool in the mix.

Thank you for the clarification.