Reverse engineering initiative for enabling SR-IOV on capable Ampere/GA10x GPUs

h1nk commented 4 years ago

There is a high amount of interest in SR-IOV technology among the VFIO/Linux gaming community. Due to Nvidia's love of market segmentation they refuse to enable SR-IOV on consumer GPUs. The current Ampere cards have the capability for SR-IOV but the feature is not enabled. I'm interested if there are any hardware hacking or firmware reverse engineering people who could explain if and how SR-IOV enablement could work if it's possible to bypass Nvidia's restrictions and I'd also like to open this issue to generate discussion around this topic.

Snawoot commented 4 years ago

Hi! I used to use Nvidia GeForce 1080 GPU for gaming inside Windows VM on Linux. All tests for this patch were conducted on Windows and Linux VMs as well. Nvidia GPU passthrough can be achieved with emulation of Hyper-V hypervisor using KVM. See this paragraph for required KVM configuration.

See also this Arch wiki page for common info on GPU passthrough on Linux. It appears to be useful for all distros, not just Arch Linux.

pengc99 commented 3 years ago

Hi, I don't think OP is talking about PCIe passthrough using KVM, I think he's talking about the use of SR-IOV for segmented GPU allocations to VMs. This allows you to break down one GPU into several smaller virtual GPUs, each of which is then passed through to a VM.

nVidia does already have this feature, mainly called vGPU and is used in the Grid line of products, but is disabled in consumer GeForce GPUs.

Snawoot commented 3 years ago

@pengc99 Hi! Thanks for clarification!

damajor commented 3 years ago

For reference: Nvidia Grid Nvidia Grid License documentation

h1nk commented 3 years ago

I'd be willing to put forward some money on a bounty for this, and I'm sure a lot of others would too

Snawoot commented 3 years ago

Looks like nobody inclined to look into this. Sorry.

h1nk commented 3 years ago

@Snawoot can this issue continue to be held open? While this issue isn't very relevant to the content of the repository it's a topic that isn't given much discussion as far as reverse engineering and I think giving it as much visability as possible is important. Perhaps this issue could even be pinned? It's only a matter of time until the right person is to try to attempt to tackle this.

h1nk commented 3 years ago

I want to begin cracking the Nvidia vGPU license server in the near future as a start in combatting this issue. I unfortunately don't own any of the vGPU "supported" cards. If there are any Quadro RTX owners please contact me

Snawoot commented 3 years ago

@h1nk No problem, reopening issue.

h1nk commented 3 years ago

In order to start reversing I need a copy of the latest NVIDIA vGPU License Server. I currently only have a copy of version 2018.06, anyone with a copy of the zip/setup.bin file please share it so I may take a look at it

damajor commented 3 years ago

As far as I know they provide it free of charge for download with trial licenses limited in time if you have a developer or partner account.

h1nk commented 3 years ago

The requirements for registering an Enterprise account for the limited vGPU evaluation are quite obnoxious (see https://enterpriseproductregistration.nvidia.com/?LicType=EVAL&ProductFamily=vGPU). I'll work on analyzing the 2018 license server I found for now until someone is able to provide the up-to-date server binary

vans163 commented 3 years ago

SR-IOV 2020 vGrid <2020

You recall that these features were in the silicon on olders cards like Maxwell and Kepler? So even if you enable it in-software/firmware-via-jumping-a-resistor, the GPU hardware will not support it.

Maybe the newer cards use the same silicon but just cut features off at the firmware level.

davidspek commented 3 years ago

I’d just like to note my interest in enabling SR-IOV for the RTX 30 series cards as well.

imtrobin commented 3 years ago

I want to begin cracking the Nvidia vGPU license server in the near future as a start in combatting this issue. I unfortunately don't own any of the vGPU "supported" cards. If there are any Quadro RTX owners please contact me

how to contact?

h1nk commented 3 years ago

@imtrobin if you have a RTX 6000/8000 you can DM me on gitter

zach78954 commented 3 years ago

Do you still need a copy, I have the most current one, it appears to be running a Java based FlexNet License server.

Let me know how I can help, I’d really like not to have to pay for VGPU license for my homelab :/

edit: I have a Tesla P4 if that helps.

h1nk commented 3 years ago

@zach78954 That would definitely be helpful, I currently only have the NVIDIA-ls-linux-2018.06.0.24304595.zip & NVIDIA-ls-linux-2020.05.0.28406365.zip files. Next weekend I will definitely set aside some time to start reversing the server finally.

zach78954 commented 3 years ago

I've uploaded a bundle of the latest server and drivers, hopefully that helps.

If you can find a solution I'll happily buy you a case of appreciation beer!

h1nk commented 3 years ago

@zach78954 Hey, thanks for the server files, do you have the Linux version of the LS? I only have a Linux machine currently

zach78954 commented 3 years ago

@h1nk, NVIDIA-ls-linux-2020.05.0.28406365 is actually the newest linux server, I was in the process of uploading it but then I realized its the same version you have so🤷

h1nk commented 3 years ago

@zach78954 In the archive I only see the Windows ZIP containing the setup.exe file:

$ unrar l NVIDIA.rar

UNRAR 5.61 beta 1 freeware      Copyright (c) 1993-2018 Alexander Roshal

Archive: NVIDIA.rar
Details: RAR 5

 Attributes      Size     Date    Time   Name
----------- ---------  ---------- -----  ----
    ..A.... 121463353  2021-03-25 20:22  NVIDIA-ls-Windows-2020.05.0.28406365.zip
    ..A.... 1088372804  2021-03-08 21:00  NVIDIA-GRID-vSphere-7.0-460.32.04-460.32.03-461.33.zip
----------- ---------  ---------- -----  ----
           1209836157                    2

zach78954 commented 3 years ago

You said you had the 2020 ls server for Linux. That’s the newest one. I can upload it if you want another copy.

Get Outlook for iOShttps://aka.ms/o0ukef

From: h1nk @.> Sent: Friday, March 26, 2021 10:10:35 AM To: keylase/nvidia-patch @.> Cc: zach78954 @.>; Mention @.> Subject: Re: [keylase/nvidia-patch] Reverse engineering initiative for enabling SR-IOV on capable Ampere/GA10x GPUs (#334)

@zach78954https://github.com/zach78954 In the archive I only see the Windows ZIP containing the setup.exe file:

$ unrar l NVIDIA.rar

Archive: NVIDIA.rar Details: RAR 5

Attributes Size Date Time Name

..A.... 121463353  2021-03-25 20:22  NVIDIA-ls-Windows-2020.05.0.28406365.zip
..A.... 1088372804  2021-03-08 21:00  NVIDIA-GRID-vSphere-7.0-460.32.04-460.32.03-461.33.zip

       1209836157                    2

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/keylase/nvidia-patch/issues/334#issuecomment-808379316, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB3FN73P2L65NIGR5DAFYHTTFS5YXANCNFSM4TDM67CA.

h1nk commented 3 years ago

@zach78954 sorry, I misread what you said, the Linux server version did not change since the one I have

zach78954 commented 3 years ago

Let me know if I can help in any other way!!

olealgoritme commented 3 years ago

Out of curiosity, has anyone tried flashing a A40 ampere vbios onto a 3090?

Edit: if anyone has the A40 vbios, give us a shout

Saentist commented 3 years ago

Out of curiosity, has anyone tried flashing a A40 ampere vbios onto a 3090?

Edit: if anyone has the A40 vbios, give us a shout

Without resistors movement?

Steve0929 commented 3 years ago

I believe there is some experimental code to try to enable vgpu in consumer GeForce GPUs here: https://github.com/DualCoder/vgpu_unlock haven't tested it yet though.

h1nk commented 3 years ago

The tool requires the NVIDIA GRID vGPU driver

Someone should upload all versions of the vGPU driver they have access to

zach78954 commented 3 years ago

The tool requires the NVIDIA GRID vGPU driver

Someone should upload all versions of the vGPU driver they have access to

I uploaded all the complete 460 driver version for ESXI.

This there something else or a specific version you need?

@h1nk any luck on the licensing?

h1nk commented 3 years ago

@zach78954 hi, I still haven't taken a look at cracking the server yet besides looking over some decompilations, but will definitely try at it soon. Using the vGPU unlocker I would like to try to partition my single GeForce card for a QEMU/KVM passthrough. If you have the drivers that would work with KVM that'd be sweet

zach78954 commented 3 years ago

If I have some time tonight, I’ll upload the Linux KVM driver pack.

Best of luck on the cracking!

Get Outlook for iOShttps://aka.ms/o0ukef

From: h1nk @.> Sent: Friday, April 9, 2021 4:59:58 PM To: keylase/nvidia-patch @.> Cc: zach78954 @.>; Mention @.> Subject: Re: [keylase/nvidia-patch] Reverse engineering initiative for enabling SR-IOV on capable Ampere/GA10x GPUs (#334)

@zach78954https://github.com/zach78954 hi, I still haven't taken a look at cracking the server yet besides looking over some decompilations, but will definitely try at it soon. Using the vGPU unlocker I would like to try to partition my single GeForce card for a QEMU/KVM passthrough. If you have the drivers that would work with KVM that'd be sweet

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/keylase/nvidia-patch/issues/334#issuecomment-817033671, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AB3FN73P2LQZATNKFA6PIKDTH6IH5ANCNFSM4TDM67CA.

zach78954 commented 3 years ago

@h1nk here is the KVM Driver Link .

WeebDataHoarder commented 3 years ago

Here's a folder with all that has been collected so far, and a few more coming up soon (slow connections).

https://mega.nz/folder/xhxmFIwS#plgmwukDdylDAKUa_-qygg

Will try to update the list below when contents change, no promises there :)

Current contents:

GRID Driver

NVIDIA-GRID-Linux-KVM-410.92-410.91-412.16.zip
NVIDIA-GRID-Linux-KVM-430.67-430.63-432.08.zip
NVIDIA-GRID-Linux-KVM-440.87-443.05.zip
NVIDIA-GRID-Linux-KVM-450.80-450.80.02-452.39.zip 
NVIDIA-GRID-Linux-KVM-450.89-452.57.zip
NVIDIA-GRID-Linux-KVM-460.32.04-460.32.03-461.33.zip
NVIDIA-GRID-vGPU-kepler-vSphere-2015-352.83-354.80.zip
NVIDIA-GRID-vGPU-kepler-XenServer-7.0-367.134-370.41.zip
NVIDIA-GRID-vSphere-7.0-460.32.04-460.32.03-461.33.zip
NVIDIA-vGPU-kepler-vSphere-6.5-367.134-370.41.zip
NVIDIA-GRID-Windows-440.87-443.05.zip

License Server

NVIDIA-ls-linux-2018.06.0.24304595.zip
NVIDIA-ls-linux-2018.10.0.25098346.zip
NVIDIA-ls-linux-2019.11.0.27609831.zip
NVIDIA-ls-linux-2020.05.0.28406365.zip
NVIDIA-ls-windows-2018.06.0.24304595.zip
NVIDIA-ls-windows-2018.10.0.25098346.zip
NVIDIA-ls-Windows-2019.11.0.27609831.zip
NVIDIA-ls-Windows-2020.05.0.28406365.zip

For NVIDIA-ls-linux-2018.10.0.25098346.zip, (and probably all other Linux zip files) on setup.bin, you can run this to take just the .zip installer contents off the shell script: $ tail -c +131073 setup.bin > setup.zip. This index is hex 0x20000 + 1 to decimal.

Tools

NVIDIA-gpumodeswitch-2016-04.zip

(Cannot find current NVIDIA-gpumodeswitch-2020-01.zip so if anyone has it, upload away, could be useful)

Snawoot commented 3 years ago

Can this help somehow on given subject? More precisely, this issue: https://github.com/DualCoder/vgpu_unlock/issues/8

keylase / nvidia-patch