Open godcrying opened 1 year ago
@godcrying Tesla M40 requires 32GB BAR to my knowledge, this is impossible to make work on Ivy Bridge with 32GB of RAM due to the 36-bit (16GB) addressing limitation. The reason the patches cause this issue is because unpatched Ivy Bridge just ignores BARs =>4GB and leaves them unallocated.
But if you use Linux it may be possible to use this GPU on Ivy Bridge along with less than 30GB of RAM. You need to turn off 4G decoding and modify DSDT to change If (((MM64 == Zero) || (OSYS <= 0x07D3)))
to If (((OSYS <= 0x07D3)))
. This will make the 64-bit MMIO region exposed (except on Windows XP) even with 4G decoding disabled.
When 4G decoding is disabled the now exposed 64-bit MMIO region should be empty so Linux should allocate the 32GB BAR Tesla M40 there without any issues.
@xCuri0 Really thanks for your answer. I've been stuck with this trouble for a long time. Did you mean that I can use this GPU in linux just by modifying the DSDT, without 4G decoding and without patch bios ?
It may work, can't give any guarantees though. You don't need to do any flashing but doing the DSDT patch in BIOS is recommended over having Linux replace it if you want to make it permanent.
But a few people had a similar method work on Haswell systems
@xCuri0 Thank you. I will try it.
Post /proc/iomem
output (as root) one you have the DSDT edit running with 4G decoding off, because I'm unsure if Intel ME gets allocated 32-bit or not (MESEG_BASE
).
@xCuri0 I have reduced RAM to 8G and turn off the 4G decoding.
The result of sudo cat /proc/iomem
with modified DSDT and 4G decoding off:
00000000-00000fff : Reserved
00001000-0009cfff : System RAM
0009d000-0009efff : Reserved
0009f000-0009ffff : System RAM
000a0000-000dffff : PCI Bus 0000:00
000c0000-000dffff : 0000:00:02.0
000e4000-000effff : PCI Bus 0000:00
000f0000-000fffff : System ROM
00100000-1fffffff : System RAM
20000000-201fffff : Reserved
20200000-3fffffff : System RAM
31c00000-32bfffff : Kernel code
32c00000-33879fff : Kernel rodata
33a00000-33c121ff : Kernel data
3441b000-347fffff : Kernel bss
40000000-401fffff : Reserved
40200000-d0585fff : System RAM
d0586000-d0586fff : Reserved
d0587000-d20c8fff : System RAM
d20c9000-d210bfff : ACPI Non-volatile Storage
d210c000-d6f08fff : System RAM
d6f09000-d6ffffff : Reserved
d7000000-d7753fff : System RAM
d7754000-d77fffff : Reserved
d7800000-d7fb3fff : System RAM
d7fb4000-d7ffffff : ACPI Tables
d8000000-d971efff : System RAM
d971f000-d97fffff : ACPI Non-volatile Storage
d9800000-dae06fff : System RAM
dae07000-dae1111b : ACPI Non-volatile Storage
dae1111c-dae11fff : System RAM
dae12000-df9fffff : Reserved
dba00000-df9fffff : Graphics Stolen Memory
dfa00000-feafffff : PCI Bus 0000:00
e0000000-efffffff : 0000:00:02.0
f0000000-f0ffffff : PCI Bus 0000:01
f0000000-f0ffffff : 0000:01:00.0
f0000000-f0ffffff : nvidia
f7800000-f7bfffff : 0000:00:02.0
f7c00000-f7c1ffff : 0000:00:19.0
f7c00000-f7c1ffff : e1000e
f7c20000-f7c2ffff : 0000:00:14.0
f7c20000-f7c2ffff : xhci-hcd
f7c30000-f7c33fff : 0000:00:1b.0
f7c30000-f7c33fff : ICH HD audio
f7c35000-f7c350ff : 0000:00:1f.3
f7c36000-f7c367ff : 0000:00:1f.2
f7c36000-f7c367ff : ahci
f7c37000-f7c373ff : 0000:00:1d.0
f7c37000-f7c373ff : ehci_hcd
f7c38000-f7c383ff : 0000:00:1a.0
f7c38000-f7c383ff : ehci_hcd
f7c39000-f7c39fff : 0000:00:19.0
f7c39000-f7c39fff : e1000e
f8000000-fbffffff : PCI MMCONFIG 0000 [bus 00-3f]
fec00000-fec00fff : Reserved
fec00000-fec003ff : IOAPIC 0
fed00000-fed03fff : Reserved
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed1c000-fed1ffff : Reserved
fed1f410-fed1f414 : iTCO_wdt.1.auto
fed1f410-fed1f414 : iTCO_wdt.1.auto iTCO_wdt.1.auto
fed90000-fed90fff : dmar0
fed91000-fed91fff : dmar1
fee00000-fee00fff : Local APIC
fee00000-fee00fff : Reserved
ff000000-ffffffff : INT0800:00
100000000-21e5fffff : System RAM
21e600000-21fffffff : RAM buffer
And I also add pci=realloc
to the grub, nvidia-smi
still return No devices were found
.
The lspci -v
shows:
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09) (prog-if 00 [Normal decode])
Subsystem: Dell Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port
Flags: bus master, fast devsel, latency 0, IRQ 26
Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
I/O behind bridge: [disabled] [16-bit]
Memory behind bridge: f0000000-f0ffffff [size=16M] [32-bit]
Prefetchable memory behind bridge: [disabled] [64-bit]
Capabilities: [88] Subsystem: Dell Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port
Capabilities: [80] Power Management version 3
Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
Capabilities: [a0] Express Root Port (Slot+), MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [140] Root Complex Link
Kernel driver in use: pcieport
...
00:1f.3 SMBus: Intel Corporation 7 Series/C216 Chipset Family SMBus Controller (rev 04)
Subsystem: Dell 7 Series/C216 Chipset Family SMBus Controller
Flags: medium devsel, IRQ 18
Memory at f7c35000 (64-bit, non-prefetchable) [size=256]
I/O ports at f040 [size=32]
Kernel driver in use: i801_smbus
Kernel modules: i2c_i801
01:00.0 3D controller: NVIDIA Corporation GM200GL [Tesla M40] (rev a1)
Subsystem: NVIDIA Corporation GM200GL [Tesla M40]
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at f0000000 (32-bit, non-prefetchable) [size=16M]
Capabilities: [60] Power Management version 3
Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [78] Express Endpoint, MSI 00
Capabilities: [100] Virtual Channel
Capabilities: [250] Latency Tolerance Reporting
Capabilities: [258] L1 PM Substates
Capabilities: [128] Power Budgeting <?>
Capabilities: [420] Advanced Error Reporting
Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
Capabilities: [900] Secondary PCI Express
Kernel driver in use: nvidia
Kernel modules: nouveau, nvidia_drm, nvidia
@godcrying Can you send output of sudo dmesg | grep "root bus resource"
, it looks like DSDT hasn't been modified because the 64-bit Pci Bus region doesn't show
@xCuri0 The result is :
[ 0.238047] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window]
[ 0.238050] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window]
[ 0.238052] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000dffff window]
[ 0.238054] pci_bus 0000:00: root bus resource [mem 0x000e4000-0x000effff window]
[ 0.238056] pci_bus 0000:00: root bus resource [mem 0xdfa00000-0xfeafffff window]
[ 0.238058] pci_bus 0000:00: root bus resource [bus 00-3e]
And I also checked DSDT, which is:
[ 0.016430] ACPI: DSDT ACPI table found in initrd [kernel/firmware/acpi/dsdt.aml][0xa11c]
[ 0.016538] ACPI: Table Upgrade: override [DSDT-DELL - CBX3 ]
[ 0.016541] ACPI: DSDT 0x00000000D7FF4188 Physical table override, new table: 0x00000000DAE07000
[ 0.016545] ACPI: DSDT 0x00000000DAE07000 00A11C (v02 DELL CBX3 00000023 INTL 20221020)
[ 0.016601] ACPI: Reserving DSDT table memory at [mem 0xdae07000-0xdae1111b]
v02 DELL CBX3 00000023 INTL 20221020
is the modified DSDT, the original is 00000022
@godcrying can you send the whole modified DSDT ? because the 64-bit region isn't there
dsdt.dsl.txt @xCuri0 here it is
I've delete the MM64==Zero
@godcrying Remove
Else
{
CreateDWordField (BUF0, \_SB.PCI0._Y0F._LEN, M4LN) // _LEN: Length
M4LN = Zero
}
And change ElseIf (E4GM)
to Else
. Make sure you have done the DSDT modification to this part described in wiki also.
You should then get full 36-bit range usable then and kernel should allocate M40 there.
Wow! brilliant! 😄😄
Sat Aug 19 01:27:55 2023
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.98 Driver Version: 535.98 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Tesla M40 24GB Off | 00000000:01:00.0 Off | Off |
| N/A 33C P0 56W / 250W | 0MiB / 24576MiB | 2% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| No running processes found |
+---------------------------------------------------------------------------------------+
@xCuri0 Really appreciate your help !
@godcrying Nice! I'm suprised it actually worked.
You could also send /proc/iomem
output (as root), because I'm unsure about M40 BAR size which might be useful later. See if pci=realloc
is actually needed also
here it is:
00000000-00000fff : Reserved
00001000-0009cfff : System RAM
0009d000-0009efff : Reserved
0009f000-0009ffff : System RAM
000a0000-000dffff : PCI Bus 0000:00
000c0000-000dffff : 0000:00:02.0
000e4000-000effff : PCI Bus 0000:00
000f0000-000fffff : System ROM
00100000-1fffffff : System RAM
20000000-201fffff : Reserved
20200000-3fffffff : System RAM
40000000-401fffff : Reserved
40200000-d0585fff : System RAM
b8400000-b93fffff : Kernel code
b9400000-ba079fff : Kernel rodata
ba200000-ba4121ff : Kernel data
bac1b000-baffffff : Kernel bss
d0586000-d0586fff : Reserved
d0587000-d20c8fff : System RAM
d20c9000-d210bfff : ACPI Non-volatile Storage
d210c000-d6f08fff : System RAM
d6f09000-d6ffffff : Reserved
d7000000-d7753fff : System RAM
d7754000-d77fffff : Reserved
d7800000-d7fb3fff : System RAM
d7fb4000-d7ffffff : ACPI Tables
d8000000-d971efff : System RAM
d971f000-d97fffff : ACPI Non-volatile Storage
d9800000-dae06fff : System RAM
dae07000-dae11100 : ACPI Non-volatile Storage
dae11101-dae11fff : System RAM
dae12000-df9fffff : Reserved
dba00000-df9fffff : Graphics Stolen Memory
dfa00000-feafffff : PCI Bus 0000:00
e0000000-efffffff : 0000:00:02.0
f0000000-f0ffffff : PCI Bus 0000:01
f0000000-f0ffffff : 0000:01:00.0
f0000000-f0ffffff : nvidia
f7800000-f7bfffff : 0000:00:02.0
f7c00000-f7c1ffff : 0000:00:19.0
f7c00000-f7c1ffff : e1000e
f7c20000-f7c2ffff : 0000:00:14.0
f7c20000-f7c2ffff : xhci-hcd
f7c30000-f7c33fff : 0000:00:1b.0
f7c30000-f7c33fff : ICH HD audio
f7c35000-f7c350ff : 0000:00:1f.3
f7c36000-f7c367ff : 0000:00:1f.2
f7c36000-f7c367ff : ahci
f7c37000-f7c373ff : 0000:00:1d.0
f7c37000-f7c373ff : ehci_hcd
f7c38000-f7c383ff : 0000:00:1a.0
f7c38000-f7c383ff : ehci_hcd
f7c39000-f7c39fff : 0000:00:19.0
f7c39000-f7c39fff : e1000e
f8000000-fbffffff : PCI MMCONFIG 0000 [bus 00-3f]
fec00000-fec00fff : Reserved
fec00000-fec003ff : IOAPIC 0
fed00000-fed03fff : Reserved
fed00000-fed003ff : HPET 0
fed00000-fed003ff : PNP0103:00
fed1c000-fed1ffff : Reserved
fed1f410-fed1f414 : iTCO_wdt.1.auto
fed1f410-fed1f414 : iTCO_wdt.1.auto iTCO_wdt.1.auto
fed90000-fed90fff : dmar0
fed91000-fed91fff : dmar1
fee00000-fee00fff : Local APIC
fee00000-fee00fff : Reserved
ff000000-ffffffff : INT0800:00
100000000-21e5fffff : System RAM
21e600000-fffffffff : PCI Bus 0000:00
400000000-fffffffff : PCI Bus 0000:01
400000000-401ffffff : 0000:01:00.0
800000000-fffffffff : 0000:01:00.0
The address has been fffffffff
.
And without pci=realloc
, it also works. :D
By the way, how much RAM space left can I use for my host memory?
@godcrying Anything under 30GB should work
@xCuri0 Finally i found it works only with 8G RAM. When I increase the RAM to 16G, my card doesn't work. Maybe there is a limit that card ram + host ram <= 32G ?
@godcrying can u send full dmesg output and also /proc/iomem
with 16gb ?
[ 0.286194] pci 0000:00:01.0: BAR 15: no space for [mem size 0xc00000000 64bit pref]
[ 0.286198] pci 0000:00:01.0: BAR 15: failed to assign [mem size 0xc00000000 64bit pref]
[ 0.286202] pci 0000:00:01.0: BAR 14: assigned [mem 0xf0000000-0xf0ffffff]
[ 0.286206] pci 0000:01:00.0: BAR 1: no space for [mem size 0x800000000 64bit pref]
[ 0.286208] pci 0000:01:00.0: BAR 1: failed to assign [mem size 0x800000000 64bit pref]
[ 0.286210] pci 0000:01:00.0: BAR 3: no space for [mem size 0x02000000 64bit pref]
[ 0.286212] pci 0000:01:00.0: BAR 3: failed to assign [mem size 0x02000000 64bit pref]
[ 0.286214] pci 0000:01:00.0: BAR 0: assigned [mem 0xf0000000-0xf0ffffff]
[ 0.286219] pci 0000:00:01.0: PCI bridge to [bus 01]
[ 0.286222] pci 0000:00:01.0: bridge window [mem 0xf0000000-0xf0ffffff]
[ 0.286228] pci_bus 0000:00: Some PCI device resources are unassigned, try booting with pci=realloc
There's the whole 32-64GB region still available, I have no idea why it's saying no space for the 32GB BAR.
Maybe realloc will fix it like it says
@xCuri0 I added pci=realloc
. It doesn't work.
iomem_pci_realloc.txt
dmesg_pci_realloc.txt
[ 0.286194] pci 0000:00:01.0: BAR 15: no space for [mem size 0xc00000000 64bit pref]
[ 0.286198] pci 0000:00:01.0: BAR 15: failed to assign [mem size 0xc00000000 64bit pref]
[ 0.286202] pci 0000:00:01.0: BAR 14: assigned [mem 0xf0000000-0xf0ffffff]
it seems like the PCI bridge needs a 48GB BAR for some reason ? I have no idea why, don't really know about how PCI bridges work. On my LGA1155 system 00:01.0
is Xeon(R) processor E3 - 1200 v2/3rd Gen Core processor PCI Express Root Port - 0151
so I assume yours is the same
48GB along with 16GB RAM and other PCIe devices isn't going to fit within 64GB addressing limit. Need to figure out why 48GB bridge window is needed and if it's possible to reduce it.
Btw I think you made a mistake in DSDT edit (not cause of this issue though) [ 0.254757] acpi PNP0A08:00: host bridge window [mem 0x41e600000-0x141e5ffffe window] ([0x1000000000-0x141e5ffffe] ignored, not CPU addressable)
, this will cause Windows to BSOD if you want to flash BIOS with this modified DSDT.
@xCuri0 Finally i found it works only with 8G RAM. When I increase the RAM to 16G, my card doesn't work. Maybe there is a limit that card ram + host ram <= 32G ?
From a quick glance it seems your BIOS has dumbly allocated BAR2 at 16GiB address instead of behind BAR1 at 32GiB. Best to fix this or if too hard then turn off 'remap' in BIOS so that you might have at least 15GiB+ of RAM.
This also works for a Xeon Phi on an Asus P8Z77-V LX. Using the acpi
config option in GRUB to override DSDT without modifying UEFI didn't work though, and both MMTool and UEFIPatch create broken pad files, unless the new DSDT is smaller than the original one.
I tried using this method on a Z97-WS (Haswell) and successfully exposed the larger memory space at the PCI root, but am running into a snag with the PLX bridge (PCI Bus 0000:01) present on the chip.
It seems no matter what I try, I can't get a large memory window assigned at the bridge so that the gpus underneath can get resized properly with the NVReg_EnableResizeableBar
operation set.
I have managed to use UEFIPatch + ReBarState to get up to 4G per card, and in that case, the PLX bridge (PCI Bus 0000:01) gets 12G of prefetchable memory assigned, but this is still inadequate for these cards.
What I'm talking about:
83fe00000-7fffffffff : PCI Bus 0000:00
fc0000000-ff1ffffff : PCI Bus 0000:01
fc0000000-ff1ffffff : PCI Bus 0000:02
fc0000000-fd1ffffff : PCI Bus 0000:04
fc0000000-fcfffffff : 0000:04:00.0
fd0000000-fd1ffffff : 0000:04:00.0
fe0000000-ff1ffffff : PCI Bus 0000:03
fe0000000-fefffffff : 0000:03:00.0
ff0000000-ff1ffffff : 0000:03:00.0
$ sudo lspci -v -s 01:00.0
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca) (prog-if 00 [Normal decode])
Subsystem: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch
Flags: bus master, fast devsel, latency 0, IRQ 16
Memory at df100000 (32-bit, non-prefetchable) [size=256K]
Bus: primary=01, secondary=02, subordinate=05, sec-latency=0
I/O behind bridge: d000-efff [size=8K] [16-bit]
Memory behind bridge: dc000000-df0fffff [size=49M] [32-bit]
Prefetchable memory behind bridge: fc0000000-ff1ffffff [size=800M] [32-bit]
Any ideas how to get the PLX chip memory window opened up too? /cc @xCuri0
EDIT: I managed to get a larger prefetchable memory in the PLX bridge by using the pci=resource_alignment
directly on the gpu pci buses themselves, but this has the side effect of resizing all of the gpu memory resources and also the nvidia driver no longer loads, claiming the BARs memory spaces are invalid.
@zcrypt0 why not use rebaruefi and let the bios do the pci allocation instead of making linux do it ?
@xCuri0 I can get up to 8gb that way, but only when I take out some of the RAM.
To enable GPU p2p communication via pcie (thanks to geohotz recent driver mod), I need 32gb BAR per gpu (preferably 4 of them).
I was reading here about the 36 bit limit but in reality Haswell has a 39 bit addressable space which led me to the dsdt modifications. I figured since the 39bit address space seems to be available in linux (whereas we are are only getting 36 bits in the bios), this might be a way to get access to the full 39 bits.
yeah the bios will only allocate 36-bit so you have to let linux do the allocation after dsdt mod to unlock 39-bit
@zcrypt0 the first output shows that the nvidia driver isn't resizing your gpus (it's still 256mb), there's a driver argument you need to make it do it. amd gpus do this automatically on linux
the plx bridge will get set properly then
I set the driver argument in that case and confirmed it was set with the appropriate commands (I don't have the link readily available, something like NVReg_EnableResizableBar=1
in a modprobe.d config file), my interpretation was that the driver was having issues with the 32bit pre-assigned space for the plx bridge.
System
My machine is same with the link https://github.com/xCuri0/ReBarUEFI/issues/11#issuecomment-1585579241. However, after patched my bios following the "Using-UEFIPatch" guide, and flushed the patched bios image, I can't boot my machine with my Nvidia Card pluged-in. I can't even boot to the bios setting UI. The screen has no signal and the number lock of the keyboard has no light.
The machine can boot with my nvidia card pluged-in using unpatched bios, but nvidia-smi reports "no device found error". The machine can't boot with my nvidia card pluged-in using patched bios, but can boot without my nvidia card.