search

xcp-ng / xcp

Entry point for issues and wiki. Also contains some scripts and sources.
https://xcp-ng.org
1.25k stars 74 forks source link

Build AMD MxGPU and package them #200

Open olivierlambert opened 5 years ago

olivierlambert commented 5 years ago

https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization

AMD already build some RPMs for XS, and the package is free, but it seems it's only built for 4.4 kernel (see mxgpu-4.4.0+10-modules-1.0.5.amd-1.x86_64.rpm)

So we should be able to package that ourselves.

stormi commented 5 years ago

I can't find the source RPM so I'll have to try to build things manually from that github repo and see if it matches the contents of the RPM.

olivierlambert commented 5 years ago

The RPM is not in Citrix ISO but on AMD website, in the driver page, you'll have a "XenServer driver download". You can find the RPMs in there.

stormi commented 5 years ago

I had found it but there is no source RPM, only the binary one.

olivierlambert commented 5 years ago

Indeed, no SRPM. Maybe there's one for KVM.

armouredking commented 5 years ago

Was able to manually build the Github repo on XCP-ng with current kernel in v8 and load the gim module. I get the proper lspci output with the card showing all the virtual interfaces on PCI. But the control interfaces don't seem to know how to use it? I can't attach the GPUs to the VM in XOA or in Center, it just tells me I can passthrough the entire card but not the virtual adapters (Virtual GPU type in Center is just "Passthrough whole GPU").

stormi commented 4 years ago

@armouredking sorry for late answer. I haven't had the opportunity to test myself yet so I don't know what to expect.

stormi commented 4 years ago

Bookmark for myself: latest officially built MxGPU GIM (for XS >= 7.4 and < 8 ): https://www.amd.com/en/support/kb/release-notes/rn-pro-mxgpu-gim-1-05

stormi commented 4 years ago

@armouredking did you follow the deployment guide from the page linked above?

stormi commented 4 years ago

The mxgpu RPM that AMD distributed for XS 7.4+ contains more than what the github repository has:

.
|-- etc
|   |-- gim_config
|   `-- mxgpu-whitelist
|-- lib
|   `-- modules
|       `-- 4.4.0+10
|           `-- updates
|               |-- gim-api.ko
|               `-- gim.ko
`-- opt
    `-- xensource
        `-- bin
            `-- gimtool

Two configuration files, a tool named gimtool (with no headers saying what the license is, though the RPM License tag is GPLv2 so we can consider it's free).

I don't know what gim-api.ko is.

In https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization master builds fine with our kernel in XCP-ng 8.0, but this pull request may improve the compatibility. I'm not able to evaluate it and haven't the hardware to test. Edit: actually the pull request seems not necessary anymore. The code since this commit takes kernel >= 4.14 into account. There's an open pull request for kernel >= 5 though, but we aren't there yet.

Arvee42 commented 4 years ago

Confirmed. Can compile the github drivers on XCP-NG 8.0. Modprobe seems happy but can only pass through the GPU on an S7150x2. Had to yum install gcc git and kernel-devel to allow the gim.sh script to run.

Arvee42 commented 4 years ago

Compared lspci output from 8.0 and 7.6. 7.6 shows a lot more GPU devices versus only 2 "Tonga XT GL" GPUs on 8.0.

Dunno if that helps.

Arvee42 commented 4 years ago

Made some progress in 8.0. I now have the GUI showing 16 slices per GPU and they can be assigned to a VM. Upon starting the VM if fails with Failure "Call to gimtool failed". Gimtool is a python script so perhaps that wouldn't be too bad to modify if i knew what was wrong or how it is supposed to work.

/var/log/xensource shows the gimtool command that was called. Entering that command into the command line yields a bit more descriptive error:

INFO 2020-02-20 13:43:37,955 Calling PF reconfigure on 23:00.00 to 16 VFs with 480 FB size Traceback (most recent call last): File "/opt/xensource/bin/gimtool", line 93, in main(sys.argv[1:]) File "/opt/xensource/bin/gimtool", line 86, in main ret = fcntl.ioctl(gim_fd, RECONFIGURE_PF, gim_ioctl_reconfig_pf) IOError: [Errno 25] Inappropriate ioctl for device

What i did to get where I am now was: (paraphrased)

Install XCP-NG 8.0 from disc. Enable centos base repo in /etc/yum.repos.d/CentOS-Base.repo yum install gcc git kernel-devel git clone https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization/ change to MxGPU-Virtualization directory ./gim.sh

edit /etc/gim_config

fb_option=0 sched_option=0 vf_num=0 pf_fb=0 vf_fb=0 sched_interval=7 sched_interval_us=0 fb_clear=1 meout=100 max_quanta=1000 self_switch=500 exclusive=1600 fair_scheduling=0 debug_level=3 clear_fb_on_flr=0 clear_fb_on_free_vf=1

edit (add) /etc/mxgpu-whitelist (this changes the GPU GUI thing from passthrough to shareable for lack of a better explanation)

6929 experimental=0 name='MxGPU.1' framebuffer_sz=7904 vgpus_per_pgpu=1 6929 experimental=0 name='MxGPU.2' framebuffer_sz=3952 vgpus_per_pgpu=2 6929 experimental=0 name='MxGPU.4' framebuffer_sz=1968 vgpus_per_pgpu=4 6929 experimental=0 name='MxGPU.8' framebuffer_sz=976 vgpus_per_pgpu=8 6929 experimental=0 name='MxGPU.10' framebuffer_sz=784 vgpus_per_pgpu=10 6929 experimental=0 name='MxGPU.16' framebuffer_sz=480 vgpus_per_pgpu=16

add /etc/modules-load.d/gim.conf (load gim at boot)

gim

Copy gimtool from working 7.6 server to /opt/xensource/bin/

stormi commented 4 years ago

Thanks, keep us posted! Next I'd try to understand where that "Inappropriate ioctl for device" comes from (related to the RECONFIGURE_PF ioctl if I understand correctly).

sotiris-bos commented 4 years ago

INFO 2020-02-20 13:43:37,955 Calling PF reconfigure on 23:00.00 to 16 VFs with 480 FB size Traceback (most recent call last): File "/opt/xensource/bin/gimtool", line 93, in main(sys.argv[1:]) File "/opt/xensource/bin/gimtool", line 86, in main ret = fcntl.ioctl(gim_fd, RECONFIGURE_PF, gim_ioctl_reconfig_pf) IOError: [Errno 25] Inappropriate ioctl for device

Hello! Any progress on this? Did you try to patch the kernel as in https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization/blob/master/patch/0002-add-pci-io-access-cap-for-ubuntu-4.4.0-75-generic.diff

? (Line 8).

Arvee42 commented 4 years ago

I don't believe I ran that patch. It is all getting a bit beyond my skills. I also had to put the test box into production as a VMware box for now due to the pandemic.

Hopefully after things get close to normal I can return to this.

sotiris-bos commented 4 years ago

OK thanks! Please do update if you happen to try it again!

jcdick1 commented 4 years ago

What would be a good way to DM someone regarding this?

jcdick1 commented 4 years ago

I'm not a developer by any means, so if there's someone who can work with me on the nuts and bolts, I have a fresh 8.1 install with a Radeon Pro W5500 that can be used for getting mxgpu working.

dynodix commented 4 years ago

AMD has released a new iso with xenserver 8.1 compatibility found on a following link:

https://www.amd.com/en/support/professional-graphics/firepro/firepro-s-series/firepro-s7150-x2

Just testing

dynodix commented 4 years ago

Eny hint on how to deal with gim 0000:07:00.0: not enough MMIO resources for SR-IOV `

dmesg | grep gim

[ 9.176818] gim_api: loading out-of-tree module taints kernel. [ 9.180943] gim: module license 'Proprietary' taints kernel. [ 9.189926] gim info:(gim_init:197) ***AMD GIM init [ 9.189930] gim info:(print_gim_version:62) GPU IOV MODULE (GIM) - version 2.00.0000 [ 9.189931] gim info:(gim_init:200) Copyright (c) 2014-2016 AMD Corporation. [ 9.190150] gim error:(gim_probe:123) gim_probe(07:00.0) [ 12.520642] gim 0000:07:00.0: not enough MMIO resources for SR-IOV [ 12.520669] gim error:(EnableSriov:410) Fail to enable sriov, status = fffffff4 [ 12.520677] gim error:(SetNewAdapter:1263) Failed to properly enable SRIOV(map_image) !!!! [ 12.633285] gim error:(gim_probe:126) Failed to create new adapter [ 12.633306] gim: probe of 0000:07:00.0 failed with error -1 [ 12.633322] gim error:(gim_probe:123) gim_probe(09:00.0) [ 15.892534] gim 0000:09:00.0: not enough MMIO resources for SR-IOV [ 15.892548] gim error:(EnableSriov:410) Fail to enable sriov, status = fffffff4 [ 15.892556] gim error:(SetNewAdapter:1263) Failed to properly enable SRIOV(map_image) !!!! [ 16.004496] gim error:(gim_probe:126) Failed to create new adapter [ 16.004517] gim: probe of 0000:09:00.0 failed with error -1 `

sotiris-bos commented 4 years ago

Eny hint on how to deal with gim 0000:07:00.0: not enough MMIO resources for SR-IOV `

dmesg | grep gim

[ 9.176818] gim_api: loading out-of-tree module taints kernel. [ 9.180943] gim: module license 'Proprietary' taints kernel. [ 9.189926] gim info:(gim_init:197) ***AMD GIM init [ 9.189930] gim info:(print_gim_version:62) GPU IOV MODULE (GIM) - version 2.00.0000 [ 9.189931] gim info:(gim_init:200) Copyright (c) 2014-2016 AMD Corporation. [ 9.190150] gim error:(gim_probe:123) gim_probe(07:00.0) [ 12.520642] gim 0000:07:00.0: not enough MMIO resources for SR-IOV [ 12.520669] gim error:(EnableSriov:410) Fail to enable sriov, status = fffffff4 [ 12.520677] gim error:(SetNewAdapter:1263) Failed to properly enable SRIOV(map_image) !!!! [ 12.633285] gim error:(gim_probe:126) Failed to create new adapter [ 12.633306] gim: probe of 0000:07:00.0 failed with error -1 [ 12.633322] gim error:(gim_probe:123) gim_probe(09:00.0) [ 15.892534] gim 0000:09:00.0: not enough MMIO resources for SR-IOV [ 15.892548] gim error:(EnableSriov:410) Fail to enable sriov, status = fffffff4 [ 15.892556] gim error:(SetNewAdapter:1263) Failed to properly enable SRIOV(map_image) !!!! [ 16.004496] gim error:(gim_probe:126) Failed to create new adapter [ 16.004517] gim: probe of 0000:09:00.0 failed with error -1 `

https://support.citrix.com/article/CTX250121 ? I suppose you have SR-IOV enabled on the motherboard and OS, correct?

dynodix commented 4 years ago

Modifying BIOS: (This is a WS460 Gen 8 BLADE with Expansion) SR_IOV was enabled had to enable Additional params.

lspci | grep VGA 01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] 07:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:02.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:02.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:02.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:02.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:02.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:02.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:02.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 07:03.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150] 09:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:02.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:02.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:02.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:02.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:02.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:02.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:02.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V] 09:03.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]

dynodix commented 4 years ago

The card is visible shareable, configurable, but cannot configure it with drivers, the drivers fail to load, trying to get rid of: `dmesg | grep gim [ 7.608797] gim_api: loading out-of-tree module taints kernel. [ 7.612106] gim: module license 'Proprietary' taints kernel. [ 7.620859] gim info:(gim_init:197) ***AMD GIM init [ 7.620863] gim info:(print_gim_version:62) GPU IOV MODULE (GIM) - version 2.00.0000 [ 7.620866] gim info:(gim_init:200) Copyright (c) 2014-2016 AMD Corporation. [ 7.621069] gim error:(gim_probe:123) gim_probe(07:00.0) [ 11.099027] gim error:(gim_probe:123) gim_probe(09:00.0)

` gim_probe fails

rushikeshjadhav commented 4 years ago

Posting some intermediate results and error that I see. XCP-NG 8.1 with AMD provided CH8.1 drivers 

# lspci | grep VGA
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
26:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7341

# lsmod | grep gim
gim                  1351680  0 
gim_api                20480  1 gim

# cat /etc/mxgpu-whitelist 
6929 experimental=0 name='MxGPU.1' framebuffer_sz=7904 vgpus_per_pgpu=1
6929 experimental=0 name='MxGPU.2' framebuffer_sz=3952 vgpus_per_pgpu=2
6929 experimental=0 name='MxGPU.4' framebuffer_sz=1968 vgpus_per_pgpu=4
6929 experimental=0 name='MxGPU.8' framebuffer_sz=976 vgpus_per_pgpu=8
6929 experimental=0 name='MxGPU.10' framebuffer_sz=784 vgpus_per_pgpu=10
6929 experimental=0 name='MxGPU.16' framebuffer_sz=480 vgpus_per_pgpu=16

# cat /etc/gim_config 
fb_option=0
sched_option=0
vf_num=0
pf_fb=0
vf_fb=0
sched_interval=7
fb_clear=1
hang_detect_timeout=100
max_quanta=1000
self_switch=500
exclusive=1600
fair_scheduling=0
debug_level=3
clear_fb_on_flr=0
clear_fb_on_free_vf=1

# xe pgpu-list uuid=640ff2bd-effc-4833-9f1b-bc7f4805aadd 
uuid ( RO)              : 640ff2bd-effc-4833-9f1b-bc7f4805aadd
       vendor-name ( RO): Advanced Micro Devices, Inc. [AMD/ATI]
       device-name ( RO): Device 7341
    gpu-group-uuid ( RW): 36e2c2d1-27df-7c45-edc0-cfd3a34672b4

# xe pgpu-param-get uuid=640ff2bd-effc-4833-9f1b-bc7f4805aadd param-name=pci-id 
0000:26:00.0

# gimtool --bdf 0000:26:00.0 --num_vfs 4 --vf_fb_size 1968 
INFO 2020-07-29 00:53:39,551 Calling PF reconfigure on 26:00.00 to 4 VFs with 1968 FB size
INFO 2020-07-29 00:53:39,552 PF reconfigure complete

# dmesg | tail
[ 3620.781888]        gim error:(BdfToAdapter:184) WARNING: bdf 0x2600 cannot be found on any adapter
[ 3620.781929]        gim error:(gim_ioctl:428) Invalid BDF

XCP-NG 8.1 with https://github.com/GPUOpen-LibrariesAndSDKs/MxGPU-Virtualization

# gimtool --bdf 0000:26:00.0 --num_vfs 4 --vf_fb_size 1968 
INFO 2020-07-29 01:26:39,292 Calling PF reconfigure on 26:00.00 to 4 VFs with 1968 FB size
Traceback (most recent call last):
  File "/opt/xensource/bin/gimtool", line 93, in <module>
    main(sys.argv[1:])
  File "/opt/xensource/bin/gimtool", line 86, in main
    ret = fcntl.ioctl(gim_fd, RECONFIGURE_PF, gim_ioctl_reconfig_pf)
IOError: [Errno 25] Inappropriate ioctl for device
dynodix commented 4 years ago

I can confirm that a fresh installation of xcp-ng 8.1 and AMD iso from https://www.amd.com/en/support/professional-graphics/firepro/firepro-s-series/firepro-s7150-x2 works, the only thing is gim "modprobe gim " must be done manually , the automatic start procedure from the upper instaructions would never sucessfully start the server (not exiting boot maineteinance mode)

rushikeshjadhav commented 4 years ago

@dynodix can you share # lspci -k from your working setup? I'm interested in which driver is controlling the gpu.

Regarding having to load gim manually, you can create # echo gim > /etc/modules-load.d/gim.conf which will autoload it on next boot.

dynodix commented 4 years ago

here my lspci -k

00:00.0 Host bridge: Intel Corporation Xeon E5/Core i7 DMI2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
00:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 06)
    Kernel driver in use: pcieport
00:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 06)
    Kernel driver in use: pcieport
00:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 06)
    Kernel driver in use: pcieport
00:02.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2b (rev 06)
    Kernel driver in use: pcieport
00:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 06)
    Kernel driver in use: pcieport
00:02.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2d (rev 06)
    Kernel driver in use: pcieport
00:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 06)
    Kernel driver in use: pcieport
00:03.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3b (rev 06)
    Kernel driver in use: pcieport
00:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 06)
    Kernel driver in use: pcieport
00:03.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3d (rev 06)
    Kernel driver in use: pcieport
00:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
00:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
00:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
00:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 05)
    Kernel driver in use: pcieport
00:1a.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 (rev 05)
    Subsystem: Hewlett-Packard Company Device 18a9
    Kernel driver in use: ehci-pci
    Kernel modules: ehci_pci
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b5)
    Kernel driver in use: pcieport
00:1c.7 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 8 (rev b5)
    Kernel driver in use: pcieport
00:1d.0 USB controller: Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 (rev 05)
    Subsystem: Hewlett-Packard Company Device 18a9
    Kernel driver in use: ehci-pci
    Kernel modules: ehci_pci
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C600/X79 series chipset LPC Controller (rev 05)
    Kernel driver in use: lpc_ich
    Kernel modules: lpc_ich
01:00.0 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Slave Instrumentation & System Support (rev 05)
    Subsystem: Hewlett-Packard Company iLO4
01:00.1 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200EH
    Subsystem: Hewlett-Packard Company iLO4
01:00.2 System peripheral: Hewlett-Packard Company Integrated Lights-Out Standard Management Processor Support and Messaging (rev 05)
    Subsystem: Hewlett-Packard Company iLO4
    Kernel driver in use: hpilo
    Kernel modules: hpilo
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
    Subsystem: Hewlett-Packard Company iLO4
    Kernel driver in use: uhci_hcd
    Kernel modules: uhci_hcd
03:00.0 RAID bus controller: Hewlett-Packard Company Smart Array Gen8 Controllers (rev 01)
    Subsystem: Hewlett-Packard Company P220i
    Kernel driver in use: hpsa
    Kernel modules: hpsa
04:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 11)
    Subsystem: Hewlett-Packard Company Flex-10 10Gb 2-port 530FLB Adapter [Meru]
    Kernel driver in use: bnx2x
    Kernel modules: bnx2x
04:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM57810 10 Gigabit Ethernet (rev 11)
    Subsystem: Hewlett-Packard Company Flex-10 10Gb 2-port 530FLB Adapter [Meru]
    Kernel driver in use: bnx2x
    Kernel modules: bnx2x
05:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
    Kernel driver in use: pcieport
06:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
    Kernel driver in use: pcieport
06:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
    Kernel driver in use: pcieport
07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
    Subsystem: Dell Device 1d35
    Kernel driver in use: gim
07:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
    Kernel driver in use: pciback
07:02.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:02.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:02.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:02.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:02.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:02.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:02.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
07:03.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XT GL [FirePro S7150]
    Subsystem: Dell Device 1d35
    Kernel driver in use: gim
09:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
    Kernel driver in use: pciback
09:02.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
    Kernel driver in use: pciback
09:02.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:02.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:02.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:02.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:02.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:02.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.4 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.5 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.6 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
09:03.7 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tonga XTV GL [FirePro S7150V]
1f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:08.2 Performance counters: Intel Corporation Device 3c41 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:08.6 System peripheral: Intel Corporation Device 3c86 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:09.2 Performance counters: Intel Corporation Device 3c42 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:09.6 System peripheral: Intel Corporation Device 3c96 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
1f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
20:00.0 PCI bridge: Intel Corporation Xeon E5/Core i7 DMI2 in PCI Express Mode (rev 06)
    Kernel driver in use: pcieport
20:01.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1a (rev 06)
    Kernel driver in use: pcieport
20:01.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 1b (rev 06)
    Kernel driver in use: pcieport
20:02.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2a (rev 06)
    Kernel driver in use: pcieport
20:02.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2b (rev 06)
    Kernel driver in use: pcieport
20:02.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2c (rev 06)
    Kernel driver in use: pcieport
20:02.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 2d (rev 06)
    Kernel driver in use: pcieport
20:03.0 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3a in PCI Express Mode (rev 06)
    Kernel driver in use: pcieport
20:03.1 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3b (rev 06)
    Kernel driver in use: pcieport
20:03.2 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3c (rev 06)
    Kernel driver in use: pcieport
20:03.3 PCI bridge: Intel Corporation Xeon E5/Core i7 IIO PCI Express Root Port 3d (rev 06)
    Kernel driver in use: pcieport
20:04.0 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:04.1 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:04.2 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:04.3 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:04.4 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 4 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:04.5 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 5 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:04.6 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 6 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:04.7 System peripheral: Intel Corporation Xeon E5/Core i7 DMA Channel 7 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
    Kernel driver in use: ioatdma
20:05.0 System peripheral: Intel Corporation Xeon E5/Core i7 Address Map, VTd_Misc, System Management (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
20:05.2 System peripheral: Intel Corporation Xeon E5/Core i7 Control Status and Global Errors (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
20:05.4 PIC: Intel Corporation Xeon E5/Core i7 I/O APIC (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:08.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:08.2 Performance counters: Intel Corporation Device 3c41 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:08.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:08.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:08.6 System peripheral: Intel Corporation Device 3c86 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:09.0 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:09.2 Performance counters: Intel Corporation Device 3c42 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:09.3 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:09.4 System peripheral: Intel Corporation Xeon E5/Core i7 QPI Link Reut 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:09.6 System peripheral: Intel Corporation Device 3c96 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0a.0 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0a.1 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0a.2 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0a.3 System peripheral: Intel Corporation Xeon E5/Core i7 Power Control Unit 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0b.0 System peripheral: Intel Corporation Xeon E5/Core i7 Interrupt Control Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0b.3 System peripheral: Intel Corporation Xeon E5/Core i7 Semaphore and Scratchpad Configuration Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0c.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0c.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0c.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0c.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0c.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0c.7 System peripheral: Intel Corporation Xeon E5/Core i7 System Address Decoder (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0d.0 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0d.1 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0d.2 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0d.3 System peripheral: Intel Corporation Xeon E5/Core i7 Unicast Register 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0d.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller System Address Decoder 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0e.0 System peripheral: Intel Corporation Xeon E5/Core i7 Processor Home Agent (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0e.1 Performance counters: Intel Corporation Xeon E5/Core i7 Processor Home Agent Performance Monitoring (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0f.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0f.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller RAS Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0f.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0f.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0f.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0f.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:0f.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Target Address Decoder 4 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.0 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.1 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.2 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 0 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.3 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 1 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.4 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.5 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller Channel 0-3 Thermal Control 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.6 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 2 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:10.7 System peripheral: Intel Corporation Xeon E5/Core i7 Integrated Memory Controller ERROR Registers 3 (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:11.0 System peripheral: Intel Corporation Xeon E5/Core i7 DDRIO (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:13.0 System peripheral: Intel Corporation Xeon E5/Core i7 R2PCIe (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:13.1 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to PCI Express Performance Monitor (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:13.4 Performance counters: Intel Corporation Xeon E5/Core i7 QuickPath Interconnect Agent Ring Registers (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:13.5 Performance counters: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 0 Performance Monitor (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
3f:13.6 System peripheral: Intel Corporation Xeon E5/Core i7 Ring to QuickPath Interconnect Link 1 Performance Monitor (rev 06)
    Subsystem: Hewlett-Packard Company Device 18a8
dynodix commented 4 years ago

Another hint, I had first setup the fresh instalation of XCP-NG 8.1 , then installed AMD ISO 2.0 as standalone machine (not into the pool) , updated with yum update , and after the update I had added the machine to my pool. The xe-install-supplemental-pack behaved diferently this way.

dynodix commented 4 years ago

Windows 10 with MxGPU drivers now behave stable and normally, now I m trying to use the GPU for transcoding purposes under ubuntu 18 but have problems initializing the card driver and obtain valid info with vainfo

nagilum99 commented 4 years ago

AFAIK: All individual changes should happen after pool integration as it might result into problems, if you do it before. There have been several issues in XS history with such things.

rushikeshjadhav commented 4 years ago

Radeon Pro W5500

Result with AMD offered driver gim.ko

[35637.728282] AMD Virt GIM API
[35637.737022] gim: module license 'Proprietary' taints kernel.
[35637.737026] Disabling lock debugging due to kernel taint
[35637.746073]        gim info:(gim_init:197) *******AMD GIM init
[35637.746076]        gim info:(print_gim_version:62) GPU IOV MODULE (GIM) - version 2.00.0000
[35637.746078]        gim info:(gim_init:200) Copyright (c) 2014-2016 AMD Corporation.
[35637.749241]        gim info:(parse_config_file:295) AMD GIM fb_option = 0
[35637.749246]        gim info:(parse_config_file:295) AMD GIM sched_option = 0
[35637.749248]        gim info:(parse_config_file:295) AMD GIM vf_num = 0
[35637.749250]        gim info:(parse_config_file:295) AMD GIM pf_fb = 0
[35637.749253]        gim info:(parse_config_file:295) AMD GIM vf_fb = 0
[35637.749255]        gim info:(parse_config_file:295) AMD GIM sched_interval = 7
[35637.749258]        gim info:(parse_config_file:295) AMD GIM fb_clear = 1
[35637.749261]        gim info:(parse_config_file:295) AMD GIM hang_detect_timeout = 100
[35637.749264]        gim info:(parse_config_file:295) AMD GIM max_quanta = 1000
[35637.749267]        gim info:(parse_config_file:295) AMD GIM self_switch = 500
[35637.749270]        gim info:(parse_config_file:295) AMD GIM exclusive = 1600
[35637.749274]        gim info:(parse_config_file:295) AMD GIM fair_scheduling = 0
[35637.749277]        gim info:(parse_config_file:295) AMD GIM debug_level = 3
[35637.749280]        gim info:(parse_config_file:295) AMD GIM clear_fb_on_flr = 0
[35637.749283]        gim info:(parse_config_file:295) AMD GIM clear_fb_on_free_vf = 1
[35637.749287]        gim info:(init_config:445) INIT CONFIG
[35637.758619]        gim info:(gim_ioctl_init:567) IOCTL device created and ready for use
[35637.758622] Running Kaveri version of GIM

Result with modified open source gim module having additional support for PCI ID 1002:7341 RadeonPro is below. By default gim.ko only recognizes 1002:6929 FirePro.

[207204.092055] gim info:(gim_init:150) Start AMD open source GIM initialization
[207204.092059] gim info:(gim_init:153) GPU IOV MODULE - version 1.1.4
[207204.092061] gim info:(gim_init:155) Copyright (c) 2014-2017 Advanced Micro Devices, Inc. All rights reserved.
[207204.092097] gim info:(parse_config_file:219) AMD GIM fb_option = 0
[207204.092100] gim info:(parse_config_file:219) AMD GIM sched_option = 0
[207204.092102] gim info:(parse_config_file:219) AMD GIM vf_num = 0
[207204.092104] gim info:(parse_config_file:219) AMD GIM pf_fb = 0
[207204.092106] gim info:(parse_config_file:219) AMD GIM vf_fb = 0
[207204.092108] gim info:(parse_config_file:219) AMD GIM sched_interval = 7
[207204.092111] gim info:(parse_config_file:219) AMD GIM sched_interval_us = 0
[207204.092113] gim info:(parse_config_file:219) AMD GIM fb_clear = 1
[207204.092147] gim info:(init_config:341) INIT CONFIG
[207204.093000] gim info:(set_new_adapter:572) curr allocated at 00000000dba9b50c
[207204.093002] gim error:(set_new_adapter:575) **asic does not support SRIOV**
[207204.093027] gim info:(gim_probe:92) AMD GIM probe: pf_count = 0

Do you guys know if Radeon Pro is supported?

ethanjosephscott commented 4 years ago

Hey @rushikeshjadhav the Radeon Pro W5500 I believe does not support MXGPU. The current list as I understand it is below.

Radeon Instinct MI6 Radeon Instinct MI8 Radeon Instinct MI25 Radeon Instinct MI50 Radeon Instinct MI60 Radeon Instinct MI100 Radeon Pro V320 Radeon Pro V340 AMD FirePro S7100X AMD FirePro s7150 AMD FirePro s7150 x2

olivierlambert commented 4 years ago

Some of them are somehow affordable to put in our lab. /me thinking on getting some extra hardware.

rushikeshjadhav commented 4 years ago

@ethanjosephscott Thanks for providing supported cards list.

savage1279 commented 4 years ago

Windows 10 with MxGPU drivers now behave stable and normally, now I m trying to use the GPU for transcoding purposes under ubuntu 18 but have problems initializing the card driver and obtain valid info with vainfo

For me Win10 has no video decoding/encoding in GPU info gpu

Is it normal behaviour for s7150 driver? does it support in general video processing?

dynodix commented 4 years ago

According to: https://en.wikipedia.org/wiki/Video_Coding_Engine

S7150 x2 should support VCE3.0 with h264 encoding decoding...

But how? ...

savage1279 commented 4 years ago

According to: https://en.wikipedia.org/wiki/Video_Coding_Engine

S7150 x2 should support VCE3.0 with h264 encoding decoding...

But how? ...

Yes. By specification it has to. But actually I tried to use ffmpeg with VAAPI in ubuntu. No chances to work. I tried it with AMF with custom build of ffmeg. The same result - It doesn't work. In Windows also the same result and the same errors.

savage1279 commented 4 years ago

I found there the problem was. It's gim driver of MxGPU. It doesn't support video processing. I switched from MxGPU to PCI passthrough

rmmod gim

xe vm-param-set other-config:pci=0/0000:67:00.0 uuid=21cd7312-d9bf-aab8-b69e-913f60983ff7

After that GPU starts working as needed. In Windows:

gpu2

In Ubuntu ffmpeg now can use VAAPI: image

jcdick1 commented 4 years ago

According to a recent post in the AMD community forums:

"When you say MxGPU works, I assume you mean you have enabled MxGPU. If so, then you will not get VCE on each VM. VCE on FirePro S7150 X2 works only if you use the entire GPU."

nagilum99 commented 4 years ago

AFAIK Technically it can't work: You have less en-/decoders than partitions per GPU and they probably can't just split that parts (and I doubt there is a big demand worth the effort).

thatsysadmin commented 3 years ago

Hey @rushikeshjadhav the Radeon Pro W5500 I believe does not support MXGPU. The current list as I understand it is below.

Radeon Instinct MI6 Radeon Instinct MI8 Radeon Instinct MI25 Radeon Instinct MI50 Radeon Instinct MI60 Radeon Instinct MI100 Radeon Pro V320 Radeon Pro V340 AMD FirePro S7100X AMD FirePro s7150 AMD FirePro s7150 x2

Sorry for reviving an 8 month old thread, but may I ask if there is any official documentation that says that the MI100 supports MxGPU?

ethanjosephscott commented 3 years ago

@thatsysadmin

Hey I made that list before the MI100 was officially announced, not sure how I knew about it? But I believe I was going under the impression all of the instinct cards would have mxgpu support. It looks like the radeon pro v520 is the newest mxgpu card (not sold to the public).

nhdnet commented 4 months ago

Radeon Pro W5500

Result with AMD offered driver gim.ko

[35637.728282] AMD Virt GIM API
[35637.737022] gim: module license 'Proprietary' taints kernel.
[35637.737026] Disabling lock debugging due to kernel taint
[35637.746073]        gim info:(gim_init:197) *******AMD GIM init
[35637.746076]        gim info:(print_gim_version:62) GPU IOV MODULE (GIM) - version 2.00.0000
[35637.746078]        gim info:(gim_init:200) Copyright (c) 2014-2016 AMD Corporation.
[35637.749241]        gim info:(parse_config_file:295) AMD GIM fb_option = 0
[35637.749246]        gim info:(parse_config_file:295) AMD GIM sched_option = 0
[35637.749248]        gim info:(parse_config_file:295) AMD GIM vf_num = 0
[35637.749250]        gim info:(parse_config_file:295) AMD GIM pf_fb = 0
[35637.749253]        gim info:(parse_config_file:295) AMD GIM vf_fb = 0
[35637.749255]        gim info:(parse_config_file:295) AMD GIM sched_interval = 7
[35637.749258]        gim info:(parse_config_file:295) AMD GIM fb_clear = 1
[35637.749261]        gim info:(parse_config_file:295) AMD GIM hang_detect_timeout = 100
[35637.749264]        gim info:(parse_config_file:295) AMD GIM max_quanta = 1000
[35637.749267]        gim info:(parse_config_file:295) AMD GIM self_switch = 500
[35637.749270]        gim info:(parse_config_file:295) AMD GIM exclusive = 1600
[35637.749274]        gim info:(parse_config_file:295) AMD GIM fair_scheduling = 0
[35637.749277]        gim info:(parse_config_file:295) AMD GIM debug_level = 3
[35637.749280]        gim info:(parse_config_file:295) AMD GIM clear_fb_on_flr = 0
[35637.749283]        gim info:(parse_config_file:295) AMD GIM clear_fb_on_free_vf = 1
[35637.749287]        gim info:(init_config:445) INIT CONFIG
[35637.758619]        gim info:(gim_ioctl_init:567) IOCTL device created and ready for use
[35637.758622] Running Kaveri version of GIM

Result with modified open source gim module having additional support for PCI ID 1002:7341 RadeonPro is below. By default gim.ko only recognizes 1002:6929 FirePro.

[207204.092055] gim info:(gim_init:150) Start AMD open source GIM initialization
[207204.092059] gim info:(gim_init:153) GPU IOV MODULE - version 1.1.4
[207204.092061] gim info:(gim_init:155) Copyright (c) 2014-2017 Advanced Micro Devices, Inc. All rights reserved.
[207204.092097] gim info:(parse_config_file:219) AMD GIM fb_option = 0
[207204.092100] gim info:(parse_config_file:219) AMD GIM sched_option = 0
[207204.092102] gim info:(parse_config_file:219) AMD GIM vf_num = 0
[207204.092104] gim info:(parse_config_file:219) AMD GIM pf_fb = 0
[207204.092106] gim info:(parse_config_file:219) AMD GIM vf_fb = 0
[207204.092108] gim info:(parse_config_file:219) AMD GIM sched_interval = 7
[207204.092111] gim info:(parse_config_file:219) AMD GIM sched_interval_us = 0
[207204.092113] gim info:(parse_config_file:219) AMD GIM fb_clear = 1
[207204.092147] gim info:(init_config:341) INIT CONFIG
[207204.093000] gim info:(set_new_adapter:572) curr allocated at 00000000dba9b50c
[207204.093002] gim error:(set_new_adapter:575) **asic does not support SRIOV**
[207204.093027] gim info:(gim_probe:92) AMD GIM probe: pf_count = 0

Do you guys know if Radeon Pro is supported?

I modified open source GIM and it works with a instinct mi6

` 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro V7300X / V7350x2] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0734 01:02.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro V7300X / V7350x2] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0734 01:02.1 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro V7300X / V7350x2] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0734 01:02.2 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro V7300X / V7350x2] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0734 01:02.3 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon Pro V7300X / V7350x2] Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device 0734

`

nagilum99 commented 4 months ago

The Radeon Pro ist, as the name says, based on Radeon chips. It's a whole different chip than the Instinct/MI ones. You can't 'split' Radeon chips.

nhdnet commented 4 months ago

its a instinct mi6, believe me. "lspci" command shows "radeon pro" but its a instinct mi6

nagilum99 commented 4 months ago

You're right, the old/first generations of Instinct were using the radeon chips (I just looked it up).