[Question] Using PCIe Address Instead of Device Path

[JigaoLuo]

Hi,

I have a question regarding the nvme set-feature command. Is it possible to use the PCIe address of an NVMe SSD instead of the device path (e.g., /dev/nvme0)?

The reason I’m asking is that one of my SSDs is not being loaded by the NVMe driver, likely due to a feature I incorrectly set. I would like to reset or correct this feature by addressing the SSD directly via its PCIe address.

Is this possible, and if so, how can I proceed?

Thank you!

[igaw]

Sorry, if there is no device file, there is no way for userland to operate on a device.

What does the kernel log?

[JigaoLuo]

Hi @igaw, thanks for your response. Let me explain what I did.

I’m trying to enable the CMB (Controller Memory Buffer) on an NVMe SSD to set a vendor-specific feature using the following command: nvme set-feature /dev/nvme1 -f 0x<VendorFeature> -v 1. The setup was successful, and I proceeded to reboot the system.

However, after the reboot, the SSD with the enabled CMB could not be loaded by the NVMe driver due to a PCIe BAR space limitation:

$ sudo dmesg | grep c4:
[    2.230813] pci 0000:c4:00.0: [1e0f:0013] type 00 class 0x010802 PCIe Endpoint
[    2.230825] pci 0000:c4:00.0: BAR 0 [mem 0xd0010000-0xd001ffff 64bit]
[    2.230843] pci 0000:c4:00.0: BAR 4 [mem 0xc0000000-0xcfffffff 64bit]
[    2.230849] pci 0000:c4:00.0: ROM [mem 0xd0000000-0xd000ffff pref]
[    2.230922] pci 0000:c4:00.0: VF BAR 0 [mem 0x00000000-0x0000ffff 64bit]
[    2.230924] pci 0000:c4:00.0: VF BAR 0 [mem 0x00000000-0x001fffff 64bit]: contains BAR 0 for 32 VFs
[    2.297489] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: can't assign; no space
[    2.297491] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: failed to assign
[    2.297830] pci 0000:c4:00.0: BAR 4 [mem size 0x10000000 64bit]: can't assign; no space
[    2.297832] pci 0000:c4:00.0: BAR 4 [mem size 0x10000000 64bit]: failed to assign
[    2.297833] pci 0000:c4:00.0: BAR 0 [mem size 0x00010000 64bit]: can't assign; no space
[    2.297835] pci 0000:c4:00.0: BAR 0 [mem size 0x00010000 64bit]: failed to assign
[    2.297837] pci 0000:c4:00.0: ROM [mem 0xd9000000-0xd900ffff pref]: assigned
[    2.297839] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: can't assign; no space
[    2.297841] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: failed to assign
[    2.298255] pci 0000:c4:00.0: BAR 4 [mem size 0x10000000 64bit]: can't assign; no space
[    2.298257] pci 0000:c4:00.0: BAR 4 [mem size 0x10000000 64bit]: failed to assign
[    2.298259] pci 0000:c4:00.0: BAR 0 [mem size 0x00010000 64bit]: can't assign; no space
[    2.298261] pci 0000:c4:00.0: BAR 0 [mem size 0x00010000 64bit]: failed to assign
[    2.298262] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: can't assign; no space
[    2.298264] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: failed to assign
[    2.298266] pci 0000:c4:00.0: BAR 4 [mem size 0x10000000 64bit]: can't assign; no space
[    2.298268] pci 0000:c4:00.0: BAR 4 [mem size 0x10000000 64bit]: failed to assign
[    2.298270] pci 0000:c4:00.0: BAR 0 [mem size 0x00010000 64bit]: can't assign; no space
[    2.298272] pci 0000:c4:00.0: BAR 0 [mem size 0x00010000 64bit]: failed to assign
[    2.298273] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: can't assign; no space
[    2.298275] pci 0000:c4:00.0: VF BAR 0 [mem size 0x00200000 64bit]: failed to assign
[    2.299490] pci_bus 0000:c4: resource 0 [io  0x8000-0x8fff]
[    2.299491] pci_bus 0000:c4: resource 2 [mem 0xd9000000-0xd9ffffff 64bit pref]
[    2.306769] pci 0000:c4:00.0: Adding to iommu group 16

For comparison, the dmesg output of another standard SSD (without setup of CMB) looks fine:

$ sudo dmesg | grep c6:
[    2.231147] pci 0000:c6:00.0: [1e0f:0013] type 00 class 0x010802 PCIe Endpoint
[    2.231159] pci 0000:c6:00.0: BAR 0 [mem 0xd4010000-0xd401ffff 64bit]
[    2.231182] pci 0000:c6:00.0: ROM [mem 0xd4000000-0xd400ffff pref]
[    2.231255] pci 0000:c6:00.0: VF BAR 0 [mem 0x00000000-0x0000ffff 64bit]
[    2.231257] pci 0000:c6:00.0: VF BAR 0 [mem 0x00000000-0x001fffff 64bit]: contains BAR 0 for 32 VFs
[    2.297515] pci 0000:c6:00.0: VF BAR 0 [mem 0xd4020000-0xd421ffff 64bit]: assigned
[    2.298298] pci 0000:c6:00.0: BAR 0 [mem 0xc2900000-0xc290ffff 64bit]: assigned
[    2.298304] pci 0000:c6:00.0: ROM [mem 0xc2910000-0xc291ffff pref]: assigned
[    2.298306] pci 0000:c6:00.0: VF BAR 0 [mem 0xc2920000-0xc2b1ffff 64bit]: assigned
[    2.299493] pci_bus 0000:c6: resource 0 [io  0x9000-0x9fff]
[    2.299495] pci_bus 0000:c6: resource 1 [mem 0xc2900000-0xc3afffff]
[    2.299496] pci_bus 0000:c6: resource 2 [mem 0xd8000000-0xd8ffffff 64bit pref]
[    2.306798] pci 0000:c6:00.0: Adding to iommu group 17
[    4.212337] nvme nvme2: pci function 0000:c6:00.0

I’m currently looking for a workaround to load this NVMe device, whether with or without CMB enabled.

• One approach I’m considering is resetting this feature via the PCIe address, as I previously asked about.
• Another option might be modifying the BIOS to allocate the necessary BAR memory space. Then the NVMe SSD with CMB might be loaded by the nvme driver. If you have any suggestions or ideas, I’d really appreciate it. Thanks!

[keithbusch]

You can ask the kernel to try to reassign the memory windows instead of using the ones the platform firmware used. Parameters are "pci=realloc,nocrs".

[JigaoLuo]

thanks for your response.

I tested the following steps:

• Edited /etc/default/grub to include: GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci=realloc,nocrs"
• sudo update-grub
• reboot

However, the issue persists, and I'm still seeing the same error in the dmesg log. (I also did with "pci=realloc" and "pci=realloc=on" but still the same issue)

[igaw]

If the PCI address space reallocs trick do not work on this machine, maybe put the disk to another machine where the PCI address space is not exhausted by the drive and reset it there?

[JigaoLuo]

Thanks for the workaround. For our PCIe5 servers, we have identical setups across the board: the same AMD CPU, SSD, and other components. Because of that, I need to carefully consider applying an out-of-place fix.

Is there any method using nvmecli to generally reset an NVMe SSD to factory settings? If so, would that also reset the CMB enable register? For this particular NVMe SSD, I don’t need to preserve any data and can safely discard everything currently on it.

I also came across a suggestion related to a GPU BAR space limitation with a similar error message, which mentioned enabling "Above 4G Decoding" in the BIOS to map GPU memory above 4G in the PCIe BAR. While I’m not sure if this applies here, given that the NVMe CMB is 256 MB, it might not be the same cause ...

[keithbusch]

The prefetchable memory should be above 4GB if your platform allows it regardless of the BAR size. You have these types of memory attributes, so it's worth a shot and may free up memory ranges.

If you can't enumerate your device though, the nvmecli tool isn't going to be helpful.

[igaw]

I'm sorry not much we can do here. nvme-cli is only able to send commands to a device when it has been fully discovered by the kernel, which depends on the PCI subsystem to configure it correctly. Maybe the vendor of your device has a tool which allows to reset it.

[JigaoLuo]

Thank you for your assistance! I am currently inquiring with the SSD vendor about a firmware update and will reset the device once I receive their response. I appreciate your help and will keep you updated!