StevenPontsler
commented
3 years ago Thanks for the report. We will look into it.
Open altexa opened 3 years ago
StevenPontsler
commented
3 years ago Thanks for the report. We will look into it.
dimstav23
commented
2 years ago Hello, I'd like to ask if there is any updated related to this issue. It would be good to know the assiged numa_node for each of the namespaces.
Thank you!
nolanhergert
commented
2 years ago Hi Dimitris, sorry for the delay. It sounds like a BIOS issue, I'm going to look into it internally first. Steven is getting back from sabbatical next week so I'll see what he was trying to do before as well. You should get an update by next Wednesday.
nolanhergert
commented
2 years ago I tried reproducing this on my 2-socket reference platform with a BIOS from mid-2020 with Fedora 27 and the region creation process is working fine for me. I assume namespaces and formatting would not modify the below:
ndctl list -Rv
[
{
"dev":"region1",
"size":3234110373888,
"available_size":3234110373888,
"type":"pmem",
"numa_node":1,
"iset_id":5481583643840687308,
"persistence_domain":"memory_controller"
},
{
"dev":"region0",
"size":3234110373888,
"available_size":3234110373888,
"type":"pmem",
"numa_node":0,
"iset_id":4183984003704433868,
"persistence_domain":"memory_controller"
}
]It is sounding like the SRAT ACPI table specifies these proximity domains, and it's possible your BIOS vendor didn't implement this properly. Can you double check this?
dimstav23
commented
2 years ago Hi Nolan, thanks for the quick response. Here is my reference system information: BIOS release date: 15/02/2022 System: x86_64-linux Host os: Linux 5.15.41, NixOS, 22.05 (Quokka), 22.05.20220520.dfd8298
lscpu | grep NUMA
NUMA node(s): 2
NUMA node0 CPU(s): 0-15,32-47
NUMA node1 CPU(s): 16-31,48-63The memory topology of the system:
ipmctl show -topology
DimmID | MemoryType | Capacity | PhysicalID| DeviceLocator
================================================================================
0x0000 | Logical Non-Volatile Device | 256.000 GiB | 0x0029 | P1-DIMMA1
0x0100 | Logical Non-Volatile Device | 256.000 GiB | 0x002d | P1-DIMMC1
0x0200 | Logical Non-Volatile Device | 256.000 GiB | 0x0031 | P1-DIMME1
0x0300 | Logical Non-Volatile Device | 256.000 GiB | 0x0035 | P1-DIMMG1
0x1000 | Logical Non-Volatile Device | 256.000 GiB | 0x0039 | P2-DIMMA1
0x1100 | Logical Non-Volatile Device | 256.000 GiB | 0x003d | P2-DIMMC1
0x1200 | Logical Non-Volatile Device | 256.000 GiB | 0x0041 | P2-DIMME1
0x1300 | Logical Non-Volatile Device | 256.000 GiB | 0x0045 | P2-DIMMG1
N/A | DDR4 | 16.000 GiB | 0x002b | P1-DIMMB1
N/A | DDR4 | 16.000 GiB | 0x002f | P1-DIMMD1
N/A | DDR4 | 16.000 GiB | 0x0033 | P1-DIMMF1
N/A | DDR4 | 16.000 GiB | 0x0037 | P1-DIMMH1
N/A | DDR4 | 16.000 GiB | 0x003b | P2-DIMMB1
N/A | DDR4 | 16.000 GiB | 0x003f | P2-DIMMD1
N/A | DDR4 | 16.000 GiB | 0x0043 | P2-DIMMF1
N/A | DDR4 | 16.000 GiB | 0x0047 | P2-DIMMH1This is the ndctl list -Rv output. As you can see both regions are assigned to numa=0.
ndctl list -Rv
{
"regions":[
{
"dev":"region1",
"size":1086626725888,
"align":16777216,
"available_size":0,
"max_available_extent":0,
"type":"pmem",
"numa_node":0,
"target_node":3,
"iset_id":-1259385221858424696,
"persistence_domain":"memory_controller",
"namespaces":[
{
"dev":"namespace1.0",
"mode":"fsdax",
"map":"dev",
"size":1069646086144,
"uuid":"f444d776-ede4-4be0-a044-a4b9d96e325f",
"raw_uuid":"ed0c5d01-f47d-4ab2-b98a-9b96567947c2",
"sector_size":512,
"align":2097152,
"blockdev":"pmem1",
"numa_node":0,
"target_node":3
}
]
},
{
"dev":"region0",
"size":1086626725888,
"align":16777216,
"available_size":0,
"max_available_extent":0,
"type":"pmem",
"numa_node":0,
"target_node":2,
"iset_id":-1873000667866822520,
"persistence_domain":"memory_controller",
"namespaces":[
{
"dev":"namespace0.0",
"mode":"fsdax",
"map":"dev",
"size":1069646086144,
"uuid":"91c69823-7d74-49fb-a580-ef6a88947e9f",
"raw_uuid":"24c7403f-064b-4f56-b770-61919855e789",
"sector_size":512,
"align":2097152,
"blockdev":"pmem0",
"numa_node":0,
"target_node":2
}
]
}
]
}I am not really familiar with the SRAT ACPI table but I found out here that it might be disabled if Node Memory Interleaving is enabled (which I suppose is the case for me). I will double check it on Monday and get back to you.
dimstav23
commented
2 years ago I am not sure whether it makes sense or not but in my case the SRAT ACPI table does not mark any region as Non-Volatile.
Does this mean that there is a mistake there that might affect the shown information from ndctl?
From what I see, even in the PM section of the motherboard manual, they do not refer to node information related to PM in the bios.
I extracted the SRAT ACPI table information by using:
acpidump > acpi.dat
acpixtract -sSRAT acpi.dat
iasl -d srat.dat
cat srat.dslIf you have any further hints or things that I could try out, please let me know.
In the meantime, I performed some straighforward benchmark measurements and when I pin a process to the first socket's CPU and access pmem0. the latency is lower than when accessing pmem1. So I can assume that region 0 is in node 0 while region 1 in node 1.
nolanhergert
commented
2 years ago Interesting. I did the same steps and found two entries with Non-Volatile set to 1 (true). I assume that corresponds to my two regions.
It is mounting the
Mine looks to be set correctly as it is changing from region to region and more importantly is listed by ndctl correctly in the end.
[EA8h 3752 1] Subtable Type : 01 [Memory Affinity]
[EA9h 3753 1] Length : 28
[EAAh 3754 4] Proximity Domain : 00000002
[EAEh 3758 2] Reserved1 : 0000
[EB0h 3760 8] Base Address : 0000001C80000000
[EB8h 3768 8] Address Length : 000002F100000000
[EC0h 3776 4] Reserved2 : 00000000
[EC4h 3780 4] Flags (decoded below) : 00000005
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 1
[EC8h 3784 8] Reserved3 : 0000000000000000
...
[F20h 3872 1] Subtable Type : 01 [Memory Affinity]
[F21h 3873 1] Length : 28
[F22h 3874 4] Proximity Domain : 00000003
[F26h 3878 2] Reserved1 : 0000
[F28h 3880 8] Base Address : 00000329A0000000
[F30h 3888 8] Address Length : 000002F100000000
[F38h 3896 4] Reserved2 : 00000000
[F3Ch 3900 4] Flags (decoded below) : 00000005
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 1
[F40h 3904 8] Reserved3 : 0000000000000000I will check the PM mappings and their respective entries in the ACPI table and report what I get. As you correctly pointed out, I suppose it has to do with the vendor's BIOS.
dimstav23
commented
2 years ago After looking into that further, I identified these regions in my ACPI table. The proximity domain field seems to be set correctly.
[2080h 8320 1] Subtable Type : 01 [Memory Affinity]
[2081h 8321 1] Length : 28
[2082h 8322 4] Proximity Domain : 00000002
[2086h 8326 2] Reserved1 : 0000
[2088h 8328 8] Base Address : 0000002080000000
[2090h 8336 8] Address Length : 000000FD00000000
[2098h 8344 4] Reserved2 : 00000000
[209Ch 8348 4] Flags (decoded below) : 00000005
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 1
[20A0h 8352 8] Reserved3 : 0000000000000000
[20D0h 8400 1] Subtable Type : 01 [Memory Affinity]
[20D1h 8401 1] Length : 28
[20D2h 8402 4] Proximity Domain : 00000003
[20D6h 8406 2] Reserved1 : 0000
[20D8h 8408 8] Base Address : 0000011D80000000
[20E0h 8416 8] Address Length : 000000FD00000000
[20E8h 8424 4] Reserved2 : 00000000
[20ECh 8428 4] Flags (decoded below) : 00000005
Enabled : 1
Hot Pluggable : 0
Non-Volatile : 1
[20F0h 8432 8] Reserved3 : 0000000000000000So, this field is not the one that causes this issue in the end.
Any further hints would be welcome to solve this weird mystery.
nolanhergert
commented
2 years ago Ah, interesting. I emailed the developers of ndctl and they'll hopefully chime in here tomorrow.
stellarhopper
commented
2 years ago The numa_node ndctl is reporting ultimately comes from the ACPI NFIT table's SPA Range structure. @dimstav23 can you dump those entries and check whether they are 0?
sscargal
commented
2 years ago @dimstav23 - The NFIT Table has pointers to other tables described in 5.2.25. NVDIMM Firmware Interface Table (NFIT)
ipmctl show -system [PCAT|NFIT|PMTT] can be used to gather the information requested by stellarhopper. Attaching the output from ipmctl show -system > ipmctl_show_-system.out would be useful.
dimstav23
commented
2 years ago Hi @stellarhopper and @sscargal,
sorry for the delay.
Here is the output file from ipmctl show -system
ipmctlshow-system.log
From a quick look, I could see that SpaRangeTablesNum: 2 and the respective table index hex values are 0x1 and 0x2.
However, you probably know better how this is handled.
Let me know if you require further details or you are able to identify the error.
Thank you for the help :)
stellarhopper
commented
2 years ago @dimstav23
---TableType=0x0
Length: 56 bytes
TypeEquals: SpaRange
AddressRangeType: 66f0d379-b4f3-4074-ac43-0d3318b78cdb
SpaRangeDescriptionTableIndex: 0x1
Flags: 0x0
ProximityDomain: 0x0
SystemPhysicalAddressRangeBase: 0x2080000000
SystemPhysicalAddressRangeLength: 0xfd00000000
MemoryMappingAttribute: 0x8008
---TableType=0x0
Length: 56 bytes
TypeEquals: SpaRange
AddressRangeType: 66f0d379-b4f3-4074-ac43-0d3318b78cdb
SpaRangeDescriptionTableIndex: 0x2
Flags: 0x0
ProximityDomain: 0x0
SystemPhysicalAddressRangeBase: 0x11d80000000
SystemPhysicalAddressRangeLength: 0xfd00000000
MemoryMappingAttribute: 0x8008Looks like proximity domain is set to 0 - I think this would be a question for the BIOS vendor.
stellarhopper
commented
2 years ago I think SpaRangeTablesNum: 2 Just implies that there are two SpaRange type tables. The actual tables are above.
dimstav23
commented
2 years ago Good point. I will get back to them to ask about it. This is irrelevant to the fact that the ACPI tables indicate different proximity domains as pointed above, right?
sscargal
commented
2 years ago Good point. I will get back to them to ask about it. This is irrelevant to the fact that the ACPI tables indicate different proximity domains as pointed above, right?
The data provided has been provided from the 2-Socket Supermicro system. Can you provide the same data from you 8-Socket system too please? I noted that your 8-Socket system is Cascade Lake and your 2-Socket looks to be Ice Lake (a best guess based on 8 DIMM Slots). These have different BIOS code bases, so it would be very odd for both platforms to have the same BIOS issue. I have several Ice Lake Supermicro systems (X12DPU-6) that work correctly.
Q) What's the manufacturer & model of your 2- and 8-Socket system? (dmidecode -t baseboard)
Q) What's the BIOS release from both systems? (dmidecode -t bios)
Q) Do both systems run NixOS? Do you have an opportunity to temporarily try a different distro such as Fedora Server 36? (I doubt this is the issue, but I'm not familiar with that distro and if it customizes the Kernel. Fedora uses a mainline Kernel).
dimstav23
commented
2 years ago Hi @sscargal,
the 8-socket system is not in my reach. It was reported by another user a long time ago.
I just came across this reported github issue while googling for a solution.
I can provide my 2-socket system specification:
Q1) demidecode -t baseboard:
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
# SMBIOS implementations newer than version 3.2.0 are not
# fully supported by this version of dmidecode.
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Supermicro
Product Name: X12DPU-6
Version: 1.02A
Serial Number: HM218S018239
Asset Tag: Base Board Asset Tag
Features:
Board is a hosting board
Board is replaceable
Location In Chassis: Part Component
Chassis Handle: 0x0003
Type: Motherboard
Contained Object Handles: 0
Handle 0x000E, DMI type 41, 11 bytes
Onboard Device
Reference Designation: ASPEED Video AST2600
Type: Video
Status: Enabled
Type Instance: 1
Bus Address: 0000:04:00.0Q2)dmidecode -t bios:
# dmidecode 3.2
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
# SMBIOS implementations newer than version 3.2.0 are not
# fully supported by this version of dmidecode.
Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
Vendor: American Megatrends International, LLC.
Version: 1.2
Release Date: 02/15/2022
Address: 0xF0000
Runtime Size: 64 kB
ROM Size: 32 MB
Characteristics:
PCI is supported
BIOS is upgradeable
BIOS shadowing is allowed
Boot from CD is supported
Selectable boot is supported
BIOS ROM is socketed
EDD is supported
Japanese floppy for NEC 9800 1.2 MB is supported (int 13h)
Japanese floppy for Toshiba 1.2 MB is supported (int 13h)
5.25"/360 kB floppy services are supported (int 13h)
5.25"/1.2 MB floppy services are supported (int 13h)
3.5"/720 kB floppy services are supported (int 13h)
3.5"/2.88 MB floppy services are supported (int 13h)
Print screen service is supported (int 5h)
Serial services are supported (int 14h)
Printer services are supported (int 17h)
CGA/mono video services are supported (int 10h)
USB legacy is supported
BIOS boot specification is supported
Targeted content distribution is supported
UEFI is supported
BIOS Revision: 5.22Q3) Currently it's not that simple to try a different distro in this machine as it is managed centrally and people are having some workloads. If I find a time-window I'll give it a try to see whether this causes any issue.
In the meantime, maybe @altexa could provide his dmidecode outputs to clear this up.
Thanks for your time!
sscargal
commented
2 years ago @dimstav23 - Sorry for the confusion on the 8-Socket.
I have the same Supermicro X12DPU-6 server running Ubuntu 22.04 with 5.15.0-30-generic. My BIOS is older 'Release Date: 04/21/2021', so I'll update it to see if this changes anything. If not, I'll try to install NixOS (minimum) to see if that replicates your issue.
sscargal
commented
2 years ago @dimstav23 There seems to be a BIOS regression problem that needs to be filed with Supermicro. The only thing I changed was updating the BIOS from v1.1 (Release Date: 04/21/2021) to v1.2 (Release Date: 04/21/2021) and I now see the same issue as you. No other changes were made to the OS or PMem config.
Working (BIOS v1.1):
# dmidecode -t bios
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
Vendor: American Megatrends International, LLC.
Version: 1.1
Release Date: 04/21/2021
[...]
# dmidecode -t bios
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
Vendor: American Megatrends International, LLC.
Version: 1.1
Release Date: 04/21/2021
[...]
# ndctl list -Rv
{
"regions":[
{
"dev":"region1",
"size":1082331758592,
"align":16777216,
"available_size":0,
"max_available_extent":0,
"type":"pmem",
"numa_node":1, <<<<<<<<<<< Good
"target_node":3,
"iset_id":8178854823874859280,
"persistence_domain":"memory_controller",
"namespaces":[
{
"dev":"namespace1.0",
"mode":"fsdax",
"map":"dev",
"size":1065418227712,
"uuid":"5ca9f18a-3eee-443a-b308-bd8031e88c58",
"raw_uuid":"91bbf45c-86b8-4b71-860e-b462cfdc755b",
"sector_size":512,
"align":2097152,
"blockdev":"pmem1",
"numa_node":1, <<<<<<<<<<< Good
"target_node":3
}
]
},
{
"dev":"region0",
"size":1082331758592,
"align":16777216,
"available_size":0,
"max_available_extent":0,
"type":"pmem",
"numa_node":0, <<<<<<<<<<< Good
"target_node":2,
"iset_id":-88628192355872496,
"persistence_domain":"memory_controller",
"namespaces":[
{
"dev":"namespace0.0",
"mode":"fsdax",
"map":"dev",
"size":1065418227712,
"uuid":"077af49b-913e-4f10-9b61-6407c2205371",
"raw_uuid":"c3a68f50-a164-417b-b258-f143979d05ac",
"sector_size":512,
"align":2097152,
"blockdev":"pmem0",
"numa_node":0, <<<<<<<<<<< Good
"target_node":2
}
]
}
]
}Not Working (BIOS v1.2):
# dmidecode -t bios
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
Handle 0x0000, DMI type 0, 26 bytes
BIOS Information
Vendor: American Megatrends International, LLC.
Version: 1.2
Release Date: 02/15/2022
# dmidecode -t baseboard
# dmidecode 3.3
Getting SMBIOS data from sysfs.
SMBIOS 3.3.0 present.
Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
Manufacturer: Supermicro
Product Name: X12DPU-6
Version: 1.02
# ndctl list -Rv
{
"regions":[
{
"dev":"region1",
"size":1082331758592,
"align":16777216,
"available_size":0,
"max_available_extent":0,
"type":"pmem",
"numa_node":0, <<<<<<<<<<< Not correct!
"target_node":3,
"iset_id":8178854823874859280,
"persistence_domain":"memory_controller",
"namespaces":[
{
"dev":"namespace1.0",
"mode":"fsdax",
"map":"dev",
"size":1065418227712,
"uuid":"5ca9f18a-3eee-443a-b308-bd8031e88c58",
"raw_uuid":"91bbf45c-86b8-4b71-860e-b462cfdc755b",
"sector_size":512,
"align":2097152,
"blockdev":"pmem1", <<<<<<<<<<< Not correct!
"numa_node":0,
"target_node":3
}
]
},
{
"dev":"region0",
"size":1082331758592,
"align":16777216,
"available_size":0,
"max_available_extent":0,
"type":"pmem",
"numa_node":0,
"target_node":2,
"iset_id":-88628192355872496,
"persistence_domain":"memory_controller",
"namespaces":[
{
"dev":"namespace0.0",
"mode":"fsdax",
"map":"dev",
"size":1065418227712,
"uuid":"077af49b-913e-4f10-9b61-6407c2205371",
"raw_uuid":"c3a68f50-a164-417b-b258-f143979d05ac",
"sector_size":512,
"align":2097152,
"blockdev":"pmem0",
"numa_node":0,
"target_node":2
}
]
}
]
}Looking at the NFIT, I see PMem modules on both sockets point to the correct SpaRange Table, eg:
---TableType=0x1
Length: 48 bytes
TypeEquals: NvDimmRegion
NfitDeviceHandle: 0x1311
NfitDeviceHandle.DimmNumber: 0x1
NfitDeviceHandle.MemChannel: 0x1
NfitDeviceHandle.MemControllerId: 0x3
NfitDeviceHandle.SocketId: 0x1 <<<<<<<<<<< Correct
NfitDeviceHandle.NodeControllerId: 0x0
NvDimmPhysicalId: 0x48
NvDimmRegionalId: 0x10
SpaRangeDescriptionTableIndex: 0x2 <<<<<<<<<<
NvdimmControlRegionDescriptorTableIndex: 0x10
NvDimmRegionSize: 0x1f80000000
RegionOffset: 0x7000
NvDimmPhysicalAddressRegionBase: 0x10000000
InterleaveStructureIndex: 0x10
InterleaveWays: 0x8
NvDimmStateFlags: 0x0020
-Notify OSPM of Smart & Health events 0x0020
---TableType=0x0
Length: 56 bytes
TypeEquals: SpaRange
AddressRangeType: 66f0d379-b4f3-4074-ac43-0d3318b78cdb
SpaRangeDescriptionTableIndex: 0x1
Flags: 0x0
ProximityDomain: 0x0
SystemPhysicalAddressRangeBase: 0x8080000000
SystemPhysicalAddressRangeLength: 0xfc00000000
MemoryMappingAttribute: 0x8008
---TableType=0x0
Length: 56 bytes
TypeEquals: SpaRange
AddressRangeType: 66f0d379-b4f3-4074-ac43-0d3318b78cdb
SpaRangeDescriptionTableIndex: 0x2
Flags: 0x0
ProximityDomain: 0x0 <<<<<<<<<<<<< Not correct. Should be 0x1
SystemPhysicalAddressRangeBase: 0x17c80000000
SystemPhysicalAddressRangeLength: 0xfc00000000
MemoryMappingAttribute: 0x8008/proc/iomem only has one entry rather than multiple that I would expect:
From the Supermicro system:
# grep Persistent /proc/iomem
8080000000-2787fffffff : Persistent MemoryFrom a working Cascade Lake Fedora host (5.18.0 Kernel)
# grep Persistent /proc/iomem
300000000-12ffffffff : Persistent Memory (legacy)
3060000000-1aa5fffffff : Persistent Memory
1b800000000-1c7ffffffff : Persistent Memory (legacy)
1da60000000-3545fffffff : Persistent Memory@sscargal Thank you very much for your time and effort.
At least the root-cause is now identified and is not an ipmctl bug.
I can shortly explain the issue to the SUPERMICRO customer support tomorrow and point them to this discussion to locate further details.
StevenPontsler
commented
2 years ago Thank you all for your effort.
sscargal
commented
2 years ago @dimstav23 Were you able to get a support ticket created with Supermicro? If needed, Supermicro can engage the Intel BIOS team through an Intel Premier Support (IPS) ticket.
dimstav23
commented
2 years ago @sscargal I haven't got in touch with them yet because I was quite busy the past week. I will either write them tomorrow or latest Monday and keep you updated. I'll also mention explicitly the IPS possibility (I am pretty sure they already are aware of it)
sscargal
commented
2 years ago No problem. Thanks for the update.
I have an 8-socket machine, which is half-populated with DRAM, and half with DCPMM.
When i use ipmctl create -goal to create my regions (and after a reboot!), the regions have all been created, but are 'attached' to numa_node 0. This is incorrect, as they should of course be created on numa_nodes 0 to 7. Note no options were passed to create -goal, as I want the DIMMS to be in AppDirect mode, which is the default.
When I then attempt to use these DCPMM (in appdirect mode, after creating namespaces, and correctly formatting and configuring) with SAP HANA, the use of the namespaces is refused by the database because of the wrong numa_node setting on the regions.
Here is the DIMM layout:
Here are the regions defined:
The regions as seen by ndctl:
Software versions:
And the DCPMM firmware version (I confirm all DIMMS are on the same firmware level):
The numa setup of the machine:
and the CPU assignments:
And finally the error messages I get from HANA (note that the namespace which is on region 0, on numa_node 0 (which is correct) is accepted for use by the DB):
My question: how can I correct the numa_node assignment for the regions? I have not found any way to specify which numa_node I want, and I thought the assignment would be automatic according to the DIMM placements.
Thanks!