[BUG] likwid-topology: socket count mismatch on dual Xeon X5560

[plusterkopp]

Describe the bug This is a 2-way server from 2009 (don’t hit me). It correctly reports 2 sockets, 4 core per socket and 2 threads per core. But when listing the logical cpus, they are all on the same die and socket. Likewise, there is only Socket 1 and it has all the cpus. The L3 cache affinity however is correct (even cpus on one cache, odd cpus on the other). NUMA also checks out with 2 domains.

To Reproduce

• LIKWID command and/or API usage likwid-topology
• LIKWID version and download source (Github, FTP, package manger, ...) latest from git: likwid-topology -- Version 5.2.3 (commit: 233ab943543480cd46058b34616c174198ba0459)
• Operating system centOS 7
• Does your application use libraries like MPI, OpenMP or Pthreads? no application
• In case of Nvidia GPUs, which CUDA version? n/a
• Are you using the MarkerAPI (CPU code instrumentation) or the NvMarkerAPI (Nvidia GPU code instrumentation)? I guess not

To Reproduce with a LIKWID command Please supply the output of the command with -V 3 added to the command:

• likwid-topology

  DEBUG - [hwloc_init_cpuInfo:361] HWLOC CpuInfo Family 6 Model 26 Stepping 5 Vendor 0x0 Part 0x0 isIntel 1 numHWThreads 16 activeHWThreads 16
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 0 Thread 0 Core 0 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 8 Thread 1 Core 0 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 2 Thread 0 Core 1 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 10 Thread 1 Core 1 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 4 Thread 0 Core 2 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 12 Thread 1 Core 2 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 6 Thread 0 Core 3 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 14 Thread 1 Core 3 Die 0 Socket 1 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 1 Thread 0 Core 4 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 9 Thread 1 Core 4 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 3 Thread 0 Core 5 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 11 Thread 1 Core 5 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 5 Thread 0 Core 6 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 13 Thread 1 Core 6 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 7 Thread 0 Core 7 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_nodeTopology:570] HWLOC Thread Pool PU 15 Thread 1 Core 7 Die 0 Socket 0 inCpuSet 1
  DEBUG - [hwloc_init_cacheTopology:791] HWLOC Cache Pool ID 0 Level 1 Size 32768 Threads 2
  DEBUG - [hwloc_init_cacheTopology:791] HWLOC Cache Pool ID 1 Level 2 Size 262144 Threads 2
  DEBUG - [hwloc_init_cacheTopology:791] HWLOC Cache Pool ID 2 Level 3 Size 8388608 Threads 8
  DEBUG - [affinity_init:539] Affinity: Socket domains 2
  DEBUG - [affinity_init:541] Affinity: CPU die domains 2
  DEBUG - [affinity_init:546] Affinity: CPU cores per LLC 4
  DEBUG - [affinity_init:549] Affinity: Cache domains 2
  DEBUG - [affinity_init:553] Affinity: NUMA domains 2
  DEBUG - [affinity_init:554] Affinity: All domains 9
  DEBUG - [treeFillNextEntries:128] Cannot find socket 0 in topology tree
  ERROR - [/home/ICUBIC/rhelbing/build/likwid/src/tree.c:tree_getChildNode:263] Node invalid
  DEBUG - [affinity_addNodeDomain:370] Affinity domain N: 8 HW threads on 4 cores
  DEBUG - [affinity_addSocketDomain:401] Affinity domain S0: 8 HW threads on 4 cores
  DEBUG - [treeFillNextEntries:128] Cannot find socket 0 in topology tree
  ERROR - [/home/ICUBIC/rhelbing/build/likwid/src/tree.c:tree_getChildNode:263] Node invalid
  DEBUG - [affinity_addSocketDomain:401] Affinity domain S1: 0 HW threads on 0 cores
  DEBUG - [affinity_addDieDomain:438] Affinity domain D0: 8 HW threads on 4 cores
  DEBUG - [treeFillNextEntries:128] Cannot find socket 0 in topology tree
  ERROR - [/home/ICUBIC/rhelbing/build/likwid/src/tree.c:tree_getChildNode:263] Node invalid
  DEBUG - [affinity_addDieDomain:438] Affinity domain D1: 0 HW threads on 0 cores
  DEBUG - [affinity_addCacheDomain:474] Affinity domain C0: 8 HW threads on 4 cores
  DEBUG - [treeFillNextEntries:128] Cannot find socket 0 in topology tree
  ERROR - [/home/ICUBIC/rhelbing/build/likwid/src/tree.c:tree_getChildNode:263] Node invalid
  DEBUG - [affinity_addCacheDomain:474] Affinity domain C1: 0 HW threads on 0 cores
  DEBUG - [affinity_addMemoryDomain:504] Affinity domain M0: 8 HW threads on 4 cores
  DEBUG - [affinity_addMemoryDomain:504] Affinity domain M1: 8 HW threads on 4 cores
  DEBUG - [create_lookups:290] T 0 T2C 0 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 1 T2C 4 T2S 1 T2D 1 T2LLC 1 T2M 1
  DEBUG - [create_lookups:290] T 2 T2C 1 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 3 T2C 5 T2S 1 T2D 1 T2LLC 1 T2M 1
  DEBUG - [create_lookups:290] T 4 T2C 2 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 5 T2C 6 T2S 1 T2D 1 T2LLC 1 T2M 1
  DEBUG - [create_lookups:290] T 6 T2C 3 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 7 T2C 7 T2S 1 T2D 1 T2LLC 1 T2M 1
  DEBUG - [create_lookups:290] T 8 T2C 0 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 9 T2C 4 T2S 1 T2D 1 T2LLC 1 T2M 1
  DEBUG - [create_lookups:290] T 10 T2C 1 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 11 T2C 5 T2S 1 T2D 1 T2LLC 1 T2M 1
  DEBUG - [create_lookups:290] T 12 T2C 2 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 13 T2C 6 T2S 1 T2D 1 T2LLC 1 T2M 1
  DEBUG - [create_lookups:290] T 14 T2C 3 T2S 1 T2D 1 T2LLC 0 T2M 0
  DEBUG - [create_lookups:290] T 15 T2C 7 T2S 1 T2D 1 T2LLC 1 T2M 1
  --------------------------------------------------------------------------------
  CPU name:       Intel(R) Xeon(R) CPU           X5560  @ 2.80GHz
  CPU type:       Intel Core Bloomfield processor
  CPU stepping:   5
  ********************************************************************************
  Hardware Thread Topology
  ********************************************************************************
  Sockets:                2
  Cores per socket:       4
  Threads per core:       2
  --------------------------------------------------------------------------------
  HWThread        Thread        Core        Die        Socket        Available
  0               0             0           0          1             *
  1               0             4           0          1             *
  2               0             1           0          1             *
  3               0             5           0          1             *
  4               0             2           0          1             *
  5               0             6           0          1             *
  6               0             3           0          1             *
  7               0             7           0          1             *
  8               1             0           0          1             *
  9               1             4           0          1             *
  10              1             1           0          1             *
  11              1             5           0          1             *
  12              1             2           0          1             *
  13              1             6           0          1             *
  14              1             3           0          1             *
  15              1             7           0          1             *
  --------------------------------------------------------------------------------
  Socket 1:               ( 0 8 2 10 4 12 6 14 1 9 3 11 5 13 7 15 )
  --------------------------------------------------------------------------------
  ********************************************************************************
  Cache Topology
  ********************************************************************************
  Level:                  1
  Size:                   32 kB
  Cache groups:           ( 0 8 ) ( 2 10 ) ( 4 12 ) ( 6 14 ) ( 1 9 ) ( 3 11 ) ( 5 13 ) ( 7 15 )
  --------------------------------------------------------------------------------
  Level:                  2
  Size:                   256 kB
  Cache groups:           ( 0 8 ) ( 2 10 ) ( 4 12 ) ( 6 14 ) ( 1 9 ) ( 3 11 ) ( 5 13 ) ( 7 15 )
  --------------------------------------------------------------------------------
  Level:                  3
  Size:                   8 MB
  Cache groups:           ( 0 8 2 10 4 12 6 14 ) ( 1 9 3 11 5 13 7 15 )
  --------------------------------------------------------------------------------
  ********************************************************************************
  NUMA Topology
  ********************************************************************************
  NUMA domains:           2
  --------------------------------------------------------------------------------
  Domain:                 0
  Processors:             ( 0 8 2 10 4 12 6 14 )
  Distances:              10 20
  Free memory:            31404.1 MB
  Total memory:           32203.2 MB
  --------------------------------------------------------------------------------
  Domain:                 1
  Processors:             ( 1 9 3 11 5 13 7 15 )
  Distances:              20 10
  Free memory:            31503.6 MB
  Total memory:           32240.9 MB
  --------------------------------------------------------------------------------

Additional context Maybe this is just a reporting problem since “hwloc_init_nodeTopology:570” seems to be on the right track (assuming a die is part of a socket). I should be able to circumvent this by focusing on L3 or NUMA affinity instead of socket, but maybe this occurs elsewhere, too.

[TomTheBear]

Can you please check whether your system is telling that there are two sockets: $ cat /sys/devices/system/cpu/cpu*/topology/physical_package_id | sort -u

A CPU socket may consist of multiple CPU dies but in your case there is only one die but (maybe wrongly) two sockets according to hwloc_init_nodeTopology:570.

The error is likeley caused by the CPU ID 0 being on socket 1. This means, the first socket-level node in the topology tree has ID 1, not 0.

[plusterkopp]

$ cat /sys/devices/system/cpu/cpu*/topology/physical_package_id | sort -u 0 1

and for completeness: $ cat /sys/devices/system/cpu/cpu*/topology/physical_package_id 1 1 0 1 0 1 0 0 1 0 1 0 1 0 1 0

I am OK with the die, since I understand now that the list Thread 0 Core 0 Die 0 Socket 1 inCpuSet 1 represents a hierarchy with local numbering.

[TomTheBear]

Yes, it represents the hierarchy but if you want to do more with LIKWID, the topology stuff should work, it's the foundation of anything in LIKWID. I will take a look.

[plusterkopp]

OK, what I want: to have something similar to Windows’ GetLogicalProcessorInformationEx for use in my https://github.com/plusterkopp/Java-Thread-Affinity lib. Totally unrelated to this issue: Is there something lighter than the full liblikwid.so if I only need the topology detection? I only want to access an .so from JNA.

[TomTheBear]

Since you don't seem to have a problem linking with a shared library, I would use hwloc. It's also used by LIKWID internally to get the topology information. If you want to have it more basic, you can write the stuff on your own. It's not that difficult if you leave out some features (hierarchical NUMA nodes, ...). I wrote such an interface in Golang for another project: ccTopology.

[plusterkopp]

Since you don't seem to have a problem linking with a shared library, I would use hwloc.

If I install that, I can just call hwloc-ls --no-io and parse the output. No need to mess with JNA parameter voodoo. Tha’ŧs actually a great idea. I’ll try that.

If you want to have it more basic, you can write the stuff on your own. It's not that difficult if you leave out some features (hierarchical NUMA nodes, ...). I wrote such an interface in Golang for another project: ccTopology.

In the case of hwloc, the data structures with their use of unions are even more complex than likwid’s. Like the Windows implementation (GetLogicalProcessorInformationEx returns a pointer to unions, too). In general I’d like to stay away from JNA with its need to recreate all data types on the Java side. Ever since I had this up and running in 2015 I was looking a for a Linux equivalent where I get better topology info than what /proc/cpuinfo provides. Didn’t think of hwloc, much less of likwid.

Vielen Dank für Eure Arbeit :)

[TomTheBear]

I would try to attach to a library to get the information in a consistent format. Parsing the CLI output is often not future-proof. There are existing wrappers: https://github.com/UDC-GAC/jhwloc

You can add a small C-snippet in front of hwloc providing a simple interface with all the information you need to avoid unions and complicated JNA type recreation. Or you write it yourself in pure Java derived from the ccTopology code.

Thanks for the flowers :tulip:

P.S. If you have hwloc installed, please send me/attach the tarball of hwloc-gather-topology $(hostname -s). Then I can analyze your topology with LIKWID and fix the problems.

[plusterkopp]

srvsd002.tar.zip stupid github doesn’t like .bz2 or .7z

I actually did what you suggested when I tried the Windows API. Couldn’t get it to work reliably with JNA since the returned topology entries were all of different size, so I ended up writing an extra DLL that took the Windows API and rearranged the result data.

But this time, I think I stay with the text output and update my parser if something changes.