Open tkoskela opened 9 months ago
Young
run of STREAM
with likwid-perfctr
and group "BRANCH" for cache hit/miss stats in Young's login02
:(EXCALIBUR) ✔ ~/Excalibur/dloads/STREAM [master|✔]
[ucaseko@login02.ib.young:7 STREAM]$ hostname
login02
(EXCALIBUR) ✔ ~/Excalibur/dloads/STREAM [master|✔]
[ucaseko@login02.ib.young:7 STREAM]$ pwd
/home/ucaseko/Excalibur/dloads/STREAM
(EXCALIBUR) ✔ ~/Excalibur/dloads/STREAM [master|✔]
[ucaseko@login02.ib.young:7 STREAM]$ ../../likwid-5.3.0/bin/likwid-perfctr -C S0:1 -g BRANCH ./stream_c.exe
--------------------------------------------------------------------------------
CPU name: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
CPU type: Intel Cascadelake SP processor
CPU clock: 2.49 GHz
--------------------------------------------------------------------------------
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 10000000 (elements), Offset = 0 (elements)
Memory per array = 76.3 MiB (= 0.1 GiB).
Total memory required = 228.9 MiB (= 0.2 GiB).
Each kernel will be executed 10 times.
The *best* time for each kernel (excluding the first iteration)
will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 1
Number of Threads counted = 1
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 8129 microseconds.
(= 8129 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 10042.2 0.016097 0.015933 0.016232
Scale: 13951.9 0.011647 0.011468 0.011744
Add: 15880.2 0.015263 0.015113 0.015490
Triad: 15587.4 0.015524 0.015397 0.015697
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
--------------------------------------------------------------------------------
Group 1: BRANCH
+------------------------------+---------+------------+
| Event | Counter | HWThread 1 |
+------------------------------+---------+------------+
| INSTR_RETIRED_ANY | FIXC0 | 2343230311 |
| CPU_CLK_UNHALTED_CORE | FIXC1 | 2288835488 |
| CPU_CLK_UNHALTED_REF | FIXC2 | 1550699600 |
| BR_INST_RETIRED_ALL_BRANCHES | PMC0 | 367655406 |
| BR_MISP_RETIRED_ALL_BRANCHES | PMC1 | 8310 |
+------------------------------+---------+------------+
+----------------------------+--------------+
| Metric | HWThread 1 |
+----------------------------+--------------+
| Runtime (RDTSC) [s] | 0.6810 |
| Runtime unhalted [s] | 0.9177 |
| Clock [MHz] | 3681.2784 |
| CPI | 0.9768 |
| Branch rate | 0.1569 |
| Branch misprediction rate | 3.546386e-06 |
| Branch misprediction ratio | 2.260269e-05 |
| Instructions per branch | 6.3734 |
+----------------------------+--------------+
Young
's login01
run of same:
(EXCALIBUR) ✔ ~/Excalibur/dloads/STREAM [master|✔]
[ucaseko@login01.ib.young:0 STREAM]$ hostname
login01
(EXCALIBUR) ✔ ~/Excalibur/dloads/STREAM [master|✔]
[ucaseko@login01.ib.young:0 STREAM]$ pwd
/home/ucaseko/Excalibur/dloads/STREAM
(EXCALIBUR) ✔ ~/Excalibur/dloads/STREAM [master|✔]
[ucaseko@login01.ib.young:0 STREAM]$ ../../likwid-5.3.0/bin/likwid-perfctr -C S0:1 -g BRANCH ./stream_c.exe
--------------------------------------------------------------------------------
CPU name: Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz
CPU type: Intel Cascadelake SP processor
CPU clock: 2.49 GHz
--------------------------------------------------------------------------------
-------------------------------------------------------------
STREAM version $Revision: 5.10 $
-------------------------------------------------------------
This system uses 8 bytes per array element.
-------------------------------------------------------------
Array size = 10000000 (elements), Offset = 0 (elements)
Memory per array = 76.3 MiB (= 0.1 GiB).
Total memory required = 228.9 MiB (= 0.2 GiB).
Each kernel will be executed 10 times.
The *best* time for each kernel (excluding the first iteration)
will be used to compute the reported bandwidth.
-------------------------------------------------------------
Number of Threads requested = 1
Number of Threads counted = 1
-------------------------------------------------------------
Your clock granularity/precision appears to be 1 microseconds.
Each test below will take on the order of 7749 microseconds.
(= 7749 clock ticks)
Increase the size of the arrays if this shows that
you are not getting at least 20 clock ticks per test.
-------------------------------------------------------------
WARNING -- The above is only a rough guideline.
For best results, please be sure you know the
precision of your system timer.
-------------------------------------------------------------
Function Best Rate MB/s Avg time Min time Max time
Copy: 10362.1 0.015691 0.015441 0.015894
Scale: 14450.7 0.011247 0.011072 0.011325
Add: 16494.9 0.014793 0.014550 0.014897
Triad: 16132.2 0.015059 0.014877 0.015234
-------------------------------------------------------------
Solution Validates: avg error less than 1.000000e-13 on all three arrays
-------------------------------------------------------------
--------------------------------------------------------------------------------
Group 1: BRANCH
+------------------------------+---------+------------+
| Event | Counter | HWThread 1 |
+------------------------------+---------+------------+
| INSTR_RETIRED_ANY | FIXC0 | 2343225425 |
| CPU_CLK_UNHALTED_CORE | FIXC1 | 2222105438 |
| CPU_CLK_UNHALTED_REF | FIXC2 | 1513890200 |
| BR_INST_RETIRED_ALL_BRANCHES | PMC0 | 367654668 |
| BR_MISP_RETIRED_ALL_BRANCHES | PMC1 | 8687 |
+------------------------------+---------+------------+
+----------------------------+--------------+ | Metric | HWThread 1 | +----------------------------+--------------+ | Runtime (RDTSC) [s] | 0.7105 | | Runtime unhalted [s] | 0.8912 | | Clock [MHz] | 3659.6906 | | CPI | 0.9483 | | Branch rate | 0.1569 | | Branch misprediction rate | 3.707283e-06 | | Branch misprediction ratio | 2.362815e-05 | | Instructions per branch | 6.3734 | +----------------------------+--------------+
- [X] Run same in one of `Kathleen`'s nodes:
Group 1: BRANCH +------------------------------+---------+------------+ | Event | Counter | HWThread 1 | +------------------------------+---------+------------+ | INSTR_RETIRED_ANY | FIXC0 | 2343250440 | | CPU_CLK_UNHALTED_CORE | FIXC1 | 2245719147 | | CPU_CLK_UNHALTED_REF | FIXC2 | 1559575000 | | BR_INST_RETIRED_ALL_BRANCHES | PMC0 | 367660744 | | BR_MISP_RETIRED_ALL_BRANCHES | PMC1 | 8244 | +------------------------------+---------+------------+
+----------------------------+--------------+ | Metric | HWThread 1 | +----------------------------+--------------+ | Runtime (RDTSC) [s] | 0.6769 | | Runtime unhalted [s] | 0.9004 | | Clock [MHz] | 3591.3189 | | CPI | 0.9584 | | Branch rate | 0.1569 | | Branch misprediction rate | 3.518190e-06 | | Branch misprediction ratio | 2.242285e-05 | | Instructions per branch | 6.3734 | +----------------------------+--------------+
❌ Attempting to run STREAM
under likwid-perfctr
on my laptop snags, with message about "unknown processor"
[ucaseko@littlebeast:18 STREAM]$ pwd
/home/ucaseko/dloads/STREAM
(EXCALIBUR) ✔ ~/dloads/STREAM [master|✔]
[ucaseko@littlebeast:18 STREAM]$ ~/likwid-5.3.0/bin/likwid-perfctr -C S0:1 -g BRANCH ./stream_c.exe
Cannot access directory /home/ucaseko/likwid-5.3.0/share/likwid/perfgroups/unknown
--------------------------------------------------------------------------------
CPU name: 13th Gen Intel(R) Core(TM) i7-1365U
CPU type: Unknown Intel Processor
CPU clock: 2.69 GHz
ERROR - [/home/ucaseko/dloads/likwid-5.3.0/src/perfmon.c:perfmon_init_maps:1184] Unsupported Processor
ERROR - [/home/ucaseko/dloads/likwid-5.3.0/src/perfmon.c:perfmon_init:2109] No such file or directory.
Failed to initialize event and counter lists for Unknown Intel Processor
Excerpt from cat /proc/cpuinfo
processor : 11
vendor_id : GenuineIntel
cpu family : 6
model : 186
model name : 13th Gen Intel(R) Core(TM) i7-1365U
stepping : 3
microcode : 0xffffffff
cpu MHz : 2688.010
cache size : 12288 KB
physical id : 0
siblings : 12
core id : 5
cpu cores : 6
apicid : 11
initial apicid : 11
fpu : yes
fpu_exception : yes
cpuid level : 28
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology tsc_reliable nonstop_tsc cpuid pni pclmulqdq vmx ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch ssbd ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi ept vpid ept_ad fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves avx_vnni umip waitpkg gfni vaes vpclmulqdq rdpid movdiri movdir64b fsrm serialize flush_l1d arch_capabilities
vmx flags : vnmi invvpid ept_x_only ept_ad ept_1gb tsc_offset vtpr ept vpid unrestricted_guest ept_mode_based_exec tsc_scaling usr_wait_pause
bugs : spectre_v1 spectre_v2 spec_store_bypass swapgs retbleed eibrs_pbrsb
bogomips : 5376.02
clflush size : 64
cache_alignment : 64
address sizes : 46 bits physical, 48 bits virtual
power management:
- For comparison, similar excerpt from `Kathleen`'s node used above:
processor : 79 vendor_id : GenuineIntel cpu family : 6 model : 85 model name : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz stepping : 7 microcode : 0x5003003 cpu MHz : 2500.000 cache size : 28160 KB physical id : 1 siblings : 40 core id : 28 cpu cores : 20 apicid : 121 initial apicid : 121 fpu : yes fpu_exception : yes cpuid level : 22 wp : yes flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 cdp_l3 invpcid_single intel_ppin intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm mpx rdt_a avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts pku ospke avx512_vnni md_clear spec_ctrl intel_stibp flush_l1d arch_capabilities bogomips : 5004.96 clflush size : 64 cache_alignment : 64 address sizes : 46 bits physical, 48 bits virtual power management:
- Not sure if the ID tag of mine,
`model name : 13th Gen Intel(R) Core(TM) i7-1365U` vs `Kathleen`'s
`model name : Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz`
makes it snag, i.e. not starting with letters but by numbers ... 🤷 ... will look into it -- maybe we need to include / request some new patterns in the `likwid` code, for correct CPU identification.
The results on young
and kathleen
look promising! Could you check what other performance groups are availble on those machines?
They have https://github.com/RRZE-HPC/likwid/issues/468 open on support for newer Intel architectures. Does not seem to be much progress happening lately, so your laptop cpu might be still unsupported.
Test profiling with
likwid-perfctr
on a system where we don't have root access (e.g. Kathleen, Young). Use a simple application, such as STREAM and investigate what metric groups in LIKWID we can have access to with user permissions. Build STREAM manually using theMakefile
in the github repository. Run in serial as first step. It's probably fine to run on a login node for debugging, because it only takes a few seconds to run. In the end we need to run on a compute node using the scheduler.Investigate what metrics are availble from LIKWID, e.g.