intel / intel-cmt-cat

User space software for Intel(R) Resource Director Technology
http://www.intel.com/content/www/us/en/architecture-and-technology/resource-director-technology.html
Other
685 stars 180 forks source link

Actual memory bw is higher than configured using MBA in MB/s #277

Closed QiongwenXu closed 2 weeks ago

QiongwenXu commented 3 weeks ago

Hi, I am using MBA to limit memory bandwidth used of a set of cores. I used the following commands to limit the total memory bandwidth of core 17-22 to below 4000 MB/s. Then I ran memory bw application on those cores. However, the actual memory bandwidth is 13633.6 MB/s (significantly higher than the 4000 MB/s limit). Could you please take a look? I am not sure if this output is expected or if I used the wrong commands. Thanks!

$ sudo pqos -a 'core:1=17-22' NOTE: Mixed use of MSR and kernel interfaces to manage CAT or CMT & MBM may lead to unexpected behavior. Allocation configuration altered. $ sudo pqos -e 'mba_max:1=4000' NOTE: Mixed use of MSR and kernel interfaces to manage CAT or CMT & MBM may lead to unexpected behavior. SOCKET 0 MBA COS1 => 4000 MBps SOCKET 1 MBA COS1 => 4000 MBps Allocation configuration altered. $ sudo pqos -s NOTE: Mixed use of MSR and kernel interfaces to manage CAT or CMT & MBM may lead to unexpected behavior. MBA COS definitions for Socket 0: MBA COS0 => 4294967295 MBps MBA COS1 => 4000 MBps MBA COS2 => 4294967295 MBps MBA COS3 => 4294967295 MBps MBA COS4 => 4294967295 MBps MBA COS5 => 4294967295 MBps MBA COS6 => 4294967295 MBps MBA COS7 => 4294967295 MBps MBA COS definitions for Socket 1: MBA COS0 => 4294967295 MBps MBA COS1 => 4000 MBps MBA COS2 => 4294967295 MBps MBA COS3 => 4294967295 MBps MBA COS4 => 4294967295 MBps MBA COS5 => 4294967295 MBps MBA COS6 => 4294967295 MBps MBA COS7 => 4294967295 MBps Core information for socket 0: Core 0, L2ID 0, L3ID 0 => COS0 Core 1, L2ID 1, L3ID 0 => COS0 Core 2, L2ID 2, L3ID 0 => COS0 Core 3, L2ID 3, L3ID 0 => COS0 Core 4, L2ID 4, L3ID 0 => COS0 Core 5, L2ID 5, L3ID 0 => COS0 Core 6, L2ID 6, L3ID 0 => COS0 Core 7, L2ID 7, L3ID 0 => COS0 Core 8, L2ID 8, L3ID 0 => COS0 Core 9, L2ID 9, L3ID 0 => COS0 Core 10, L2ID 10, L3ID 0 => COS0 Core 11, L2ID 11, L3ID 0 => COS0 Core 12, L2ID 12, L3ID 0 => COS0 Core 13, L2ID 13, L3ID 0 => COS0 Core 14, L2ID 14, L3ID 0 => COS0 Core 15, L2ID 15, L3ID 0 => COS0 Core 32, L2ID 0, L3ID 0 => COS0 Core 33, L2ID 1, L3ID 0 => COS0 Core 34, L2ID 2, L3ID 0 => COS0 Core 35, L2ID 3, L3ID 0 => COS0 Core 36, L2ID 4, L3ID 0 => COS0 Core 37, L2ID 5, L3ID 0 => COS0 Core 38, L2ID 6, L3ID 0 => COS0 Core 39, L2ID 7, L3ID 0 => COS0 Core 40, L2ID 8, L3ID 0 => COS0 Core 41, L2ID 9, L3ID 0 => COS0 Core 42, L2ID 10, L3ID 0 => COS0 Core 43, L2ID 11, L3ID 0 => COS0 Core 44, L2ID 12, L3ID 0 => COS0 Core 45, L2ID 13, L3ID 0 => COS0 Core 46, L2ID 14, L3ID 0 => COS0 Core 47, L2ID 15, L3ID 0 => COS0 Core information for socket 1: Core 16, L2ID 16, L3ID 1 => COS0 Core 17, L2ID 17, L3ID 1 => COS1 Core 18, L2ID 18, L3ID 1 => COS1 Core 19, L2ID 19, L3ID 1 => COS1 Core 20, L2ID 20, L3ID 1 => COS1 Core 21, L2ID 21, L3ID 1 => COS1 Core 22, L2ID 22, L3ID 1 => COS1 Core 23, L2ID 23, L3ID 1 => COS0 Core 24, L2ID 24, L3ID 1 => COS0 Core 25, L2ID 25, L3ID 1 => COS0 Core 26, L2ID 26, L3ID 1 => COS0 Core 27, L2ID 27, L3ID 1 => COS0 Core 28, L2ID 28, L3ID 1 => COS0 Core 29, L2ID 29, L3ID 1 => COS0 Core 30, L2ID 30, L3ID 1 => COS0 Core 31, L2ID 31, L3ID 1 => COS0 Core 48, L2ID 16, L3ID 1 => COS0 Core 49, L2ID 17, L3ID 1 => COS0 Core 50, L2ID 18, L3ID 1 => COS0 Core 51, L2ID 19, L3ID 1 => COS0 Core 52, L2ID 20, L3ID 1 => COS0 Core 53, L2ID 21, L3ID 1 => COS0 Core 54, L2ID 22, L3ID 1 => COS0 Core 55, L2ID 23, L3ID 1 => COS0 Core 56, L2ID 24, L3ID 1 => COS0 Core 57, L2ID 25, L3ID 1 => COS0 Core 58, L2ID 26, L3ID 1 => COS0 Core 59, L2ID 27, L3ID 1 => COS0 Core 60, L2ID 28, L3ID 1 => COS0 Core 61, L2ID 29, L3ID 1 => COS0 Core 62, L2ID 30, L3ID 1 => COS0 Core 63, L2ID 31, L3ID 1 => COS0 PID association information: COS1 => (none) COS2 => (none) COS3 => (none) COS4 => (none) COS5 => (none) COS6 => (none) COS7 => (none)

Using MBM to monitor the memory bw used by the application.

     PID     CORE         IPC      MISSES   MBL[MB/s]   MBR[MB/s]
 3911943      err        0.18     158092k     13633.6         0.0

and memory bw used by each core

    CORE         IPC      MISSES   MBL[MB/s]   MBR[MB/s]
      17        0.18      26324k      2283.4         0.0
      18        0.18      26356k      2144.1         0.0
      19        0.18      26355k      2303.3         0.0
      20        0.18      26365k      2329.3         0.0
      21        0.18      26358k      2263.1         0.0
      22        0.18      26355k      2259.5         0.0
rkanagar commented 2 weeks ago

Hi QiongwenXu,

The Memory Bandwidth Allocation (MBA) feature provides indirect and approximate control over memory bandwidth available per-core. Please refer Intel SDM Volume 3 and Chapter 18.19.7.

  I can see the per-core MB is less than 4000MB.