CMK Isolation on different NUMA nodes

[adpempx]

Hi,

are you planning to add "NUMA node configuration" on CMK isolate? Because the idea is to specify the NUMA node during the isolation, in order to get CPUs from a NUMA node or from another NUMA node on basis of application. In that way, it allows us the splitting of different workloads on different NUMA nodes.

The best use case is run a DPDK application on isolated CPUs of one NUMA node and then run other applications on isolated CPUs of other NUMA node.

Please let me know.

Thanks

[adpempx]

Hi,

do you have any news?

Thanks!

[pbrownlow7]

Hi @adpempx,

CMK allows for NUMA specification with the isolate command. To accomplish this use the --socket-id flag to specify which NUMA socket you would like the specified core/number of cores to come from: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/974133f8726118f7c804c5cba126761d62c43ccb/cmk.py#L85

This link leads to the documentation that explains how to set CMK up with multiple NUMA sockets as well if you need it: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/master/docs/operator.md#multi-socket-support-experimental

Hopefully this is the solution you are looking for, if not let me know!

Thanks, Philip

[adpempx]

Hi Philip,

actually we are using mode policy "packed" and I also tried to use --socket-id=< num > option during the isolation, but it looks it does not work correctly.

As we have 2 NUMA nodes, I tried to set --socket-id=0 or --socket-id=1, but python code does now allow us to set 0 and or 1.

Could you please confirm if I am using the correct mode policy and socket ID numbers?

Thanks

[pbrownlow7]

Hi @adpempx,

I had a look through the isolate code and it turns out it was in fact a bug. The socket-id gets passed in to isolate.py as a string but the keys in the config are actually ints.

I'll be fixing this in the next release for CMK, which I'm hoping to cut in the next couple of weeks. In the meantime if you want the fix, you can edit the isolate.py file on line 55: https://github.com/intel/CPU-Manager-for-Kubernetes/blob/974133f8726118f7c804c5cba126761d62c43ccb/intel/isolate.py#L55 to be the following: selected_socket = int(socket_id)

In regards to the mode policy, packed won't take into consideration the NUMA nodes from which it takes cores, although you may get lucky and it could take them from both if enough cores are selected. To guarantee that cores are taken evenly from both NUMA nodes (so if you set CMK up to have 8 exclusive cores it will take 4 from NUMA node 0 and 4 from NUMA node 1), then you want to use the 'spread' mode policy.

Thanks, Philip

[adpempx]

Hi Philip,

if we want to ensure CMK will provides cores from only NUMA node 0 for a process and cores from only NUMA node 1 for another process, like:

• Process 0: 8 exclusive cores on only NUMA node 0, never on NUMA node 1
• Process 1: 8 exclusive cores on only NUMA node 1, never on NUMA node 0

Which is the correct policy? Packed or spread?

Can you suggest your guidelines?

Thanks

[adpempx]

Hi,

I confirm I resolved the issue with your patch:

to be the following: selected_socket = int(socket_id)

in CPU-Manager-for-Kubernetes/intel/isolate.py, and policy Packed.

Everything works as expected.

I'll close this ticket, just please remember to put the above patch in the next release.

Many thanks!