NUMA-awareness

[malikm]

BOINC client needs to be NUMA-aware to support systems with multiple CPU Groups (NUMA nodes).

• Manage pool of all client processes and explicitly assign Group Affinity via SetThreadGroupAffinity.

See: http://boinc.berkeley.edu/dev/forum_thread.php?id=10124

[kszysiu]

A friend of mine whose heart BOINC is dear to directed me here :)

I'll just drop this short note -- remember that Windows CPU grouping doesn't need to match NUMA nodes. More often than not it actually does NOT match. For instance, in a 64-core, 8 node system you will normally observe single CPU group (because number of CPUs doesn't exceed maximal group size).

In other words, this bug has to do with CPU grouping in Windows (when >64 CPUs are present). NUMA is in the picture but rather in the background...

malikm, for the sake of experiment, you could disable HyperThreading and I'm betting $10 you will see all 36 cores loaded evenly.

[malikm]

AFAIK you can even setup a system with <64 logical CPUs to use multiple groups for testing purposes.

[kszysiu]

Not sure... But in case it's not possible, one can fake extra CPUs in a VM and accomplish the same goal. More than one way to skin a cat :)

[kszysiu]

Indeed it is possible to change the way CPUs are assigned to groups. While browsing MSDN, I've learned about boot configuration option in Windows, called groupsize.

bcdedit /set groupsize 4 sets CPU group size to 4 logical processors which on, say, 16t machine creates 4 groups.

Seems it should be enough for development and testing :)

[malikm]

BUMP More and more machines are going to have >64 threads, so I believe this problem should be addressed!

[agallagh42]

Old thread, but I'm currently running BOINC on a machine with 4 sockets X 18 cores, so 72 physical cores/144 threads. I can confirm that this is still an issue.

[davidpanderson]

I'll work on this. Design doc: https://boinc.berkeley.edu/trac/wiki/WinMulticore

[davidpanderson]

@agallagh42, what are you seeing? Are only some of the cores being used? What is the processor group structure?

[agallagh42]

Here's a screenshot of the CPU utilization with the graph set to "NUMA Nodes" view. As you can see, only one of the nodes is getting fully utilized, and overall I'm seeing about 75% utilization.

[davidpanderson]

Thanks. Is BOINC running enough jobs to use all the cores? Are they single- or multi-thread jobs, or a mixture? Can you see what "processor group" the jobs are running in? (there may be a tool that shows this, though I don't know of one).

[malikm]

I was not quite accurate with my initial statement about NUMA awareness. The relation between NUMA nodes and CPU Groups doesn't have to be 1:1. So this problem is rather because all BOINC tasks (child processes) are assigned to one CPU group only. I don't have such machine now, but when I observed this, it was a 72-core (CPU threads) machine running 72 WCG single-thread tasks.

[malikm]

One can simulate this problem also on machines with < 64 threads by creating multiple CPU groups using "bcdedit /set groupsize nn" command (run elevated as Admin): https://docs.microsoft.com/en-us/windows-hardware/drivers/devtest/boot-parameters-to-test-drivers-for-multiple-processor-group-support

[agallagh42]

Thanks. Is BOINC running enough jobs to use all the cores?

I can't find an option to manage the number of jobs BOINC creates. It seems to be automatic. I'm only running one project (SETI) for now, if that matters.

Are they single- or multi-thread jobs, or a mixture?

No idea

Can you see what "processor group" the jobs are running in?

By the looks of it, the processor groups match the NUMA nodes, so all groups are getting some jobs, but group 0 is getting more than the other 3.

[agallagh42]

On further investigation, it looks like this is not necessarily a numa problem. I'm actually hitting a 100 task limit somewhere. Since this server has 144 threads, it would need 144 tasks running in order to max out all cores. Do you know how to increase this limit?

[RichardHaselgrove]

You mentioned that

I'm only running one project (SETI) for now

SETI has a server-imposed task limit of 100 concurrent tasks at any one time, irrespective of the CPU or core count. Test on a less busy project.

[davidpanderson]

There's still an issue to be addressed: Windows assigns jobs to processor groups round-robin, which may be highly non-optimal, especially if some of the jobs are multithread. If it turns out that Windows lets you assign jobs to processor groups (which I have yet to verify) then BOINC should do this in an intelligent way, as sketched above.

[AenBleidd]

NUMA is something you need to take care of when you're developing stuff that works on the kernel level. Other user space processes should not take care about such a stuff, and let the kernel to decide which core to assign every particular process to. After the internal discussion it was decided to close this ticket without any further actions.