OpenCL Backend NUMA Issues

[naibaf7]

Excerpt from my current thesis:

An issue that came up testing the OpenCL hybrid backend was that the performance did not scale as expected with systems that have more than one CPU. Such systems have non-unified memory access (NUMA) because the CPUs share one address space for memory, but every processor has its own cache and memory interface. Accessing data across the other CPU comes with a large performance penalty. Compute kernels, such as the matrix-matrix multiplication in the BLAS library or the custom OpenCL kernels, cause the threads to work on adjacent data. This means a write operation of one CPU is likely to invalidate cache lines across both CPUs. At this point, the synchronization overhead seems to become larger than any speedup of having additional cores working on the al- gorithms.

To get the expected speedup, the two (or more) processors need to be presented to the Caffe library as separate devices. Then the library can be used in two individual instances. As the OpenCL hybrid backend uses two separate parallelization mech- anisms (OpenCL kernels and a parallelized BLAS), two solutions would need to be applied:

• The Caffe frontend needs to be tied to the cores of one CPU, so that the BLAS library does not show NUMA issues.
• The OpenCL backend needs to split up the processor setup into sub-devices using device fission. The splitting rule needs to be that all cores belonging to one processor (tested by cache affinity) are tied to the same sub-device. Only one is then used per Caffe instance. Device fission is an extension to OpenCL that is already available (cl_ext_fission).
• The cores used in the frontend and selected sub-device need to be the same.

[bhack]

See also https://software.intel.com/en-us/node/540545

[naibaf7]

@bhack Ok, maybe not directly related (this affects CPUs while your article is about (multiple) Xeon Phi card(s)). But definitely an interesting read, thanks. Sadly, currently don't have such a card available for testing anyways. However the Xeon Phi should not behave much different from regular GPUs when using OpenCL.

[bhack]

Yes it is something not related to Numa but to consider on some Intel device. Also interesting https://software.intel.com/en-us/articles/opencl-device-fission-for-cpu-performance

[naibaf7]

@bhack Yup, this link is exactly what needs to be added to the device initialization in the OpenCL backend. :) You see, this is also a reason why I want to have multi-device training in OpenCL as with CUDA, as the training on a multiprocessor-NUMA system would benefit very heavily.

[bhack]

Other than Intel https://www.dcl.hpi.uni-potsdam.de/teaching/numasem/slides/NUMASem_OpenCL.pdf

[naibaf7]

And also APU systems like AMD HSA Kaveri and Intel Broadwell, as this PDF points out, yup.