Poor Thread Scaling Behaviour of SGEMM

[FabianSchuetze]

It seem to me that the SGEMM kernel has poor scaling behaviour with more threads.

I am using the CPP scheduler for the example/neon_sgemm.cpp code. There seem to be a few issues:

• Without setting the number of threads manually, scheduler.num_threads_hint() reports 1

• Increasing the number of threads manually does not offer good performance improvements, see the table below:

  | Threads   | ms    |
  |---------  |----   |
  | 1         | 22    |
  | 2         | 22    |
  | 4         | 14    |
  | 8         | 17    |

  The relevant output from the logs is:

  [ComputeLibrary][03-07-2024 10:31:35][INFO]  arm_compute::cpu::CpuGemm::configure() : 
  a: Shape=1024,1024,DataLayout=NCHW,DataType=F32
  b: Shape=1024,1024,DataLayout=NCHW,DataType=F32
  c: nullptr
  d: Shape=1024,1024,DataLayout=NCHW,DataType=F32
  alpha: 1
  beta: 0
  gemm_info: {is_a_reshaped=0,is_b_reshaped=0,reshape_b_only_on_first_run=1,depth_output_gemm3d=0,reinterpret_input_as_3d=0,retain_internal_weights=0,fp_mixed_precision=0,broadcast_bias=0,pretranspose_B=0,}
  
  [CORE][03-07-2024 10:31:35][INFO]  "Set CPPScheduler to Linear mode, with 8 threads to use\n"

  Using openmp scheduler yields worse performance. My device consist of X3, A715 and A510 processors (Samsung S9 Tablet)

What scaling behavior should I expect in general? Is there anything I can do to debug the issue?

[morgolock]

Hi @FabianSchuetze

The openmp scheduler will scale better both on Android and Linux. Could you please try building ACL with openmp=1 cppthreads=0

You should not have to set the number of threads, ACL will use the best number for your system

Hope this helps,