Mean used for lnorm benchmark is not actual mean of the data

[ghost]

Summary

The benchdnn test for lnorm fails when oneDNN is built against the ACL backend. While investigating I found that the mean used in /tests/benchdnn/lnorm/ref_lnorm.cpp does not seem to be the mean of the underlying data (a similar mismatch exists for the variance). Can someone shed some light on this discrepancy?

Version

Report oneDNN version and githash. Version information is printed to stdout in verbose mode. oneDNN v3.6.0 (commit 094cc1dda4a24ccf0a54987a34c4475e00a926f9)

Environment

oneDNN includes hardware-specific optimizations and may behave differently on depending on the compiler and build environment. Include the following information to help reproduce the issue:

• aarch64, CPUs: 16, Flags: fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrc pc flagm ssbs paca pacg dcpodp svei8mm svebf16 i8mm bf16 dgh rng
• OS version: Linux 6.5.0-1020-aws 20~22.04.1-Ubuntu SMP Wed May 1 16:38:06 UTC 2024 aarch64 aarch64 aarch64 GNU/Linux
• Compiler version: gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
• CMake version: 3.22.1
• CMake output log:

  -- The C compiler identification is GNU 11.4.0
  -- The CXX compiler identification is GNU 11.4.0
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Check for working C compiler: /usr/bin/cc - skipped
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Check for working CXX compiler: /usr/bin/c++ - skipped
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  -- DNNL_TARGET_ARCH: AARCH64
  -- DNNL_LIBRARY_NAME: dnnl
  -- Looking for pthread.h
  -- Looking for pthread.h - found
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD
  -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
  -- Found Threads: TRUE  
  -- Found OpenMP_C: -fopenmp (found version "4.5") 
  -- Found OpenMP_CXX: -fopenmp (found version "4.5") 
  -- Found OpenMP: TRUE (found version "4.5")  
  -- Found ACL: /home/sidmen01/ComputeLibrary  
  -- Arm Compute Library: /home/sidmen01/ComputeLibrary/build/libarm_compute.so;/home/sidmen01/ComputeLibrary/build/libarm_compute_graph.so
  -- Arm Compute Library headers: /home/sidmen01/ComputeLibrary;/home/sidmen01/ComputeLibrary/include
  -- Could NOT find Doxygen (missing: DOXYGEN_EXECUTABLE) 
  -- Could NOT find Doxyrest (missing: DOXYREST_EXECUTABLE) 
  -- Found PythonInterp: /usr/bin/python (found suitable version "3.10.12", minimum required is "2.7") 
  -- Could NOT find Sphinx (missing: SPHINX_EXECUTABLE) 
  -- Found Git: /usr/bin/git (found version "2.34.1") 
  -- Enabled testing coverage: CI
  -- Enabled workload: TRAINING
  -- Enabled primitives: ALL
  -- Enabled primitive CPU ISA: ALL
  -- Enabled primitive GPU ISA: ALL
  -- Enabled GeMM kernels ISA: ALL
  -- Primitive cache is enabled
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/sidmen01/oneDNN/build

• git hash: 094cc1dda4a24ccf0a54987a34c4475e00a926f9

Steps to reproduce

Patch /tests/benchdnn/lnorm/ref_lnorm.cpp:

--- a/tests/benchdnn/lnorm/ref_lnorm.cpp
+++ b/tests/benchdnn/lnorm/ref_lnorm.cpp
@@ -18,6 +18,8 @@

 #include "lnorm/lnorm.hpp"

+#include <stdexcept>
+
 namespace lnorm {

 void compute_ref_fwd(const prb_t *prb, const args_t &args) {
@@ -52,6 +54,19 @@ void compute_ref_fwd(const prb_t *prb, const args_t &args) {
         float svar = var.get_elem(n);
         float sqrt_var = sqrtf(svar + prb->eps);

+        float sum = 0, calc_mean, num_elements = prb->c;
+
+        for (int i = 0; i < num_elements; ++i) {
+            float data = src.get_elem(i);
+            sum += data;
+        }
+
+        calc_mean = sum / num_elements;
+
+        if (calc_mean != smean) {
+            throw std::runtime_error("smean differs from calculated mean!\n");
+        }
+
         for (int64_t c = 0; c < prb->c; ++c) {
             float gamma = (use_sc ? sc.get_elem(c) : 1.0f) / sqrt_var;
             float beta = use_sh ? sh.get_elem(c) : 0;

Then run the benchmark with OMP_NUM_THREADS=1 ONEDNN_VERBOSE=all tests/benchdnn/benchdnn -v5 --lnorm --flags=G 70x70

Observed behavior

The added exception is thrown indicating that the calculated mean of the src data does not match the mean stored in smean.

Expected behavior

I would assume that smean holds the actual mean of the data. In case I have misunderstood something I would greatly appreciate any clarification. Thank you.

[dzarukin]

Hi @smarm0, when the --flag=G is specified, all the library does is reads the mean and variance values (they are inputs in this case, not outputs) and uses them as the part of norm operation. It doesn't have to use the mean and variance of the data to perform the check of this mode. Thus, they don't match in benchdnn either.

[ghost]

@dzarukin Thank you for the clarification, much appreciated.