Barretenberg build failure on Apple M1 processor

[jfecher]

Attempting to compile barretenberg on an apple M1 currently fails during cmake configuration and fixing it seems non-trivial.

First, configuring cmake for barretenberg currently gives an error due to M1s storing binaries at different locations than intel-based macs:

 --- stderr
  CMake Error at CMakeLists.txt:9 (project):
    The CMAKE_CXX_COMPILER:

      /usr/local/opt/llvm/bin/clang++

    is not a full path to an existing compiler tool.

    Tell CMake where to find the compiler by setting either the environment
    variable "CXX" or the CMake cache entry CMAKE_CXX_COMPILER to the full path
    to the compiler, or to the compiler name if it is in the PATH.

  CMake Error at CMakeLists.txt:9 (project):
    The CMAKE_C_COMPILER:

      /usr/local/opt/llvm/bin/clang

    is not a full path to an existing compiler tool.

    Tell CMake where to find the compiler by setting either the environment
    variable "CC" or the CMake cache entry CMAKE_C_COMPILER to the full path to
    the compiler, or to the compiler name if it is in the PATH.

On my system, the path for apple clang is at /usr/bin/clang instead. Using this gives an error for -fopenmp not being supported however. Using the homebrew clang at /opt/homebrew/opt/llvm/bin/clang works instead. Then changing barretenberg/cmake/toolchains/x86_64-apple-clang.cmake from

set(CMAKE_CXX_COMPILER "/usr/local/opt/llvm/bin/clang++")
set(CMAKE_C_COMPILER "/usr/local/opt/llvm/bin/clang")

to

set(CMAKE_C_COMPILER $ENV{CC})
set(CMAKE_CXX_COMPILER $ENV{CXX})

works as a quick band-aid to the problem, though perhaps cmake should detect an M1 processor and defer to a new arm64-apple-clang.cmake file instead. Running cmake . again from there we run into a similar error to #65:

CMake Error at CMakeLists.txt:46 (include):
include could not find load file:

    cmake/build.cmake

For some reason, this last error isn't present on the branch kw/noir-dsl-mac. Continuing on that branch, building via make fails due to clang (13.0.0) not supporting -march=native on an apple M1:

$ make
[  1%] Building CXX object src/aztec/env/CMakeFiles/env_objects.dir/logstr.cpp.o
clang-13: fatal error: the clang compiler does not support '-march=native'
make[2]: *** [src/aztec/env/CMakeFiles/env_objects.dir/logstr.cpp.o] Error 1
make[1]: *** [src/aztec/env/CMakeFiles/env_objects.dir/all] Error 2
make: *** [all] Error 2
exit 2

arch.cmake can be patched up a bit to get past this:

include(CheckCCompilerFlag)
check_c_compiler_flag("-march=native" marchNativeSupported)
if(NOT WASM AND marchNativeSupported)
    add_compile_options(-march=native)
endif()

After which make gives a different error since libomp is installed to a different directory (/opt/homebrew/lib/libomp.a) instead of the expected:

make[2]: *** No rule to make target `/usr/local/lib/libomp.a', needed by `src/aztec/numeric/numeric_tests'.  Stop.

After manually creating a link for the library to the expected location the build fails in a dependency:

/Users/.../barretenberg/_deps/benchmark-src/src/complexity.cc:85:10: error: variable 'sigma_gn' set but not used [-Werror,-Wunused-but-set-variable]
  double sigma_gn = 0.0;
         ^

This error is fatal unless we build in debug mode instead. Compiling in debug gets us a bit further but fails here instead:

/Users/.../barretenberg/src/aztec/ecc/curves/bn254/scalar_multiplication/../../../groups/group_impl_asm.tcc:107:30: fatal error: unknown token in expression
        __asm__ __volatile__("xorq %%r8, %%r8                              \n\t"
                             ^
<inline asm>:1:7: note: instantiated into assembly here
        xorq %r8, %r8                              
             ^
1 error generated.

[jfecher]

Since the assembly that failed to compile doesn't seem to be ARM, I think the culprit could be these lines in CMakeLists.txt:

if(ARM)
    message(STATUS "Compiling for ARM.")
    set(DISABLE_ASM ON)
    set(RUN_HAVE_STD_REGEX 0)
    set(RUN_HAVE_POSIX_REGEX 0)
endif()

in which the if(ARM) is false on my M1, causing DISABLE_ASM to remain off.

[skardas]

Is there anybody who solve the compilation error on Mac?

[zac-williamson]

I've pushed a temporary fix for the compiler issues on M1 Macs.

Running bootstrap.sh APPLE_M1 should work now. Multithreading is disabled, however, as there are lingering linker errors with OpenMP.