OF7 and openmpi for Mac M1

[matteoicardi]

I am having issues linking the mpi library on a Mac M1, it looks like the paths are all correct but the pstream library is not linked

wmake mpi ... xcrun c++ -arch x86_64 -std=c++14 -Ddarwin64 -DWM_ARCH_OPTION=64 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wno-unused-parameter -Wno-overloaded-virtual -Wno-unused-variable -Wno-unused-local-typedef -Wno-invalid-offsetof -Wno-deprecated-register -Wno-undefined-var-template -O3 -DNoRepository -ftemplate-depth-100 -DOMPI_SKIP_MPICXX -I/opt/homebrew/Cellar/open-mpi/4.1.0/include -IlnInclude -I. -I/Users/xxxxx/OpenFOAM/OpenFOAM-7/src/OpenFOAM/lnInclude -I/Users/xxxxx/OpenFOAM/OpenFOAM-7/src/OSspecific/POSIX/lnInclude -fPIC -isysroot /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk -Wl,-dylib,-undefined,dynamic_lookup,-w /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UOPwrite.o /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UIPread.o /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/UPstream.o /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/src/Pstream/mpi/PstreamGlobals.o -L/Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32OptSYSTEMOPENMPI/lib \ -L/opt/homebrew/opt/libevent/lib -L/opt/homebrew/Cellar/open-mpi/4.1.0/lib -lmpi -o /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib

blockMesh dyld: Symbol not found: _ompi_mpi_byte Referenced from: /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib Expected in: flat namespace in /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib Abort trap: 6

env | grep DYLD FOAM_DYLD_LIBRARY_PATH=/Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system:/opt/homebrew/Cellar/open-mpi/4.1.0/lib:/Users/xxxxx/OpenFOAM/xxxxx-7/platforms/darwin64ClangDPInt32Opt/lib:/Users/xxxxx/OpenFOAM/site/7/platforms/darwin64ClangDPInt32Opt/lib:/Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib:/Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/dummy

otool -L /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib: /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib (compatibility version 0.0.0, current version 0.0.0) /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 904.4.0) /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1) `

openmpi seems to be installed correctly with homebrew but while for intel Mac it was in /usr/local, it is now in /opt/homebrew

[mrklein]

Hi,

But is there the symbol libmpi? Here is output on my laptop (Intel):

$ nm /usr/local/lib/libmpi.dylib | grep _ompi_mpi_byte
00000000000da900 D _ompi_mpi_byte
$ nm $FOAM_LIBBIN/openmpi-system/libPstream.dylib  | grep _ompi_mpi_byte
                 U _ompi_mpi_byte

Since I do not have access to M1 Macs, I can only guess what is wrong.

[matteoicardi]

nm /opt/homebrew/Cellar/open-mpi/4.1.0/lib/libmpi.dylib | grep _ompi_mpi_byte
00000000000de8d0 D _ompi_mpi_byte

nm $FOAM_LIBBIN/openmpi-system/libPstream.dylib  | grep _ompi_mpi_byte
                 U _ompi_mpi_byte

it looks the same here. the problem seems to be that the libmpi.dylib file is not listed in the output of otool as it should, although the correct linking folder is specified when compiling

[matteoicardi]

this should be how the otool should look like (with an intel laptop)

>> otool -L /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib
/Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib:
    /Users/xxxxx/OpenFOAM/OpenFOAM-7/platforms/darwin64ClangDPInt32Opt/lib/openmpi-system/libPstream.dylib (compatibility version 0.0.0, current version 0.0.0)
    /usr/local/opt/open-mpi/lib/libmpi.40.dylib (compatibility version 61.0.0, current version 61.2.0)
    /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 800.7.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)

[mrklein]

It seems, I misinterpreted the problem.

blockMesh in the initial message was a reference to an attempt to execute the utility? If this is the case, there is a problem with DYLD_LIBRARY_PATH variable. For some reason you only have FOAM_DYLD_LIBRARY_PATH. And I do not know why. What shell do you use?

[matteoicardi]

yes, It is strange indeed, I have bash 5.1.4 and clang 12.0.0 for arm64-apple-darwin20.3.0 DYLD_LIBRARY_PATH does not appear in env but it does when echo $DYLD_LIBRARY_PATH (I think it must have to do with some mac security). Anyway it's the same in the intel and the compilation line seems ok anyway. I suspect the problem might be in the compiler

[matteoicardi]

actually in my intel FOAM_DYLD_LIBRARY_PATH is empty while it's equal to DYLD_LIBRARY_PATH in the M1

[BrushXue]

I see you are compiling with x86. Do you have any idea compiling it natively on ARM64? I got the following error:

xcrun c++ -std=c++14 -m64 -pthread -ftrapping-math -DOPENFOAM=2012 -DWM_DP -DWM_LABEL_SIZE=32 -Wall -Wextra -Wold-style-cast -Wnon-virtual-dtor -Wno-unused-parameter -Wno-invalid-offsetof -Wno-undefined-var-template -Wno-unknown-warning-option  -O3  -DNoRepository -ftemplate-depth-100  -iquote. -IlnInclude -I/Users/xuegy/OpenFOAM/OpenFOAM-v2012/src/OpenFOAM/lnInclude -I/Users/xuegy/OpenFOAM/OpenFOAM-v2012/src/OSspecific/POSIX/lnInclude   -fPIC -c signals/sigFpe.C -o /Users/xuegy/OpenFOAM/OpenFOAM-v2012/build/darwin64ClangDPInt32Opt/src/OSspecific/POSIX/signals/sigFpe.o
warning: overriding currently unsupported use of floating point exceptions on this target [-Wunsupported-floating-point-opt]
In file included from signals/sigFpe.C:48:
signals/feexceptErsatz.H:39:24: error: no member named '__control' in 'fenv_t'
    old_excepts = fenv.__control & FE_ALL_EXCEPT;
                  ~~~~ ^
signals/feexceptErsatz.H:42:10: error: no member named '__control' in 'fenv_t'
    fenv.__control &= ~new_excepts;
    ~~~~ ^
signals/feexceptErsatz.H:43:10: error: no member named '__mxcsr' in 'fenv_t'
    fenv.__mxcsr   &= ~(new_excepts << 7);
    ~~~~ ^
signals/feexceptErsatz.H:60:24: error: no member named '__control' in 'fenv_t'
    old_excepts = fenv.__control & FE_ALL_EXCEPT;
                  ~~~~ ^
signals/feexceptErsatz.H:63:10: error: no member named '__control' in 'fenv_t'
    fenv.__control |= new_excepts;
    ~~~~ ^
signals/feexceptErsatz.H:64:10: error: no member named '__mxcsr' in 'fenv_t'
    fenv.__mxcsr   |= new_excepts << 7;
    ~~~~ ^
1 warning and 6 errors generated.
make: *** [/Users/xuegy/OpenFOAM/OpenFOAM-v2012/build/darwin64ClangDPInt32Opt/src/OSspecific/POSIX/signals/sigFpe.o] Error 1

I don't know where to find the documentation to deal with sigfpe in Apple M1.

[mrklein]

@BrushXue

Hi,

I do not have access to the hardware, so no I do not have ideas on how to fix it. If you can provide remote access, I can try to figure out how to fix the error. I do not plan to buy M1 hardware at least till 2022.

fent_t structure is different on M1, there is no control field. The easiest way to suppress the error is to disable sigFpe on Apple platforms by removing all occurrences of `defined(APPLE__)fromsigFre.Cand removing inclusion onfeexceptErsatz.H` in the beginning of the file.

[BrushXue]

Some updates: I disabled __APPLE__ macros in sigFpe.C and removed -ftrapping-math then OpenFOAM can compile without any issue. I have to say Apple did this job better than I expected. It took 36 minutes to compile the whole package. The aluminum case is just little warm and I can't hear fan spinning. That's much better than my 2018 MBP which took 2hr+ and the fan was taking off. I tested the motorbike case. It took 155s to run 500 iterations with 4 cores. As a comparison, my 2018 MBP (i5-8259U, 4 cores) took 232s. @matteoicardi Would you like to share your M1 result with Rosetta 2? @mrklein The only issue left is sigFpe. I would like to provide remote access but I'm in the university and don't have public IP so maybe you have to use something like teamviewer.

[matteoicardi]

Thanks @BrushXue can you share a patch for the changes you have made to compile natively for ARM? I guess you have no issues with MPI?

[BrushXue]

Thanks @BrushXue can you share a patch for the changes you have made to compile natively for ARM? I guess you have no issues with MPI?

Yes I’ve no issue with MPI. Did you mix up x86 OpenFOAM with ARM OpenMPI? Have you tried to install homebrew in x86 mode?

[matteoicardi]

Solved it! Thanks for the hint @BrushXue! I did mix x86 and ARM. Following now your suggestion of just removing all APPLE/Darwin macros in sigFpe.C and compiling with m64 everything works, including MPI! I no longer have the benchmark data I ran but at a first glance it looks like the speed is comparable, I can do some test now against my Intel Macbook

[BrushXue]

Update: I found 2 issues 1. mpi runs with runParallel command but if I use mpirun -np... in a script it gives

--> FOAM FATAL ERROR: (openfoam-2012)
The dummy Pstream library cannot be used in parallel mode

    From static bool Foam::UPstream::init(int &, char **&, const bool)
    in file UPstream.C at line 50.

FOAM exiting

This issue was caused by zsh(So etc/bashrc is not 100% compatible with zsh). Using bash will solve the problem.

2. When compiling the code again, it will start over. This is confusing as .o and .dep files are available. There's no point to build again. Maybe Apple made some changes to Clang? Not happening today. Really weird.

Add these two lines as the path has changed in ARM64 Homebrew

export CPATH=/opt/homebrew/include
export LIBRARY_PATH=/opt/homebrew/lib

[matteoicardi]

@BrushXue has anyone solved the issue with sigFpe.C on Apple Silicon?

[BrushXue]

@BrushXue has anyone solved the issue with sigFpe.C on Apple Silicon?

No. Currently Apple Clang doesn't support sigfpe on ARM64. https://developer.apple.com/forums/thread/689159