MMCLAB crash dump with AppleSilicon M1

andreafarina commented 3 months ago

Dear Dr. Fang, I'm Andrea Farina from Milano, Italy. I have an issue using mmclab on my M1 MacBook Pro. I downloaded the last release (2024.2) of the whole suite but in this version the mmc/mexmaca64 file was missing, so I downloaded the mmclab_arm64 package of the nightly build and I've substituted the relative folder in the main MCXtudio path. I've done the same with mmc_arm64 and mcxcl_arm64. I've checked in MATLAB R2023b with the command mexext that my platform is native AppleSilicon M1. What I've realized is that the binary mmc from command line, after assigning the permission of executable file, runs smoothly, but the mmclab in Matlab always generates a crush-dump error. Could you please check this issue?

I've also checked the mcxlabcl in Matlab and it works fine!

Thanks a lot best regards

Andrea

fangq commented 3 months ago

hi @andreafarina, hope things are going well!

thanks for reporting this. I don't have an M1 mac, but when testing the nightly build on my M2 mac, it seems to work ok, see log attached.

to troubleshoot this issue, can you confirm

standalone mmc executable works on your machine - by running cd mmc/examples/validation/ && ./run_tess.sh
in matlab, can you type which mmc and see if you had added path to the arm64 mmc.mexmaca64 file?
what is your mmclab('gpuinfo') output? if you type mmc and enter in matlab, do you see output?

can you run otool -L mmc.mexmaca64 and see if all libraries are present? on my M2 mac, it looks like this:

fangq@~/space/git/Temp/mmclab/mmclab$ otool -L mmc.mexmaca64
mmc.mexmaca64:
/System/Library/Frameworks/Cocoa.framework/Versions/A/Cocoa (compatibility version 1.0.0, current version 24.0.0)
/System/Library/Frameworks/OpenCL.framework/Versions/A/OpenCL (compatibility version 1.0.0, current version 1.0.0)
@rpath/libmx.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libmex.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 1700.255.5)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.120.2)

fangq@~/space/git/Temp/mmclab/mmclab$ matlab -nodesktop
WARNING: package sun.awt.X11 not in java.desktop
WARNING: package sun.awt.X11 not in java.desktop
2024-08-17 11:38:34.598 MATLAB_maca64[81605:53876675] [JRSAppKitAWT markAppIsDaemon]: Process manager already initialized: can't fully enable headless mode.

                                                                                                                                 < M A T L A B (R) >
                                                                                                                       Copyright 1984-2024 The MathWorks, Inc.
                                                                                                                   R2024a Update 3 (24.1.0.2603908) 64-bit (maca64)
                                                                                                                                     May 2, 2024

PostVMInit failed to initialize com.mathworks.mwswing.MJStartupForDesktop

To get started, type doc.
For product information, visit www.mathworks.com.

>> mmc
MMCLAB v2024.2
Usage:
    [flux,detphoton]=mmclab(cfg);

Please run 'help mmclab' for more details.
>> which mmclab     
/Users/fangq/space/git/Temp/mmclab/mmclab/mmclab.m
>> addpath(pwd)
>> cd example/
>> mexext

mexmaca64

>> mmclab('gpuinfo')
Platform [0] Name Apple
============ GPU device ID 0 [1 of 1]: Apple M2 Pro  ============
 Device 1 of 1:     Apple M2 Pro
 Compute units   :  16 core(s)
 Global memory   :  11453251584 B
 Local memory    :  32768 B
 Constant memory :  1073741824 B
 Clock speed     :  1000 MHz
 Vendor name    :   AppleGPU
 Auto-thread    :   12288
 Auto-block     :   64

ans = 

  struct with fields:

          name: 'Apple M2 Pro'
            id: 1
      devcount: 1
         major: 0
         minor: 0
     globalmem: 1.1453e+10
      constmem: 1.0737e+09
     sharedmem: 32768
      regcount: 0
         clock: 1000
            sm: 16
          core: 16
     autoblock: 64
    autothread: 12288
       maxgate: 1

>> which mmc
/Users/fangq/space/git/Temp/mmclab/mmclab/mmc.mexmaca64
>> cfg.nphoton = 1e6;
>> [cfg.node, face, cfg.elem] = meshabox([0 0 0], [60 60 30], 6);
generating tetrahedral mesh from closed surfaces ...
creating volumetric mesh from a surface mesh ...
volume mesh generation is complete
>> cfg.elemprop = ones(size(cfg.elem, 1), 1);
>> cfg.srcpos = [30 30 0];
>> cfg.srcdir = [0 0 1];
>> cfg.prop = [0 0 1 1; 0.005 1 0 1.37];
>> cfg.tstart = 0;
>> cfg.tend = 5e-9;
>> cfg.tstep = 5e-9;
>> cfg.debuglevel = 'TP';
>> cfg.issaveref = 1;  % in addition to volumetric fluence, also save surface diffuse reflectance
>> cfg.method = 'elem';
>> 
>> %% run the simulation
>> 
>> flux = mmclab(cfg);
Launching MMCLAB - Mesh-based Monte Carlo for MATLAB & GNU Octave ...
Running simulations for configuration #1 ...
mmc.nphoton=1e+06;
mmc.nn=6512;
mmc.elem=[34605,4];
mmc.ne=34605;
mmc.srcpos=[30 30 0];
mmc.srcdir=[0 0 1 0];
mmc.prop=1;
mmc.tstart=0;
mmc.tend=5e-09;
mmc.tstep=5e-09;
mmc.debuglevel='TP';
mmc.issaveref=1;
mmc.method='elem';
mmc.facenb=[34605,4];
mmc.evol=34605;
mmc.e0=34344;
    done    376
simulating ... 
###############################################################################
#                     Mesh-based Monte Carlo (MMC) - OpenCL                   #
#          Copyright (c) 2010-2024 Qianqian Fang <q.fang at neu.edu>          #
#              https://mcx.space/#mmc  &  https://neurojson.io/               #
#                                                                             #
#Computational Optics & Translational Imaging (COTI) Lab  [http://fanglab.org]#
#   Department of Bioengineering, Northeastern University, Boston, MA, USA    #
###############################################################################
#    The MCX Project is funded by the NIH/NIGMS under grant R01-GM114365      #
###############################################################################
#  Open-source codes and reusable scientific data are essential for research, #
# MCX proudly developed human-readable JSON-based data formats for easy reuse.#
#                                                                             #
#Please visit our free scientific data sharing portal at https://neurojson.io/#
# and consider sharing your public datasets in standardized JSON/JData format #
###############################################################################
$Rev::efc749$v2024.2 $Date::2024-06-09 15:58:33 -04$ by $Author::Qianqian Fang$
###############################################################################
- code name: [MMC-OpenCL] compiled with OpenCL version [1]
- compiled with: [RNG] xorshift128+ RNG [Seed Length] 4
initializing streams ...    init complete : 0 ms
Building kernel with option: -cl-mad-enable -DMCX_USE_NATIVE -DMCX_SIMPLIFY_BRANCH -DMCX_VECTOR_INDEX -DMCX_SRC_PENCIL   -DUSE_ATOMIC -DMCX_SAVE_DREF -DMCX_DO_REFLECTION -DUSE_BLBADOUEL
build program complete : 74 ms
- [device 0(1): Apple M2 Pro] threadph=81 oddphotons=4672 np=1000000.0 nthread=12288 nblock=64 repetition=1
set kernel arguments complete : 75 ms 3
lauching mcx_main_loop for time window [0.0ns 5.0ns] ...
simulation run# 1 ... 
Progress: [==============================================================] 100%
kernel complete:    1505 ms
retrieving flux ...     transfer complete:        1508 ms
normalizing raw data ...    simulated 1000000 photons (1000000) with 1 devices (ray-tet 255575040)
MCX simulation speed: 699.30 photon/ms
total simulated energy: 1000000.00  absorbed: 26.29546%
(loss due to initial specular reflection is excluded in the total)
    done    1891

andreafarina commented 3 months ago

Dear @fangq , thank you very much for your super fast reply! I've tried all your steps and the only one that give me an error is mmclab('gpuinfo'). I tried to run matlab from the terminal, as in your screenshot, and here is the crush dump error:

>> mcxsuite_addpath
>> mmc
MMCLAB v2024.2
Usage:
    [flux,detphoton]=mmclab(cfg);

Please run 'help mmclab' for more details.
>> mmclab('gpuinfo')
OMP: Error #15: Initializing libomp.a, but found libomp.dylib already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. 
That is dangerous, since it can degrade performance or cause incorrect results. 
The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, 
e.g. by avoiding static linking of the OpenMP runtime in any library. 
As an unsafe, unsupported, undocumented workaround you can set the environment variable
 KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, 
but that may cause crashes or silently produce incorrect results. 
For more information, please see http://openmp.llvm.org/

MATLAB is exiting because of fatal error
zsh: killed     ./matlab -nodesktop

After that I've set the environment variable as suggested and now it works! Actually by running matlab in -nodesktop allowed to catch the error message!! Thanks a lot again! Best regards Andrea

fangq commented 3 months ago

@andreafarina, glad to hear that you have found the source of the issue. I am also surprised to know mac is crashing an application with statically linked binaries.

I just disabled the static linking with libomp.a on arm64 mac only with the above commit. can you please download the nightly build again and let me know if the mex file can run without needing to set the KMP_DUPLICATE_LIB_OK flag (you can call unset KMP_DUPLICATE_LIB_OK in a shell) before starting matlab.

andreafarina commented 3 months ago

Dear @fangq thanks a lot for fixing the issue. After upgrading the mmc.mexmaca64 file with the last one the execution goes smoothly without the need of setting the environment variable. The first time you run mmclab, OsX blocks the execution for security reasons but after allowing its execution in the "Privacy & Security" Settings, no problem! Thanks a lot! Andrea

fangq / mmc

MMCLAB crash dump with AppleSilicon M1 #89