MacOS builds fail due to "-dead_strip_dylibs" in LDFLAGS

[mattchan-tencent]

TL;DR: Builds on MacOS using mkl fail because the -dead_strip_dylibs flag is set by default in LDFLAGS and that causes libmkl_sequential, libmkl_core, etc to get optimized out in the final binary.

---

The following link command:

$BUILD_PREFIX/bin/x86_64-apple-darwin13.4.0-gfortran -Wl,-pie -Wl,-headerpad_max_install_names -Wl,-dead_strip_dylibs -Wl,-rpath,$PREFIX/lib -L$PREFIX/lib -o pw.x \
   pwscf.o  libpw.a ... <more .a libs> -lmkl_intel_lp64  -lmkl_sequential -lmkl_core -lpthread   <repeat link libs>

generates this incorrect DYLIB debug (missing mkl_sequential, mkl_core):

dyld: loaded: <5DDFA61A-C829-3B91-BC98-271F81B355BF> $WORKDIR/work/bin/pw.x
dyld: loaded: <803DD8AC-4799-3838-8344-55D4FC373664> $PREFIX/lib/libmkl_intel_lp64.dylib
dyld: loaded: <87036AAE-73CD-3C93-B6C8-E7C4F766FE0A> $PREFIX/lib/libgfortran.3.dylib
... <MacOS libs>

When you remove the dead_strip_dylibs you get the correct DYLIB debug:

dyld: loaded: <4A8D5DBE-1988-3E5B-8C36-660876CCEB1F> $WORKDIR/work/bin/pw.x
dyld: loaded: <803DD8AC-4799-3838-8344-55D4FC373664> $PREFIX/lib/libmkl_intel_lp64.dylib
dyld: loaded: <4EDD7B70-4F4B-3A52-A420-25B282BCAC65> $PREFIX/lib/libmkl_sequential.dylib
dyld: loaded: <735589BA-904A-3D27-9264-01DB2C877ADF> $PREFIX/lib/libmkl_core.dylib
dyld: loaded: <87036AAE-73CD-3C93-B6C8-E7C4F766FE0A> $PREFIX/lib/libgfortran.3.dylib
... <MacOS libs>

I don't know what the correct long term solution is. Removing that line will eventually run into the MacOS symbol limit.

---

Additional debug info:

[I] mattchan@MATTCHAN-MB0 ~/P/q/q-e (qe-6.5-conda)> conda build --version                                                                                                                       (packaging) 
conda-build 3.18.11
[I] mattchan@MATTCHAN-MB0 ~/P/q/q-e (qe-6.5-conda)> conda info                                                                                                                                  (packaging) 

     active environment : packaging
    active env location : /Users/mattchan/miniconda3/envs/packaging
            shell level : 2
       user config file : /Users/mattchan/.condarc
 populated config files : 
          conda version : 4.8.2
    conda-build version : not installed
         python version : 3.7.1.final.0
       virtual packages : __osx=10.15.3
       base environment : /Users/mattchan/miniconda3  (writable)
           channel URLs : https://repo.anaconda.com/pkgs/main/osx-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/mattchan/miniconda3/pkgs
                          /Users/mattchan/.conda/pkgs
       envs directories : /Users/mattchan/miniconda3/envs
                          /Users/mattchan/.conda/envs
               platform : osx-64
             user-agent : conda/4.8.2 requests/2.22.0 CPython/3.7.1 Darwin/19.3.0 OSX/10.15.3
                UID:GID : 502:20
             netrc file : None
           offline mode : False

Conda meta.yaml requirements section:

requirements:
  build:
    - git
    - {{ compiler("fortran") }}
    - {{ compiler("c") }}
    - automake
    - autoconf 
    - libtool
    - make
  host:
    - mkl
    - mkl-devel
  run:
    - mkl 

[mingwandroid]

Do you have a recipe in particular here? Clearly we don't want to avoid dead stripping in general, so I'd like to know why it goes wrong.

Can you also possibly test with conda-forge's macOS compilers too?

[mattchan-tencent]

Thanks for your help!

Yes, here's the recipe in question. If you uncomment the LDFLAGS export in build.sh then it should work again. (Also, I've short-circuited the build before the install phase to test the binary quicker.)

conda.zip

[Rullec]

Same problem, it is fully due to the initialization of anaconda.

[mattchan-tencent]

@mingwandroid also, sorry it took awhile, but this problem doesn't happen with conda-forge. Their LDFLAGS don't include -dead-strip-dylibs.

[mathstuf]

I've run across similar behavior with executables using libpython through other libraries here: conda-forge/paraview-feedstock#107. I imagine that something similar is happening here: MKL APIs are being used from a library that does not link to MKL (for whatever reason), then the executable needs to link to it to anchor the MKL APIs somewhere. The linker then notices "no direct MKL usage" and this flag then "helpfully" strips those libraries.

In general, projects are not ready for -dead_strip_dylibs and without a -no_dead_strip_dylibs flag, are basically helpless in the face of it (feel free to yell into the void and hope Apple hears you). There should be an easy way to tell Anaconda "this project can't work with it, remove the flag".

[ScottBailey]

Closing due to age.

[mathstuf]

Has it actually been addressed? I suspect it is still a thing.