Improve user experience with macOS compilers

[chrisburr]

@conda-forge/core @henryiii @SylvainCorlay This is a suggestion for modifying the macOS compiler packages to make them more easily usable outside of conda-build. I'm primarily thinking of users who do data analysis using C++ but it is generally useful for anyone who want to compile against the conda ecosystem on macOS.

Problem

Currently using the macOS compilers are broken¹ due to the fact that conda-forge can't create a package for the 10.9 SDK. Example:

$ clang++ energy_test.cpp -o energy_test -std=c++14 -I .                                                                                    (debug_compilers)
In file included from energy_test.cpp:1:
In file included from /Users/cburr/miniconda3/envs/debug_compilers/bin/../include/c++/v1/algorithm:638:
In file included from /Users/cburr/miniconda3/envs/debug_compilers/bin/../include/c++/v1/cstring:61:
/Users/cburr/miniconda3/envs/debug_compilers/bin/../include/c++/v1/string.h:61:15: fatal error: 'string.h' file not found
#include_next <string.h>
              ^~~~~~~~~~
1 error generated.

This can be worked around by explicitly passing the path to the 10.9 SDK using -isysroot or by setting SDKROOT:

clang++ energy_test.cpp -o energy_test -std=c++14 -I . -isysroot $HOME/Development/MacOSX-SDKs/MacOSX10.9.sdk
# or
export SDKROOT=$HOME/Development/MacOSX-SDKs/MacOSX10.9.sdk
clang++ energy_test.cpp -o energy_test -std=c++14 -I .

Current solution used by cling and root

This is also a problem for root and cling as they provide a C++ interpreter causing the SDK to be needed at runtime. This uses a variant of clangdev that includes patches that are needed for cling.

To make this work more reliably in conda an additional patch has been added which tries to search for the macOS SDK using this logic:

1. Use the CONDA_BUILD_SYSROOT env variable if it is set, is an absolute path that exists and isn't /.
2. Use the SDKROOT env variable if it is set, is an absolute path that exists and isn't /.
3. Use /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk if it contains /usr/include/sys/types.h
4. Use /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk if it contains /usr/include/sys/types.h
5. Use / if /usr/include/sys/types.h exists (i.e. macOS <= 10.13)
6. Run xcode-select -p and check if the command line tools are installed in a non standard location
7. Run xcode-select -p and check if xcode is installed in a non standard location

root also prints a warning when activating the environment so users know something is potentially wrong (the second line refers to ROOT's compilation syntax):

$ conda create --name example_root_environment python=3.7 root
$ conda activate example_root_environment
WARNING: Compiling likely won't work unless you: download the macOS 10.9 SDK, set CONDA_BUILD_SYSROOT and reactivate the environment.
You can probably ignore this warning and just omit + or ++ when executing ROOT macros.

Proposed solution

I propose adding the patch mentioned above to all of the macOS clang packages (clangdev, clang_osx-64, clangxx_osx-64, ...) with the following modifications:

• Check that the SDK version is 10.9 if MACOSX_DEPLOYMENT_TARGET is not set.²
• Print a warning between steps 2 and 3 stating that it is likely to fail and CONDA_BUILD_SYSROOT should be set.

The packages in this feedstock will also have an post setup hook added to warn the user if CONDA_BUILD_SYSROOT is not set, as is currently done in ROOT.

Does this seem reasonable? If so, I can prepare a pull request for the llvm-compilers-feedstock in AnacondaRecipes/aggregate. If not, can you think of an alternative for making it easier to use conda's compilers on macOS?

Footnotes

¹ This is less broken on <= 10.13 due to the SDK being available in /, though I suspect there are incompatibilities due to the SDK version being wrong. Linking with the conda compilers is definitely broken when using 10.14 SDK. ² Assuming it's possible to easily inspect the version of the SDK.

[SylvainCorlay]

:+1: on this!

Crazy idea: maybe part of the sysroot detection logic changes could go upstream in clang.

[minrk]

One question: Why is / ignored if it is explicitly given? Seems like if a user asks you to use a sysroot, that should be respected, especially if it works.

I like the idea of making it more likely that conda compilers work without intervention for user compiliation, but I think the proposed level of complexity for the search is too much right now.

Here are two examples:

1. If the user has mac 10.13 or earlier and xcode installed, no arguments are needed and / works. No intervention or search is needed.

2. if the user has macOS 10.14 and installed compiler tools:

   xcode-select --install

   and subsequently installed the headers no longer copied to /usr by default from the xcode 10 release notes:

   open /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

   then additionally, no arguments are needed and compilation with SYSROOT=/ or unspecified works without intervention or any search step.

Because of these and the fact that llvm <8 doesn't support .tbd TAPI stubs, then I think / should actually be the default and preferred to the current mac sdk root.

So I would do:

1. CONDA_BUILD_SYSROOT if set and non-empty, regardless of value
2. SDKROOT if set and non-empty, regardless of value

3. Use / (default, pass no sysroot args) possibly followed by a warning in an activation script describing setting SDKROOT and/or describing the missing installation steps of:

   xcode-select --install /Library/Developer/CommandLineTools/Packages/macOS_SDK_headers_for_macOS_10.14.pkg

   which make / a valid sysroot. if /usr/include/sys/types.h is not found.

and stop there. Basically, this amounts to the only change being checking if / is a valid sysroot and warning about how to make it one and/or install SDKs.

When llvm is out, the default can be xcrun --show-sdk-path instead of / since the .tbd workaround is no longer needed. It doesn't appear that there needs to be more than 3 levels, though (really 2: env set or default).

Re: xcode-select -p, The conda compiler packages currently query xcrun --show-sdk-path for the default value of CONDA_BUILD_SYSROOT, which I think is more appropriate than xcode-select -p, which doesn't resolve the SDK, just the developer tools root.

[minrk]

• Check that the SDK version is 10.9 if MACOSX_DEPLOYMENT_TARGET is not set.²

This, too, I think is not the right thing to do. Setting it to a conservative value is good for redistributable packages, but unnecessary and I think unhelpful for users not making conda packages. Very few users are going to have the 10.9 SDK these days, and there's no reason they should need it unless they are making redistributable packages.

• Print a warning between steps 2 and 3 stating that it is likely to fail and CONDA_BUILD_SYSROOT should be set.

I would hold the warning only if the default sdkrook (/ or xcrun --show-sdk-path) is not found. If you want to be fancy, you could check if xcrun --show-sdk-path finds .tbd files and warn that the SDK is too new, and they should use / or an old sdk. This would only be a temporary measure until llvm 8, though.

[mingwandroid]

I am looking at building boost 1.71 and boost.python is having some trouble with sysroots. Now we can read .tbd files, do you have any different suggestions about how to proceed @minrk, @chrisburr, @SylvainCorlay and @isuruf?

[chrisburr]

From my perspective this is working now .tbd files can be read and CONDA_BUILD_SYSROOT is set automatically by clang_osx-64 using xcrun --show-sdk-path. This issue can probably be closed? I guess the next step would be https://github.com/conda-forge/conda-forge.github.io/issues/865.

I am looking at building boost 1.71 and boost.python is having some trouble with sysroots.

What is the problem?

[mingwandroid]

A few things:

1. No 37 suffix on libnames (correcting via a symlink at the end is risky incase it gets embedded (likely) in metadata for other build systems at build-boost time)). I think I have got to the bottom of that (pass --python-buildid on b2 commandline).
2. Missing <library-path> in python.jam for darwin (probably same for linux).

There are a few warnings from the build about MPI. Do we want MPI here? I'd also like to add zstd support while I'm updating boost but this is a bit off topic. I'll open a PR in boost-feedstock when it's ready for review and you'll see what I have done, WIP at https://github.com/AnacondaRecipes/boost-feedstock/tree/master/recipe

[phil-blain]

Hi all, thanks for all the work put into this subject.

I just want to give a recent update on my experience using the conda compilers on recent macOS (I'm just using the compilers to compile our software against conda-installed libraries, I'm not using conda-build). @chrisburr you wrote :

From my perspective this is working now .tbd files can be read and CONDA_BUILD_SYSROOT is set automatically by clang_osx-64 using xcrun --show-sdk-path.

For clang installed by conda (from conda-forge), I have to add either -isysroot$(xcrun --show-sdk-path) (or --sysroot=$(xcrun --show-sdk-path), not sure if they do the same thing) to my clang invocation for the compiler to find the system header files, since on macOS >= 10.14 they are not installed to / (they can be installed there manually on macOS 10.14 but not on 10.15 as this part of the filesysem is readonly, unless one disables SIP).

I do activate my environment, but it is not enough for clang to find the headers. So maybe the activation script would have to be refined ? I couldn't follow everything in there. EDIT 14/06/2020: I'm pretty sure now that it's not working because our build scripts are written in csh, and the compiler packages do not have activation/deactivation scripts for csh, see https://github.com/conda-forge/compilers-feedstock/issues/20#issuecomment-600666554

Some examples of failing builds if I don't use -isysroot or --sysroot: 10.15 (Xcode 11.2.1): https://github.com/phil-blain/CICE-conda/commit/62139f733855ef6c6c9dff6dd7ea7b1036d24927/checks?check_suite_id=398537354#step:8:50 10.14 (Xcode 10.3): https://travis-ci.com/phil-blain/CICE-conda/jobs/275092694#L386 10.14 (Xcode 11.3): https://travis-ci.com/phil-blain/CICE-conda/jobs/275092695#L388

[phil-blain]

Another note: Since the activate script for clang now use xcrun --show-sdk-path, a graphical popup shows up upon environment activation when the developer tools or Xcode are not installed. Maybe we could check if they are installed before running xcrun, and show a message to the user in the console ? this would be more user-friendly, I think.

[isuruf]

PRs welcome to https://github.com/conda-forge/clang-compiler-activation-feedstock

[isuruf]

Closing this now as it's fixed for bash/zsh. If you want to support csh, please send a PR to the above repo.