unable to install hdf5-parallel on my laptop with Mac OS 15

[weipengyao]

Hello,

I am trying to install Smilei on my own laptop (again).

This time, I can't install the hdf5-parallel, from brew install iltommi/brews/smilei --HEAD --only-dependencies.

==> Installing iltommi/brews/smilei dependency: hdf5-parallel
==> cmake .. -DHDF5_ENABLE_PARALLEL=ON -DBUILD_STATIC_LIBS=OFF -DHDF5_BUILD_CPP_LIB=OFF -DHDF5_BUILD_FORTR
Last 15 lines from /Users/yz/Library/Logs/Homebrew/hdf5-parallel/01.cmake:
    make[1]: *** [cmTC_041f9] Error 1
    make: *** [cmTC_041f9/fast] Error 2

  CMake will not be able to correctly generate this project.
Call Stack (most recent call first):
  CMakeLists.txt:2 (project)

-- Configuring incomplete, errors occurred!
See also "/tmp/hdf5-parallel-20201012-20556-uot26r/hdf5-1.12.0/build/CMakeFiles/CMakeOutput.log".
See also "/tmp/hdf5-parallel-20201012-20556-uot26r/hdf5-1.12.0/build/CMakeFiles/CMakeError.log".

However, when I try to find some clues in the log file, there's nothing in the /tmp/ about hdf5-parallel. Thus, I can't provide more information.

Thanks in advance!

[iltommi]

that's really strange (I just recompiled and seems ok and Micka just did a fresh install on a new mac today). What os/version do you have? Is this a new computer or you're reinstalling on an updated version?

did you upgrade brew with brew upgrade ? Can you check if you have problems with brew doctor

If you want to see the logs, can you run: brew install --debug iltommi/brews/hdf5-parallel

now with the debug option, it will be possible to get a shell and check what happened

[weipengyao]

I am using macOS Catalina, version 10.15.6 for my laptop.

I successfully installed Smilei on my laptop a few months ago, but somehow recently it can not run, with some libraries missing. That's why I decided to recompile it again.

And then I met with the former CLANG vs. GNU problem, so I started googling about how to install open-mpi with gnu-gcc, not clang-gcc.

I failed on this and decided to start over from the very beginning. And now is hdf5-parallel issue.

I use brew upgrade several times, and brew doctor seems to be fine, with two warnings

Warning: "config" scripts exist outside your system or Homebrew directories.
`./configure` scripts often look for *-config scripts to determine if
software packages are installed, and which additional flags to use when
compiling and linking.

and

Warning: Unbrewed header files were found in /usr/local/include.
If you didn't put them there on purpose they could cause problems when
building Homebrew formulae, and may need to be deleted.

With the debug option, I get to the log files, and the error message seems to be ld issue. I attach the files here.

CMakeOutput.log CMakeError.log

Thanks for this timely reply.

[iltommi]

It looks like there's a mess with apple compilers: try to remove them and reinstall:

sudo rm -rf /Library/Developer/CommandLineTools
xcode-select --install

https://stackoverflow.com/a/63614795/2743307

[weipengyao]

In fact, I did it already. xcode-select --install gives me a message saying that Can't install the software because it is not currently available from the Software Update server., and I download Command_Line_Tools_for_Xcode_12.2_beta_2.dmg from the Apple website and installed it.

I tried this again, but still the same error.

In addition, gcc --version gives me

(base) ➜  ~ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.6)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

And from what I struggled recently, we need GNU-gcc, not CLANG-gcc. Am I right? And gcc-8 from brew is gnu, while starting from gcc-9, it becomes CLANG. I am not sure whether this may be the cause or not?

I attach the new log files here, in case it provides more information.

CMakeError.log CMakeOutput.log

[iltommi]

In principle we're telling brew to use the gcc (and not to use clang https://github.com/iltommi/homebrew-brews/blob/master/hdf5-parallel.rb#L15) coming from brew itself which should be gcc-10 (lines 19-24)

It seems that this is ok. but what I can see from the error files you posted is that there is some mess with the linker ld which comes with the apple command line stuff:

ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/local/include"
ignoring nonexistent directory "/usr/local/Cellar/gcc/10.2.0/lib/gcc/10/gcc/x86_64-apple-darwin19/10.2.0/../../../../../../x86_64-apple-darwin19/include"
ignoring nonexistent directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/Library/Frameworks"
ignoring duplicate directory "/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include"

and

ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib/libSystem.tbd' for architecture x86_64
collect2: error: ld returned 1 exit status
make[1]: *** [cmTC_51052] Error 1
make: *** [cmTC_51052/fast] Error 2

Can you check you can compile a simple hello world program:

echo -e "#include <stdio.h>\nint main(void){printf(\"Hello world\\\n\");}" | gcc-10 -xc -

?

[weipengyao]

No, this simple hello world program returns an error when compiling.

(base) ➜  ~ echo -e "#include <stdio.h>\nint main(void){printf(\"Hello world\\\n\");}" | gcc-10 -xc -
ld: unsupported tapi file type '!tapi-tbd' in YAML file '/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib/libSystem.tbd' for architecture x86_64
collect2: error: ld returned 1 exit status

And I have a couple of gcc, which is weird because I tried to uninstall them this morning.

(base) ➜  ~ gcc
gcc            gcc-10         gcc-ar-10      gcc-nm-10      gcc-ranlib-10  gccmakedep

And

(base) ➜  ~ which gcc
/usr/bin/gcc

(base) ➜  ~ which gcc-10
/usr/local/bin/gcc-10

(base) ➜  ~ gcc --version
Configured with: --prefix=/Library/Developer/CommandLineTools/usr --with-gxx-include-dir=/Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include/c++/4.2.1
Apple clang version 12.0.0 (clang-1200.0.32.6)
Target: x86_64-apple-darwin19.6.0
Thread model: posix
InstalledDir: /Library/Developer/CommandLineTools/usr/bin

(base) ➜  ~ gcc-10 --version
gcc-10 (Homebrew GCC 10.2.0) 10.2.0
Copyright (C) 2020 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

[iltommi]

Here is an explanation: gcc is the one that comes with apple and in reality it's a clang (it's a very poor old version renamed as gcc for a reason I can't figure out) but it's not the one we're using to compile hdf5. gcc-10 is the one up-to-date installed via brew that we're telling brew to use to compile hdf5.

Unfortunately even using the new gcc-10 we're still using using the linker ld that comes with the apple command line tools because to create an executable/library we compile and link. And in my opinion that link part using ld is bugged on your system.

I don't know what happened to mess all this, and that's why I suggested to reinstall all apple stuff by erasing the directory /Library/Developer/CommandLineTools and running xcode-select --install In addition remove the XCode app too (as suggested by Apple itself https://developer.apple.com/library/archive/technotes/tn2339/_index.html#//apple_ref/doc/uid/DTS40014588-CH1-HOW_CAN_I_UNINSTALL_THE_COMMAND_LINE_TOOLS_ ) as well as the directory above

I don't know what else to do right now

[weipengyao]

Thanks for your detailed explanation and suggestions.

Yes, I think this ld issue happened quite a few times on my laptop. And that's why I tried before to uninstall the apple command line tools. I uninstalled xcode a long time ago, because I know it is really unfriendly to me and the codes I want to use.

Could it be something I add in my ~/.zshrc? I try to be careful with this at the beginning, then it became quite messy. zshrc_bak.txt

Could it be the anaconda I installed? Because I think everything is fine before I tried some conda stuff.

Thanks! I will try google more about this command line tools.

[weipengyao]

In addition about the anaconda guess, I have the following:

(base) ➜  ~ which clang
/Users/yz/opt/anaconda3/bin/clang
(base) ➜  ~ clang
CLANG                     clang-cl                  clang-offload-wrapper
CLANGXX                   clang-cpp                 clang-query
clang                     clang-doc                 clang-refactor
clang++                   clang-extdef-mapping      clang-rename
clang-10                  clang-format              clang-reorder-fields
clang-9                   clang-import-test         clang-scan-deps
clang-apply-replacements  clang-include-fixer       clang-tidy
clang-change-namespace    clang-move                clangd
clang-check               clang-offload-bundler     clangd-xpc-test-client

should I try to remove these all??

[iltommi]

Could it be something I add in my ~/.zshrc? I try to be careful with this at the beginning, then it became quite messy. zshrc_bak.txt

Could it be the anaconda I installed? Because I think everything is fine before I tried some conda stuff.

In principle, to avoid messing up with user installed software, brew compiles in an empty environment.

You can check this when you build with --debug and once it breaks, enter the shell console and check the environment (env $PATH etc..)

And anyway the test

echo -e "#include <stdio.h>\nint main(void){printf(\"Hello world\\\n\");}" | gcc-10 -xc -

isn't using any other software than gcc-10 (which will call the system ld) so it's not about conda and similar

[weipengyao]

surprisingly, I have:

(base) ➜  ~ which ld
/Users/yz/opt/anaconda3/bin/ld

This is not good, right?

[iltommi]

ok, seems like I underestimated the mess potential of conda...

try to remove all modifications to the PATH done by conda and retry

[weipengyao]

I commented the conda-related PATH in the rc file, at least now the hello world test is good.

I am trying to install smilei again.

[iltommi]

It looks like conda does ship the compiler tools on their side https://docs.conda.io/projects/conda-build/en/latest/resources/compiler-tools.html

but it seems you need to install aside the SDK :

The macOS compilers require the macOS 10.9 SDK. The SDK license prevents it from being bundled in the conda package. We know of 2 current sources for the macOS 10.9 SDK:

    https://github.com/devernay/xcodelegacy

    https://github.com/phracker/MacOSX-SDKs

conda is a rabbit hole to me.

[weipengyao]

Thanks for the additional information.

But for me, I will delete conda on my laptop, and I am not installing it again, ever...

Now the hdf5-parallel has been successfully installed.

So many thanks! I suffered from this for a very long time...

[weipengyao]

BTW, is it ok if the python3 in my laptop shows a Clang-12:

➜  Smilei git:(work) ✗ python3
Python 3.8.6 (default, Oct  8 2020, 14:06:32)
[Clang 12.0.0 (clang-1200.0.32.2)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>>

How can I use brew to install the GNU-python?

[iltommi]

By default brew uses the apple gcc which is a clang. This is my python:

Python 3.8.6 (default, Oct  8 2020, 14:07:53) 
[Clang 11.0.0 (clang-1100.0.33.17)] on darwin
Type "help", "copyright", "credits" or "license" for more information.

So it's ok.

We need to compile hdf5-parallel using mpi and gcc-10 because we need to support openmp + opemnpi in smilei (this can't be done with default apple gcc)

[weipengyao]

Thanks for the explanation. Now the code compiles successfully and runs smoothly.

Feels so good!

Maybe there needs some notation about this conda-issue in the doc online.

Many thanks! I will close this issue now.