Closed Z10Frank closed 2 years ago
Hi @Z10Frank,
could it be the absence of the whitespace? Looks like GCC may accept this without the whitespace.
Perhaps you can modify your build system to include it?
Kind regards,
Thank you!
That's strange, compiling with gcc without Mercurium did not need the whitespace between the flag and the filename.
Anyway, adding the whitespace in the makefile the error message disappears and the *.d
files for the dependencies are generated.
That's great progress, but the *.o
files are still not generated, so something is still missing.
Maybe the compilation log can help? compile_smilei_Mercurium.txt
Hi @Z10Frank,
That's strange, compiling with gcc without Mercurium did not need the whitespace between the flag and the filename.
Yeah, probably GCC undertands this but Mercurium doesn't.
Maybe the compilation log can help? compile_smilei_Mercurium.txt
I don't see any obvious failure, do the commands printed in the log have actually executed?
Kind regards,
Sorry, my bad, it was a dry print for make. Also, some environment variables were missing for the Python part, which is now causing some errors like:
Compiling src/Diagnostic/DiagnosticRadiationSpectrum.cpp
/gpfs/users/massimof/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:30: error: syntax error, unexpected __attribute__, expecting ',' or '}'
Compilation failed for file 'src/Collisions/CollisionalFusionDD.cpp'
Normally it does not do that with gcc, maybe Mercurium has an interaction with the Python part?
Compiling src/Diagnostic/DiagnosticRadiationSpectrum.cpp /gpfs/users/massimof/.local/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:84:30: error: syntax error, unexpected __attribute__, expecting ',' or '}' Compilation failed for file 'src/Collisions/CollisionalFusionDD.cpp'
That is probably Mercurium choking with C++ headers. The support of C++ in Mercurium has always been a bit difficult.
Do you need to use Mercurium (you only need it if you use Fortran or some of the unsupported features described in [1]) or you can use llvm-ompss instead?
Find it in https://github.com/bsc-pm/llvm/releases Documentation: https://pm.bsc.es/ftp/ompss-2/doc/user-guide/llvm/index.html
Sorry for the detour with this.
[1] https://pm.bsc.es/ftp/ompss-2/doc/user-guide/llvm/index.html#unsupported-features
Hello, thank you, sorry for my late answer, I managed to compile llvm-ompss 2 from the release page:
tar -xvf llvm-github-release-2021.11.tar
cd llvm-github-release-2021.11
cd llvm
mkdir install
cd install
module load llvm/9.0.1/gcc-9.2.0
module load anaconda2/2019.10/gcc-9.2.0
module load openmpi/4.0.2/gcc-9.2.0
module load cmake/3.16.2/intel-19.0.3.199
export INSTALLATION_PREFIX_NANOS6=/gpfs/users/massimof/nanos6/install
export INSTALLATION_PREFIX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLATION_PREFIX -DLLVM_ENABLE_PROJECTS=clang -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$INSTALLATION_PREFIX_NANOS6
make
make install
Now, compiling the code with llvm (adding the flag --ompss-2
in the makefile and using SMILEICXX ?= mpic++
):
module load openmpi/4.0.2/gcc-9.2.0
module load hdf5/1.10.7/gcc-9.2.0-openmpi
export HDF5_ROOT_DIR=/gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/hdf5-1.10.7-j6enx3ruksp5mjo5ed6lhywrzozihamq
export I_MPI_CXX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++
export OMPI_CXX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++
export LD_LIBRARY_PATH=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/lib:$LD_LIBRARY_PATH
make -j 10
an error occurs:
/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++: error while loading shared libraries: libc++.so.1: cannot open shared object file: No such file or directory
In general I can't find the library libc++.so.1
in the installation folder of llvm, so even adding the lib
subfolder to the LD_LIBRARY_PATH
does not find it. Was there an error in my build of llvm from the last release present in https://github.com/bsc-pm/llvm/releases
?
Hi @Z10Frank
In general I can't find the library
libc++.so.1
in the installation folder of llvm, so even adding thelib
subfolder to theLD_LIBRARY_PATH
does not find it. Was there an error in my build of llvm from the last release present inhttps://github.com/bsc-pm/llvm/releases
?
This is strange. I am not sure why are you seeing this.
What is happening is that the clang++
binary that you have compiled is using the LLVM C++ library (libc++.so
). When compiling with g++
the GNU C++ library is used (libstdcxx.so
). So it looks like you have compiled LLVM with clang
, which is OK (AFAICT g++
can only use libstdcxx.so
).
Perhaps the clang
you're using to compile llvm-ompss has been configured to use libc++
by default (I think this is a possibility) but somehow the library is not found in your system.
However, I would expect you to be able to find those libraries in general when running a program compiled with the clang
you have used to compile llvm-ompss
. Otherwise a very simple C++ "hello world" should fail for you as well.
So to be fair I don't have a good answer to your problem. Can you look into your environment and verify that the compiler you used to compile llvm-ompss can be used to compile a C++ "hello world" that you can actually run?
Kind regards,
Thank you, indeed the Hello_World.cpp test with my environment creates the error:
/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++: error while loading shared libraries: libtinfow.so.6: cannot open shared object file: No such file or directory
Should I try to build llvm-ompss with gcc instead?
Hi @Z10Frank,
This is confusing. We're using one compiler to compile the llvm-ompss compiler. Forget about the latter (llvm-ompss compiler), you care about the first one.
You can check about the compiler used to compile llvm-ompss, opening the CMakeCache.txt
that you will find in the directory where you ran cmake
and there should be a CMAKE_CXX_COMPILER
option. This is the compiler you used to compile.
Can you try to use that compiler to compile a simple hello world?
// hello.cc
#include <iostream>
int main() { std::cout << "Hello C++\n"; }
$ <the-compiler-you-found-in-cmakecache> -o hello hello.cc
Now try to run hello
$ ./hello
Does it work for you?
Kind regards,
Thank you, in the cluster I am working on there is an installed version of LLVM (not llvm-ompss), which works perfectly when compiling Smilei, with this environment
module load anaconda2/2019.10/gcc-9.2.0
module load hdf5/1.10.7/gcc-9.2.0-openmpi
module load llvm/9.0.1/gcc-9.2.0
module load openmpi/4.0.2/gcc-9.2.0
export LANG=en_US.utf8
export LC_ALL=en_US.utf8
export I_MPI_CXX=clang++
export HDF5_ROOT_DIR=/gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/hdf5-1.10.7-j6enx3ruksp5mjo5ed6lhywrzozihamq
In the associated directory, /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib
(which is automatically appended to the LD_LIBRARY_PATH when loading the module) there is for example the file libc++abi.so.1
, which is absent from my build of llvm-ompss. A simple test for hello world in c++ and even a more complex one like compiling Smilei works perfectly with this installation present in the machine, so I suspect that something is misplaced in how I built llvm-ompss.
Thank you, in the cluster I am working on there is an installed version of LLVM (not llvm-ompss), which works perfectly when compiling Smilei, with this environment
module load anaconda2/2019.10/gcc-9.2.0 module load hdf5/1.10.7/gcc-9.2.0-openmpi module load llvm/9.0.1/gcc-9.2.0 module load openmpi/4.0.2/gcc-9.2.0 export LANG=en_US.utf8 export LC_ALL=en_US.utf8 In the associated directory, `/gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib` (which is automatically appended to the LD_LIBRARY_PATH when loading the module) there is for example the file `libc++abi.so.1 `, which is absent from my build of llvm-ompss.
Ok but this is not the problem you're having. The problem is that you cannot execute the compiler that you just have compiled. And you can't because the libc++.so
library is not found. And you should observe a similar problem if you compile a hello world with the compiler of your environment (not the compiler that you are compiler).
Can you verify that your clang compiler is operating as expected for a hello world such as the one I showed you above?
If it does work, ldd hello
on the binary will tell you where libc++.so
is found. And then run ldd
on the clang++
of llvm-ompss that is failing for you and let's try to investigate from here.
Kind regards,
Ok, thank you, so with the clang++ compiler of the machine that works (ldd hello
, where hello
is compiled with clang++ -o hello Hello_world.cpp
):
linux-vdso.so.1 => (0x00007ffdd6d72000)
libc++.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++.so.1 (0x00002b999b1bb000)
libc++abi.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++abi.so.1 (0x00002b999b298000)
libm.so.6 => /lib64/libm.so.6 (0x00002b999b3bc000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002b999b6be000)
libc.so.6 => /lib64/libc.so.6 (0x00002b999b8d4000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b999bca2000)
librt.so.1 => /lib64/librt.so.1 (0x00002b999bebe000)
/lib64/ld-linux-x86-64.so.2 (0x00002b999b198000)
I don't know if it is important, but the llvm on the machine was compiled with gcc-9.2.0.
Now, using ldd
on the clang++
created from my build of llvm-ompss (ldd /gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++
) I obtain:
linux-vdso.so.1 => (0x00007ffce5901000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002abd81306000)
librt.so.1 => /lib64/librt.so.1 (0x00002abd81522000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002abd8172a000)
libm.so.6 => /lib64/libm.so.6 (0x00002abd8192e000)
libz.so.1 => /lib64/libz.so.1 (0x00002abd81c30000)
libtinfow.so.6 => not found
libc++.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++.so.1 (0x00002abd81116000)
libc++abi.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/llvm-9.0.1-t4ctj2mm5onq5s2qrug5pvnzdx6tsfut/lib/libc++abi.so.1 (0x00002abd811f3000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00002abd81e46000)
libc.so.6 => /lib64/libc.so.6 (0x00002abd8205c000)
/lib64/ld-linux-x86-64.so.2 (0x00002abd810e2000)
Ok thanks!
libtinfow.so.6 => not found
this seems the culprit. Looks like you will need ncurses
? Perhaps you need to load a module in your environment.
Kind regards,
Thank you, I have installed ncurses
, now the command ldd /gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/bin/clang++
seems to show everything in place:
linux-vdso.so.1 => (0x00007ffd5212f000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00002b47fff95000)
librt.so.1 => /lib64/librt.so.1 (0x00002b48001b1000)
libdl.so.2 => /lib64/libdl.so.2 (0x00002b48003b9000)
libz.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-cascadelake/gcc-9.2.0/zlib-1.2.9-jthi6nubs2nrhi47wv7yexyau5v27cwu/lib/libz.so.1 (0x00002b48005bd000)
libtinfo.so.5 => /lib64/libtinfo.so.5 (0x00002b48007d4000)
libstdc++.so.6 => /gpfs/softs/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-11.2.0-mpv3i3uebzvnvz7wxn6ywysd5hftycj3/lib64/libstdc++.so.6 (0x00002b48009fe000)
libm.so.6 => /lib64/libm.so.6 (0x00002b4800e05000)
libgcc_s.so.1 => /gpfs/softs/spack/opt/spack/linux-centos7-haswell/gcc-4.8.5/gcc-11.2.0-mpv3i3uebzvnvz7wxn6ywysd5hftycj3/lib64/libgcc_s.so.1 (0x00002b4801107000)
libc.so.6 => /lib64/libc.so.6 (0x00002b4801320000)
/lib64/ld-linux-x86-64.so.2 (0x00002b47ffd71000)
However when building Smilei this error occurs: fatal error: 'omp.h' file not found
(I know that OpenMP will not be used when OmpSs-2 is activated, but the headers are still included). To avoid putting preprocessor directives before the include of 'omp.h', I wanted to build also OpenMP with the llvm-ompSs together with clang++
, so restarting the process:
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLATION_PREFIX -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$INSTALLATION_PREFIX_NANOS6
The make
command afterwards goes well, but the make install
command gives this error:
CMake Error at projects/openmp/runtime/src/cmake_install.cmake:47 (file):
file INSTALL cannot find
"/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/lib/libomp.so":
No such file or directory.
Call Stack (most recent call first):
projects/openmp/runtime/cmake_install.cmake:42 (include)
projects/openmp/cmake_install.cmake:42 (include)
projects/cmake_install.cmake:42 (include)
cmake_install.cmake:57 (include)
make: *** [install] Error 1
Hi @Z10Frank
I'm at loss here. I've never seen this error before and I'd expect your cmake line to work fine when building openmp
.
CMake Error at projects/openmp/runtime/src/cmake_install.cmake:47 (file): file INSTALL cannot find "/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install/lib/libomp.so": No such file or directory. Call Stack (most recent call first): projects/openmp/runtime/cmake_install.cmake:42 (include) projects/openmp/cmake_install.cmake:42 (include) projects/cmake_install.cmake:42 (include) cmake_install.cmake:57 (include) make: *** [install] Error 1
Kind regards,
Hello, thank you, is there any intermediate step missing? I mean, after downloading the llvm-ompss version, in theory can I directly compile clang and OpenMP like this?
tar -xvf llvm-github-release-2021.11.tar
cd llvm-github-release-2021.11
cd llvm
mkdir install
cd install
module load gcc/11.2.0/gcc-4.8.5
module load anaconda2/2019.10/gcc-9.2.0
module load cmake/3.16.2/intel-19.0.3.199
module load openmpi/4.0.2/gcc-9.2.0
module load zlib/1.2.9/gcc-9.2.0
export INSTALLATION_PREFIX_NANOS6=/gpfs/users/massimof/nanos6/install
export INSTALLATION_PREFIX=/gpfs/workdir/massimof/llvm-github-release-2021.11/llvm/install
cmake .. -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALLATION_PREFIX -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$INSTALLATION_PREFIX_NANOS6
make -j 10
make install
Specifically, is there something to build with LLVM before, to enable the subsequent LLVM compilation of clang and OpenMP? It is my first experience with LLVM, so any crucial step you may consider obvious is probably missing from my attempts.
Hi @Z10Frank,
I tried to reproduce your problem without success.
I used the following steps in case you want to try.
$ wget https://github.com/bsc-pm/llvm/archive/refs/tags/github-release-2021.11.tar.gz
$ tar xf github-release-2021.11.tar.gz
$ cd llvm-github-release-2021.11/
$ mkdir build
$ cd build/
$ make ../llvm -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=$INSTALL_PREFIX -DLLVM_ENABLE_PROJECTS="clang;openmp" -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON -DCLANG_DEFAULT_NANOS6_HOME=$NANOS6_INSTALL
$ make -j$(nproc)
$ make install
My environment is an Ubuntu 18.04 LTS and cmake is 3.19.3.
Thank you! Now I hope we are converging, the build of clang and OpenMP seems to arrive almost to the end. My error was to create a destination install
directory that was also the place where CMake was called to build clang and OpenMP.
Creating a build directory, different from the $INSTALL_PREFIX
directory like in your commands, made all smooth except for the make install
step:
Install the project...
-- Install configuration: "Release"
CMake Error at cmake_install.cmake:41 (file):
file cannot create directory: /lib/cmake/llvm. Maybe need administrative
privileges.
Since I have no administrative privileges in this cluster, is it possible to specify a different path for CMake to create the llvm directory in this last step?
Since I have no administrative privileges in this cluster, is it possible to specify a different path for CMake to create the llvm directory in this last step?
I didn't need administrative privileges earlier. cmake
should honour CMAKE_INSTALL_PREFIX
here.
Maybe something is still off in your environment?
Thank you! Fixing the environment the LLVM-OmpSs compiler with OpenMP is built successfully, now I'll try to use it to compile Smilei.
Hello, I am trying to compile the code Smilei (https://github.com/SmileiPIC/Smilei) with Mercurium to use OmpSs-2, but the generation of dependencies between the source files seems to have a problem.
Trying to imitate the compiling command in Issue #39 (
I_MPI_CXX=mcxx MPICH_CXX=mcxx OMPI_CXX=mcxx mpiicpc -O3 -std=c++11 -o test_OmpSs-2 --ompss-2 test_OpenMP_OmpSs-2.cpp
), for the moment I have modified the makefile to force some variables to the necessary values: makefile.txtSince the code has many source files, the makefile first creates dependency files
*.d
, then the sources are compiled into*.o
files used to compile the main file. However at the phase of dependency generation this error occurs for each source file:Without these flags the dependency files
*.d
and the*.o
files are not created in thebuild
directory as they should, so the compiled commands seem to have the correct flags and wrappers (except for the-MF
and-MT
flags), but do not generate the files needed by the main file and the executable is not created.An example of the resulting compiling commands:
With
mcxx --help
this message is given:Does it mean that these flags cannot be used? In this case, is there another way to compile a multi-source code easily with the
make
utility?