search

lczech / grenedalf

Toolkit for Population Genetic Statistics from Pool-Sequenced Samples, e.g., in Evolve and Resequence experiments
GNU General Public License v3.0
33 stars 2 forks source link

Open MP issue #1

Closed LennWhy closed 1 year ago

LennWhy commented 3 years ago

Hiya,

I am trying to install grenedalf to test the software on my pool seq data, but having a hard time installing it. I've encountered an issue related to OpenMP. Sorry, i am a bit of novice when it comes to compiling. But below is my error output:

####################### Running CMake... CMake Deprecation Warning at CMakeLists.txt:26 (cmake_minimum_required): Compatibility with CMake < 2.8.12 will be removed from a future version of CMake.

Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions.

-- The CXX compiler identification is AppleClang 12.0.5.12050022 -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- grenedalf build type: RELEASE -- Static linking of system libraries: OFF CMake Deprecation Warning at libs/genesis/CMakeLists.txt:28 (cmake_minimum_required): Compatibility with CMake < 2.8.12 will be removed from a future version of CMake.

Update the VERSION argument value or use a ... suffix to tell CMake that the project does not need compatibility with older versions.

-- Configuring Genesis -- CMake version 3.21.0-rc2 -- The C compiler identification is AppleClang 12.0.5.12050022 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped -- Detecting C compile features -- Detecting C compile features - done -- OS: Darwin 20.5.0 -- Genesis version: v0.25.0 -- Building Genesis as a dependency -- Build type: RELEASE -- Unity build: FULL -- C++ compiler: AppleClang 12.0.5.12050022 at /Library/Developer/CommandLineTools/usr/bin/c++ -- C compiler : AppleClang 12.0.5.12050022 at /Library/Developer/CommandLineTools/usr/bin/cc -- Looking for zlib -- Found ZLIB: /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/lib/libz.tbd (found version "1.2.11") -- Found zlib: /Library/Developer/CommandLineTools/SDKs/MacOSX11.3.sdk/usr/include 1.2.11 -- Using zlib -- Looking for Threads -- Looking for pthread.h -- Looking for pthread.h - found -- Performing Test CMAKE_HAVE_LIBC_PTHREAD -- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success -- Found Threads: TRUE
-- Found Threads: -- Using Threads -- Looking for OpenMP -- Using find_package( OpenMP ) patch -- Try OpenMP C flag = [ ] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-fopenmp=libiomp5] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-fopenmp=libomp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-Xpreprocessor -fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-Xclang -fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [/openmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-Qopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-openmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-xopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [+Oopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-qsmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP C flag = [-mp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [ ] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-fopenmp=libiomp5] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-fopenmp=libomp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-Xpreprocessor -fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-Xclang -fopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [/openmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-Qopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-openmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-xopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [+Oopenmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-qsmp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed -- Try OpenMP CXX flag = [-mp] -- Performing Test OpenMP_FLAG_DETECTED -- Performing Test OpenMP_FLAG_DETECTED - Failed CMake Warning (dev) at /Applications/CMake.app/Contents/share/cmake-3.21/Modules/FindPackageHandleStandardArgs.cmake:438 (message): The package name passed to find_package_handle_standard_args (OpenMP) does not match the name of the calling package (OpenMP_patch). This can lead to problems in calling code that expects find_package result variables (e.g., _FOUND) to follow a certain pattern. Call Stack (most recent call first): libs/genesis/tools/cmake/FindOpenMP_patch.cmake:384 (find_package_handle_standard_args) libs/genesis/CMakeLists.txt:336 (find_package) This warning is for project developers. Use -Wno-dev to suppress it.

-- Could NOT find OpenMP (missing: OpenMP_C_FLAGS OpenMP_CXX_FLAGS) -- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES) -- Could NOT find OpenMP_CXX (missing: OpenMP_CXX_FLAGS OpenMP_CXX_LIB_NAMES) -- Could NOT find OpenMP (missing: OpenMP_C_FOUND OpenMP_CXX_FOUND) -- OpenMP not found -- You are trying to compile with Clang and OpenMp. This does not seem to work right now. Try installing libiomp-dev first, or the equivalent for your system. -- To build without OpenMP support, call CMake with -DGENESIS_USE_OPENMP=OFF CMake Error at libs/genesis/CMakeLists.txt:389 (message): Required package OpenMP not found.

####################

Best, Lenny

lczech commented 3 years ago

Hi Lenny,

thanks for your interest in grenedalf, and thanks for being an early tester!

OpenMP is a programming tool for using multiple cores on a computer, so it is recommended to use OpenMP support for speed. However, issues with Mac and OpenMP are quite common, as for some reason, Apple decided to not officially have OpenMP support in their compiler - the C++ community does not really understand that decision, and so we have to find workarounds for this... Let's try to get this to work for you as well! Worst case, we can disable it, and your program will just be slightly slower, but probably still fast enough for most things! But let's see how far we get in fixing it :-)

First, please make sure that you have OpenMP installed:

brew install libomp

With that, try compiling again:

make clean
make

If that still does not work, can you please attach the file build/CMakeFiles/CMakeError.log and build/CMakeFiles/CMakeOutput.log from the grenedalf directory here, so that I can have a closer look at what is going on?!

Thanks and so long Lucas

LennWhy commented 3 years ago

Hi Lucas,

Many thanks for your quick reply! I've completed the steps you recommended. I think I've installed OpenMP, but the grenedalf command is still not working. Below, I've attached the two log files, as you requested.

Thanks in advance for your help!

Lenny

CMakeError.log CMakeOutput.log

lczech commented 3 years ago

Hi Lenny,

sorry for the late reply, super busy at the moment! So, just to make sure that I understand the current issue: The log files you sent unfortunately do not mention whether OpenMP is working now... Sorry for that - I thought they included information on that as well. Could you share the current general CMake output again, as you did in your initial post?

Also, I have been able to reproduce your issue in a virtual machine. I needed the following to get the basics to work:

brew install libomp xz

Was that the same for you? That is, did you get an error about liblzma as well? If so, installing xz should solve that. But even after that, I ran into more OpenMP issues (omp.h not found), which is probably where you are at right now. This is unfortunately due to the Apple ecosystem constantly changing their compiler tool chain, and you are using macOS Big Sur, one of the latest versions, so I have not had the chance to get everything to work on that... Thanks for your patience!

Once you have posted the CMake log I mentioned above, I can then confirm that we are seeing the same errors, and I can start fixing it. But I'll be travelling for the next week, so it might take a bit until I get to do this... Hope you are not too much in a rush.

Alternatively, for now, I suggest to deactivate OpenMP, so that you can at least start testing grenedalf. For this, in the grenedalf directory, run

make clean
mkdir build
cd build
cmake -DGENESIS_USE_OPENMP=OFF ..
make
cd ..

and then you should find the binary in bin. This worked for me in my macOS Big Sur virtual machine.

Hope that helps :-) So long Lucas

lczech commented 3 years ago

Hi @LennWhy , any news on this?

LennWhy commented 3 years ago

Hi Lucas,

Sorry for the late reply. I tried to follow your suggestions then, but I think it didn't work. Given that you were traveling at that time or on holiday, I didn't want to bother you then. Unfortunately, I can't remember and didn't record the issue. Then, I kind of put it on the back burner as I became busy with other things. I can try to run it again in the few days and let you know what the issue was. Is that okay with you?

Best, Lenny

lczech commented 3 years ago

Hi Lenny, no worries, just wanted to check in and see if I can help. If this is on your back burner, no need to take it off there. Let me know once you get back to this, and if then there are still issues remaining ;-) Lucas

poecile-palustris commented 2 years ago

Hi Lucas,

I would like to use gendalf without OpenMP, since it seems to be missing on the cluster I am using. But there were some issues( Could you help me, please? I'm new at this.

I launched:

git clone --recursive https://github.com/lczech/grenedalf.git
cd grenedalf
make

It produced:

-- Could NOT find OpenMP (missing:  OpenMP_CXX_FLAGS) 
-- OpenMP not found
-- To build without OpenMP support, call CMake with -DGENESIS_USE_OPENMP=OFF
CMake Error at libs/genesis/tools/cmake/IncludeOpenMP.cmake:90 (message):
  Required package OpenMP not found.
Call Stack (most recent call first):
  libs/genesis/CMakeLists.txt:280 (include)

-- Configuring incomplete, errors occurred!
See also "/home/elena-pt/grenedalf/build/CMakeFiles/CMakeOutput.log".
See also "/home/elena-pt/grenedalf/build/

Then:

make clean
mkdir build
cd build
cmake -DGENESIS_USE_OPENMP=OFF ..

It produced:

-- OpenMP was not found or has been deactivated. This results in a considerably slower runtime for the program. Try to use a different compiler. Alternatively, if you are sure your compiler supports OpenMP, you can add the needed compiler flags to the CMake script.
-- CMAKE_EXE_LINKER_FLAGS   
-- GENESIS_LINK_LIBRARIES /usr/lib64/libz.so;-lpthread;/home/elena-pt/grenedalf/build/libs/genesis/genesis-htslib/lib/libhts.a;z;/usr/lib64/liblzma.so;/usr/lib64/libbz2.so;genesis_lib_static
-- Configuring incomplete, errors occurred!
See also "/home/elena-pt/grenedalf/build/CMakeFiles/CMakeOutput.log".
See also "/home/elena-pt/grenedalf/build/CMakeFiles/CMakeError.log".

Then (in grenedalf/build): make It produced: make: *** No goals set and no makefile found. Stop. As far as I understand, this is due to the fact that there is no Makefile in the directory.

And if (in grenedalf)

cd ..
make

It gives:

Running make...
makes -C builder
make[1]: *** No goals set and no makefile found. Stop.
make: *** [build] Error 2

I will be very grateful for your help! Thank you in advance.

Best, Poecile

lczech commented 2 years ago

Hi @poecile-palustris,

thank you for your interest in grenedalf, and thanks for the very thorough report! Let's see that we get this solved ;-)

The output of your step cmake -DGENESIS_USE_OPENMP=OFF .. contains the line

-- Configuring incomplete, errors occurred!

which indicates that something else went wrong, but I cannot tell what. Could you please repeat your steps

make clean
mkdir build
cd build
cmake -DGENESIS_USE_OPENMP=OFF ..

but this time attach the whole output of that step?

Thanks and so long Lucas

PS: I'm currently working on some big improvements for grenedalf - stay tuned for more features, and maybe some more speed in the next few weeks ;-)

poecile-palustris commented 2 years ago

Lucas,

Thank you very much for your reply! The files are attached. I will be glad to use new features:) CMakeError.log CMakeOutput.log

Best, Poecile

lczech commented 2 years ago

Hi @poecile-palustris,

thanks, could you also please post the output that you see in your terminal as well?

Cheers Lucas

poecile-palustris commented 2 years ago

Hi Lucas,

Yes, of course, here is the output:

(base) [elena-pt@head02 build]$ cmake -DGENESIS_USE_OPENMP=OFF ..
CMake Error at CMakeLists.txt:46 (cmake_policy):
  Policy "CMP0025" is not known to this version of CMake.

-- The CXX compiler identification is GNU 4.4.7
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- grenedalf build type: RELEASE
-- Static linking of system libraries: OFF
-- Configuring Genesis
-- CMake version 2.8.12.2
-- The C compiler identification is GNU 4.4.7
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- OS: Linux 2.6.32-573.12.1.el6_lustre.x86_64
-- Genesis version: v0.25.0
-- Building Genesis as a dependency
-- Build type: RELEASE
-- Unity build: FULL
-- C++ compiler: GNU 4.4.7 at /usr/bin/c++
-- C compiler  : GNU 4.4.7 at /usr/bin/cc
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for zlib
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.3") 
-- Found zlib: /usr/include 1.2.3
-- Using zlib
-- Looking for Threads
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE  
-- Found Threads: -lpthread
-- Using Threads
-- Looking for htslib
-- Found Git: /usr/bin/git (found version "1.7.1") 
-- Looking for LibLZMA
-- Looking for lzma_auto_decoder in /usr/lib64/liblzma.so
-- Looking for lzma_auto_decoder in /usr/lib64/liblzma.so - found
-- Looking for lzma_easy_encoder in /usr/lib64/liblzma.so
-- Looking for lzma_easy_encoder in /usr/lib64/liblzma.so - found
-- Looking for lzma_lzma_preset in /usr/lib64/liblzma.so
-- Looking for lzma_lzma_preset in /usr/lib64/liblzma.so - found
-- Found LibLZMA: /usr/include (found version "4.999.9") 
-- Found LibLZMA: /usr/lib64/liblzma.so
-- Looking for BZip2
-- Found BZip2: /usr/lib64/libbz2.so (found version "1.0.5") 
-- Looking for BZ2_bzCompressInit in /usr/lib64/libbz2.so
-- Looking for BZ2_bzCompressInit in /usr/lib64/libbz2.so - found
-- Found BZip2: /usr/lib64/libbz2.so
-- HTSLIB_INCLUDE_DIR: /home/elena-pt/grenedalf/build/libs/genesis/genesis-htslib/include
-- HTSLIB_LINK_DIR:    /home/elena-pt/grenedalf/build/libs/genesis/genesis-htslib/lib
-- HTSLIB_LIBRARY:     /home/elena-pt/grenedalf/build/libs/genesis/genesis-htslib/lib/libhts.a
-- HTSLIB_LZMA:        /usr/lib64/liblzma.so
-- HTSLIB_BZ2:         /usr/lib64/libbz2.so
-- Found htslib: /home/elena-pt/grenedalf/build/libs/genesis/genesis-htslib/lib/libhts.a
-- Using htslib
-- Building static lib
-- Final build flags:
-- CMAKE_CXX_FLAGS         :  -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -pedantic -pedantic-errors -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -pedantic -pedantic-errors 
-- CMAKE_CXX_FLAGS_RELEASE :  -O3 -DNDEBUG -O3 -DNDEBUG 
-- GENESIS_CXX_FLAGS       :  
-- GENESIS_EXE_LINKER_FLAGS:  
-- GENESIS_LINK_LIBRARIES  :  
-- Finished configuring Genesis
-- OpenMP was not found or has been deactivated. This results in a considerably slower runtime for the program. Try to use a different compiler. Alternatively, if you are sure your compiler supports OpenMP, you can add the needed compiler flags to the CMake script.
-- CMAKE_EXE_LINKER_FLAGS   
-- GENESIS_LINK_LIBRARIES /usr/lib64/libz.so;-lpthread;/home/elena-pt/grenedalf/build/libs/genesis/genesis-htslib/lib/libhts.a;z;/usr/lib64/liblzma.so;/usr/lib64/libbz2.so;genesis_lib_static
-- Configuring incomplete, errors occurred!
See also "/home/elena-pt/grenedalf/build/CMakeFiles/CMakeOutput.log".
See also "/home/elena-pt/grenedalf/build/CMakeFiles/CMakeError.log".

I checked my version of cmake, it meets the requirements (> 2.8.7):

(base) [elena-pt@head02 build]$ cmake --version
cmake version 2.8.12.2

Thanks for the help, Poecile

lczech commented 2 years ago

Ah interesting, thanks. I forgot a check for the CMake version in the script, which I will fix this in the upcoming version. For now, so that you can continue, you can simply delete the cmake_policy statement in the CMakeLists.txt file in the main directory of grenedalf. Let me know if that works ;-)

Also, have you checked in with your cluster admins whether OpenMP is really not available? I've worked on clusters where it needed to be activated as a module or part of some other module.

s21y commented 2 years ago

Hi Lucas, We also got the issue to compile with OpenMP. attache the log files and the error image. CMakeOutput.log

We have tried many cmake version, from 2.9.4 up tp 3.23.1. Our system is centos 7.

Thanks a lot for your help, Best, YS

error_image CMakeError.log

lczech commented 2 years ago

Hey @s21y,

thanks for reporting this! For now, you can disable OpenMP, as described in the above posts.

I did not expect that, and had assumed that it is widely available these days... On clusters, it might be hidden in some module related to building/compiling (like a GCC, Clang, CMake, build-essentials, or similar module). But if those are not available on your cluster, then I guess I'll have to implement a different solution other than OpenMP. I'm currently working on a solution for this, as OpenMP seems to be a recurring issue... Might take me a while though, sorry for that!

Cheers Lucas

lczech commented 1 year ago

Hey @LennWhy, @poecile-palustris, and @s21y,

took me long enough, but I've finally reworked all OpenMP-related code, and use my own threading implementation now. Hence, this issue should not appear any more.

The changes are on the master branch now, please get a fresh copy grenedalf from there, and compile again. Once I've played around with it and tested it a bit more, the fix will then be part of the next release version v0.2.0.

Hope this helps, and let me know if you have any further issues or feedback.

Best Lucas