search

lczech / gappa

A toolkit for analyzing and visualizing phylogenetic (placement) data
GNU General Public License v3.0
56 stars 7 forks source link

Compiling with OpenMP (omp.h) on Mac OS X #15

Closed lczech closed 3 years ago

lczech commented 3 years ago

The following issue was sent to me via email. I am posting it here for future users that might run into the same issue.

Compilation fails with AppleClang on MacOSX due to missing omp.h although OpenMP is reported as being used:

$ make
Running CMake...
CMake Deprecation Warning at CMakeLists.txt:26 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The CXX compiler identification is AppleClang 11.0.0.11000033
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Gappa build type: RELEASE
-- Static linking of system libraries: OFF
CMake Deprecation Warning at libs/genesis/CMakeLists.txt:28 (cmake_minimum_required):
  Compatibility with CMake < 2.8.12 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- Configuring Genesis
-- CMake version 3.19.3
-- The C compiler identification is AppleClang 11.0.0.11000033
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Genesis version: v0.24.0
-- Building Genesis as a dependency
-- Build type: RELEASE
-- Unity build: FULL
-- C++ compiler: AppleClang 11.0.0.11000033 at /Library/Developer/CommandLineTools/usr/bin/c++
-- C compiler  : AppleClang 11.0.0.11000033 at /Library/Developer/CommandLineTools/usr/bin/cc
-- Check if the system is big endian
-- Searching 16 bit integer
-- Looking for sys/types.h
-- Looking for sys/types.h - found
-- Looking for stdint.h
-- Looking for stdint.h - found
-- Looking for stddef.h
-- Looking for stddef.h - found
-- Check size of unsigned short
-- Check size of unsigned short - done
-- Searching 16 bit integer - Using unsigned short
-- Check if the system is big endian - little endian
-- Looking for zlib
-- Found ZLIB: /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib/libz.tbd (found version "1.2.11") 
-- Found zlib: /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/include 1.2.11
-- Using zlib
-- Looking for Threads
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE  
-- Found Threads: 
-- Using Threads
-- Looking for OpenMP
-- Using find_package( OpenMP ) patch
-- Try OpenMP C flag = [ ]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-fopenmp=libiomp5]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-fopenmp=libomp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-Xpreprocessor -fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-Xclang -fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [/openmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-Qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-openmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-xopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [+Oopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-qsmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP C flag = [-mp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [ ]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-fopenmp=libiomp5]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-fopenmp=libomp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-Xpreprocessor -fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-Xclang -fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [/openmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-Qopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-openmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-xopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [+Oopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-qsmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
-- Try OpenMP CXX flag = [-mp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Failed
CMake Warning (dev) at /usr/local/Cellar/cmake/3.19.3/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:426 (message):
  The package name passed to `find_package_handle_standard_args` (OpenMP)
  does not match the name of the calling package (OpenMP_patch).  This can
  lead to problems in calling code that expects `find_package` result
  variables (e.g., `_FOUND`) to follow a certain pattern.
Call Stack (most recent call first):
  libs/genesis/tools/cmake/FindOpenMP_patch.cmake:384 (find_package_handle_standard_args)
  libs/genesis/CMakeLists.txt:332 (find_package)
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Could NOT find OpenMP (missing: OpenMP_C_FLAGS OpenMP_CXX_FLAGS) 
-- Found OpenMP_C: -Xclang -fopenmp (found version "3.1") 
-- Found OpenMP_CXX: -Xclang -fopenmp (found version "3.1") 
-- Found OpenMP: TRUE (found version "3.1")  
-- Found OpenMP: -Xclang -fopenmp
-- Using OpenMP
-- Building static lib
-- Final build flags:
-- CMAKE_CXX_FLAGS         :  -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -pedantic -pedantic-errors -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -pedantic -pedantic-errors -Xclang -fopenmp 
-- CMAKE_CXX_FLAGS_RELEASE :  -O2 -DNDEBUG -O2 -DNDEBUG 
-- GENESIS_CXX_FLAGS       :  -Xclang -fopenmp 
-- GENESIS_EXE_LINKER_FLAGS:  
-- GENESIS_LINK_LIBRARIES  :  
-- Finished configuring Genesis
-- CMAKE_EXE_LINKER_FLAGS   
-- GENESIS_LINK_LIBRARIES /Library/Developer/CommandLineTools/SDKs/MacOSX10.15.sdk/usr/lib/libz.tbd;genesis_lib_static
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/username/gappa/build
Running make...
/Library/Developer/CommandLineTools/usr/bin/make -s -C build
Scanning dependencies of target genesis_lib_static
[  2%] Building CXX object libs/genesis/lib/genesis/CMakeFiles/genesis_lib_static.dir/__/__/__/__/genesis_unity_sources/lib/all.cpp.o
/Users/username/gappa/build/genesis_unity_sources/lib/all.cpp:172:13: fatal error: 'omp.h' file not found
#   include <omp.h>
            ^~~~~~~
1 error generated.
make[3]: *** [libs/genesis/lib/genesis/CMakeFiles/genesis_lib_static.dir/__/__/__/__/genesis_unity_sources/lib/all.cpp.o] Error 1
make[2]: *** [libs/genesis/lib/genesis/CMakeFiles/genesis_lib_static.dir/all] Error 2
make[1]: *** [all] Error 2
make: *** [build] Error 2
lczech commented 3 years ago

It is a known issue that AppleClang on MacOSX does not work well with OpenMP. As I currently do not have access to a Mac, I cannot test this, but I'll try to provide potential solutions:

  1. Install gappa via conda, instead of compiling on your own: https://anaconda.org/bioconda/gappa I don't think that this version uses OpenMP, so it will be slower for some commands - read on if you want/need speed.

  2. Instead of AppleClang, use "proper" clang:

    # Install the llvm package, which includes clang.
brew install llvm libomp

# We need to set custom paths here, so that the new Clang is used.
export PATH="$(brew --prefix llvm)/bin:$PATH";
export COMPILER=/usr/local/opt/llvm/bin/clang++
export CFLAGS="-I /usr/local/include -I/usr/local/opt/llvm/include"
export CXXFLAGS="-I /usr/local/include -I/usr/local/opt/llvm/include"
export LDFLAGS="-L /usr/local/lib -L/usr/local/opt/llvm/lib"
export CXX=${COMPILER}

# Bugfix for https://github.com/Homebrew/homebrew-core/issues/52461-mlinker-version=450
# Otherwise, we get "ld: unknown option: -platform_version"
# Not needed for all MacOS versions, apparently.
# See if that works for you, and otherwise, leave out these two lines.
export CXXFLAGS="${CXXFLAGS} -mlinker-version=450"
export LDFLAGS="${LDFLAGS} -mlinker-version=450"

# Now go to the genesis main directory, clean up, and build again.
cd wherever/you/have/stored/gappa
make clean
make

    I am not entirely sure that the extra libomp is needed in the package installation step, but it also doesn't hurt to install this (unless that package is not being found, in which case you can try to just brew install llvm instead). You might have to run an update on the machine first.

  3. Turn off OpenMP. This is not recommended, as this will result in slower execution times. But if your dataset is small enough to be analyzed on a Mac laptop, it is probably also small enough to not need OpenMP anyway.

    # We need to invoke CMake directly, telling it to not look for OpenMP.
# Go the the genesis main directory, and execute:
cd wherever/you/have/stored/gappa
make clean
mkdir build
cd build
cmake -DGENESIS_USE_OPENMP=OFF ..
make

    This will result in a warning, which you can ignore, since we deactivated OpenMP on purpuse.

Hope that helps! Let me know here about your progress!

PaulTillich commented 3 years ago

Thank you very much! The 2nd solution that you posted recommending the use of "proper" clang instead of AppleClang fixed the problem.

Though we have now run into an additional problem that may or may not be related. I am posting the code below in case it is related or in case you have any suggestions that could help. The program paprica is only creating half of the files that it is supposed to (which is an improvement from before when it didn't make any).

Thank you again for your time and help with this, we appreciate it very much!!!

Gabriels-MacBook-Air:paprica gabrielprice$ ./paprica-run.sh test bacteria
# cmalign :: align sequences to a CM 
# INFERNAL 1.1.1 (July 2014)
# Copyright (C) 2014 Howard Hughes Medical Institute.
# Freely distributed under the GNU General Public License (GPLv3).
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
# CM file:                                     /Users/gabrielprice/paprica/models/bacteria_ssu.cm
# sequence file:                               /Users/gabrielprice/paprica/test.bacteria.clean.unique.fasta
# CM name:                                     SSU_rRNA_bacteria
# saving alignment to file:                    /Users/gabrielprice/paprica/test.bacteria.clean.unique.align.sto
# output alignment format specified as:        Pfam
# output alignment alphabet:                   DNA
# number of worker threads:                    4
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
#                                                                              running time (s)                 
#                                                                       -------------------------------          
# idx  seq name      length  cm from    cm to  trunc    bit sc  avg pp  band calc  alignment      total  mem (Mb) 
# ---  ------------  ------  -------  -------  -----  --------  ------  ---------  ---------  ---------  --------
    1  SRR953432.50      84      968     1054     3'     39.93   0.974       0.03       0.04       0.07      2.91
    2  Firmicutes_1     252      524      776  5'&3'    231.71   0.997       0.08       0.02       0.10      5.77
    3  Firmicutes_2     253      524      776  5'&3'    238.32   1.000       0.08       0.02       0.10      5.83
#
# CPU time: 0.41u 0.05s 00:00:00.46 Elapsed: 00:00:00.32
# Saving alignment to file     /Users/gabrielprice/paprica/test.bacteria/test.bacteria.combined_16S.bacteria.tax.phylum_reps.clean.unique.align.sto     ... done
#
# CPU time: 0.00u 0.01s 00:00:00.01 Elapsed: 00:00:00.02
INFO Selected: Output dir: /Users/gabrielprice/paprica/test.bacteria/
INFO Selected: Query file: /Users/gabrielprice/paprica/test.bacteria/test.bacteria.clean.unique.align.newlength.fasta
INFO Selected: Tree file:     /Users/gabrielprice/paprica/ref_genome_database/bacteria/phylum_reps/combined_16S.23S.bacteria.tax.phylum_reps.f.   inal.bestTree
INFO Selected: Reference MSA:     /Users/gabrielprice/paprica/test.bacteria/combined_16S.bacteria.tax.phylum_reps.clean.align.newlength.fasta
INFO Selected: Automatic switching of use of per rate scalers
INFO Selected: Preserving the root of the input tree
INFO Selected: Specified model file:     /Users/gabrielprice/paprica/ref_genome_database/bacteria/phylum_reps/combined_16S.23S.bacteria.tax.phylum_reps.f.   inal.bestModel
INFO    Rate heterogeneity: GAMMA (4 cats, mean),  alpha: 0.469537 (user),  weights&rates: (0.25,0.0278902)     (0.25,0.231564) (0.25,0.796771) (0.25,2.94377) 
        Base frequencies (user): 0.220039 0.236043 0.303232 0.240686 
        Substitution rates (user): 0.893602 2.60232 1.46235 0.917758 3.56951 1
INFO     ______ ____   ___           _   __ ______
        / ____// __ \ /   |         / | / // ____/
       / __/  / /_/ // /| | ______ /  |/ // / __  
      / /___ / ____// ___ |/_____// /|  // /_/ /  
     /_____//_/    /_/  |_|      /_/ |_/ \____/ (v0.3.6)
INFO Output file: /Users/gabrielprice/paprica/test.bacteria/epa_result.jplace
INFO 3 Sequences done!
INFO Time spent placing: 0s
INFO Elapsed Time: 0s
                                              ....      ....  
                                             '' '||.   .||'   
                                                  ||  ||      
                                                  '|.|'       
     ...'   ....   ... ...  ... ...   ....        .|'|.       
     |  ||  '' .||   ||'  ||  ||'  || '' .||      .|'  ||      
     |''   .|' ||   ||    |  ||    | .|' ||     .|'|.  ||     
    '....  '|..'|'. ||...'   ||...'  '|..'|.    '||'    ||:.  
    '....'          ||       ||                               
                   ''''     ''''   v0.6.1 (c) 2017-2020
                                   by Lucas Czech and Pierre Barbera

Invocation:                        gappa examine edpl --allow-file-overwriting --out-dir     /Users/gabrielprice/paprica/test.bacteria/ --file-prefix test.bacteria.phylum_reps.edpl --jplace-path
                                   /Users/gabrielprice/paprica/test.bacteria/test.bacteria.phylum_reps.jplace
Command:                           gappa examine edpl

Input:
  --jplace-path                    /Users/gabrielprice/paprica/test.bacteria/test.bacteria.phylum_reps.jplace

Settings:
  --histogram-bins                 25
  --histogram-max                  -1
  --no-list-file                   false

 Output:
   --out-dir                        /Users/gabrielprice/paprica/test.bacteria/
   --file-prefix                    test.bacteria.phylum_reps.edpl
  --file-suffix

 Global Options:
   --allow-file-overwriting         true
   --verbose                        false
   --threads                        2
  --log-file

Run the following command to get the references that need to be cited:
`gappa tools citation Czech2020-genesis-and-gappa Matsen2011-edgepca-and-squash-clustering`

Started 2021-01-28 21:31:54

Found 1 jplace file
Writing output files.

Finished 2021-01-28 21:31:54
The following arguments were not expected: test.bacteria.phylum_reps --tree-file-prefix
Run with --help for more information.
Traceback (most recent call last):
  File "/Users/gabrielprice/paprica/paprica-place_it.py", line 1340, in <module>
    edpl = pd.read_csv(temp_dir + query + '.' + phylum_ref + '.edpllist.csv', index_col = 1)
  File "/Library/Python/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 685, in parser_f
    return _read(filepath_or_buffer, kwds)
  File "/Library/Python/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 457, in _read
    parser = TextFileReader(fp_or_buf, **kwds)
  File "/Library/Python/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 895, in __init__
    self._make_engine(self.engine)
  File "/Library/Python/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1135, in _make_engine
    self._engine = CParserWrapper(self.f, **self.options)
  File "/Library/Python/miniconda3/lib/python3.7/site-packages/pandas/io/parsers.py", line 1906, in __init__
    self._reader = parsers.TextReader(src, **kwds)
  File "pandas/_libs/parsers.pyx", line 380, in pandas._libs.parsers.TextReader.__cinit__
  File "pandas/_libs/parsers.pyx", line 687, in pandas._libs.parsers.TextReader._setup_parser_source
 FileNotFoundError: [Errno 2] File b'/Users/gabrielprice/paprica/test.bacteria/test.bacteria.phylum_reps.edpllist.csv'     does not exist: b'/Users/gabrielprice/paprica/test.bacteria/test.bacteria.phylum_reps.edpllist.csv'
lczech commented 3 years ago

Thanks for the feedback, I'm glad that the preferred solution that uses OpenMP worked!

As for paprica: I have never heard of that tool, interesting. As the error seems to be on their end, this is an issue that they need to fix. Please open up another issue at their issue page and report this there! You can link to this issue here, in case that helps them figuring out what is going on.

Here are a few hints though that you can forward to them: In your error message above, there is the message

The following arguments were not expected: test.bacteria.phylum_reps --tree-file-prefix

The function that is being called there, gappa examine heat-tree, recently changed its command line slightly (see here for the current usage), and went from the option --tree-file-prefix in gappa v0.6.1 (which you are using, so it should work) to just --file-prefix in gappa v0.7.0. If you or paprica update to that gappa version, this will need to be changed.

However, this does not seem to be connected to the actual error FileNotFoundError (last line in your log). That simply looks like some file paths are wrongly set by paprica. Ask them, and I hope they can help!

I'll close this issue for now, but feel free to comment or re-open as needed.