search

gptune / GPTune

Other
67 stars 19 forks source link

Install GPTune on mac os #4

Closed jaehoonkoo closed 2 years ago

jaehoonkoo commented 2 years ago

Hi @liuyangzhuan,

I am trying to install GPTune on Mac os (with sudo access).

I am using config_macbook.zsh by running zsh config_macbook.zsh.

I set options for my settings. However, I have several errors while installing. 

  Building wheel for ConfigSpace (pyproject.toml) ... error
  ERROR: Command errored out with exit status 1:
   command: /Users/jkoo/research/github/GPTune/env/bin/python3.9 /Users/jkoo/research/github/GPTune/env/lib/python3.9/site-packages/pip/_vendor/pep517/in_process/_in_process.py build_wheel /var/folders/yz/05_ymqts7pbcby1t854lhpq00000gn/T/tmpce92lbe2
       cwd: /private/var/folders/yz/05_ymqts7pbcby1t854lhpq00000gn/T/pip-install-_uns7rup/configspace_90f7f905f02f47f5a212823d6286fe5b
  Complete output (39 lines):
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.macosx-12-x86_64-3.9
  creating build/lib.macosx-12-x86_64-3.9/ConfigSpace
  copying ConfigSpace/__init__.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace
  copying ConfigSpace/__version__.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace
  copying ConfigSpace/exceptions.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace
  creating build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx
  copying ConfigSpace/nx/exception.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx
  copying ConfigSpace/nx/release.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx
  copying ConfigSpace/nx/__init__.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx
  creating build/lib.macosx-12-x86_64-3.9/ConfigSpace/read_and_write
  copying ConfigSpace/read_and_write/pcs_new.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/read_and_write
  copying ConfigSpace/read_and_write/__init__.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/read_and_write
  copying ConfigSpace/read_and_write/json.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/read_and_write
  copying ConfigSpace/read_and_write/pcs.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/read_and_write
  creating build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/classes
  copying ConfigSpace/nx/classes/graph.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/classes
  copying ConfigSpace/nx/classes/__init__.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/classes
  copying ConfigSpace/nx/classes/digraph.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/classes
  creating build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/algorithms
  copying ConfigSpace/nx/algorithms/dag.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/algorithms
  copying ConfigSpace/nx/algorithms/__init__.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/algorithms
  copying ConfigSpace/nx/algorithms/cycles.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/algorithms
  creating build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/algorithms/components
  copying ConfigSpace/nx/algorithms/components/strongly_connected.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/algorithms/components
  copying ConfigSpace/nx/algorithms/components/__init__.py -> build/lib.macosx-12-x86_64-3.9/ConfigSpace/nx/algorithms/components
  running build_ext
  creating build/temp.macosx-12-x86_64-3.9
  creating build/temp.macosx-12-x86_64-3.9/ConfigSpace
  clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -I/private/var/folders/yz/05_ymqts7pbcby1t854lhpq00000gn/T/pip-build-env-iv4shcnh/overlay/lib/python3.9/site-packages/numpy/core/include -c ConfigSpace/hyperparameters.c -o build/temp.macosx-12-x86_64-3.9/ConfigSpace/hyperparameters.o
  In file included from ConfigSpace/hyperparameters.c:6:
  /usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
  #include <stdio.h>
           ^~~~~~~~~
  1 error generated.
  error: command '/usr/local/opt/llvm/bin/clang' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for ConfigSpace
Failed to build ConfigSpace
ERROR: Could not build wheels for ConfigSpace, which is required to install pyproject.toml-based projects
checking for header 'dlfcn.h' ...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
_configtest.c:1:10: fatal error: 'dlfcn.h' file not found
#include <dlfcn.h>
         ^~~~~~~~~
1 error generated.
failure.
removing: _configtest.c _configtest.o
failure.
checking for library 'dl' ...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
clang -flat_namespace -undefined suppress _configtest.o -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -ldl -o _configtest
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/openmpi-4.0.1/lib'
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/examples/SuperLU_DIST/superlu_dist/parmetis-4.0.3/install/lib/'
ld: library not found for -ldl
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
failure.
removing: _configtest.c _configtest.o
checking for function 'dlopen' ...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
clang _configtest.o -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -ldl -o _configtest
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/openmpi-4.0.1/lib'
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/examples/SuperLU_DIST/superlu_dist/parmetis-4.0.3/install/lib/'
ld: library not found for -ldl
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
failure.
removing: _configtest.c _configtest.o
building 'mpi4py.dl' extension
creating build/temp.macosx-12-x86_64-3.9
creating build/temp.macosx-12-x86_64-3.9/src
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/dynload.c -o build/temp.macosx-12-x86_64-3.9/src/dynload.o
In file included from src/dynload.c:5:
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
warning: build_ext: command '/usr/local/opt/llvm/bin/clang' failed with exit code 1

warning: build_ext: building optional extension "mpi4py.dl" failed

checking for MPI compile and link ...
/usr/local/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
_configtest.c:1:10: fatal error: 'stdlib.h' file not found
#include <stdlib.h>
         ^~~~~~~~~~
1 error generated.
failure.
removing: _configtest.c _configtest.o
error: Cannot compile MPI programs. Check your configuration!!!
WARNING: The wheel package is not available.
running install
running build
running build_src
running build_py
running build_ext
error: '/Users/jkoo/research/github/GPTune/openmpi-4.0.1/bin/mpicc' not found
error: '/Users/jkoo/research/github/GPTune/openmpi-4.0.1/bin/mpicxx' not found
error: '/Users/jkoo/research/github/GPTune/openmpi-4.0.1/bin/mpif90' not found
MPI configuration: [mpi] from 'mpi.cfg'
MPI C compiler:    /usr/local/bin/mpicc
MPI C++ compiler:  /usr/local/bin/mpicxx
MPI F compiler:    /usr/local/bin/mpifort
MPI F90 compiler:  /usr/local/bin/mpif90
MPI F77 compiler:  /usr/local/bin/mpif77
checking for dlopen() availability ...
checking for header 'dlfcn.h' ...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
_configtest.c:1:10: fatal error: 'dlfcn.h' file not found
#include <dlfcn.h>
         ^~~~~~~~~
1 error generated.
failure.
removing: _configtest.c _configtest.o
failure.
checking for library 'dl' ...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
clang -flat_namespace -undefined suppress _configtest.o -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -ldl -o _configtest
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/openmpi-4.0.1/lib'
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/examples/SuperLU_DIST/superlu_dist/parmetis-4.0.3/install/lib/'
ld: library not found for -ldl
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
failure.
removing: _configtest.c _configtest.o
checking for function 'dlopen' ...
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
clang _configtest.o -L/usr/local/lib -L/usr/local/opt/openssl@1.1/lib -L/usr/local/opt/sqlite/lib -ldl -o _configtest
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/openmpi-4.0.1/lib'
ld: warning: directory not found for option '-L/Users/jkoo/research/github/GPTune/examples/SuperLU_DIST/superlu_dist/parmetis-4.0.3/install/lib/'
ld: library not found for -ldl
clang-13: error: linker command failed with exit code 1 (use -v to see invocation)
failure.
removing: _configtest.c _configtest.o
building 'mpi4py.dl' extension
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c src/dynload.c -o build/temp.macosx-12-x86_64-3.9/src/dynload.o
In file included from src/dynload.c:5:
/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9/Python.h:25:10: fatal error: 'stdio.h' file not found
#include <stdio.h>
         ^~~~~~~~~
1 error generated.
warning: build_ext: command '/usr/local/opt/llvm/bin/clang' failed with exit code 1

warning: build_ext: building optional extension "mpi4py.dl" failed

checking for MPI compile and link ...
/usr/local/bin/mpicc -Wno-unused-result -Wsign-compare -Wunreachable-code -fno-common -dynamic -DNDEBUG -g -fwrapv -O3 -Wall -I/usr/local/include -I/usr/local/opt/openssl@1.1/include -I/usr/local/opt/sqlite/include -I/Users/jkoo/research/github/GPTune/env/include -I/usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/include/python3.9 -c _configtest.c -o _configtest.o
_configtest.c:1:10: fatal error: 'stdlib.h' file not found
#include <stdlib.h>
         ^~~~~~~~~~
1 error generated.
failure.

Is there any tutorial to provide a clean version of installing gptune on macOS or linux? Any help will be appreciated.

liuyangzhuan commented 2 years ago

It seems to be that you somehow still uses Clang, but the script was meant to use GNU only. Can you attach your modified config_macbook.zsh?

jaehoonkoo commented 2 years ago

I am retying by installing from scratch. I am getting the error in 2.2.2 GPTune C code. I can see that AppleClang is picked for C compiler identification. Should it be changed?

conda create --name gptune python=3.7 conda activate gptune

git clone https://github.com/gptune/GPTune.git cd GPTune
export GPROOT=$PWD

export PYTHONWARNINGS=ignore 
export CCC=/usr/local/bin/mpicc
 export CCCPP=/usr/local/bin/mpic++ 
export FTN=/usr/local/bin/mpif90
 export RUN=/usr/local/bin/mpirun
 export BLAS_LIB=/usr/local/Cellar/openblas/0.3.19/lib/libblas.dylib
export LAPACK_LIB=/usr/local/Cellar/lapack/3.10.0/lib/liblapack.dylib export SCALAPACK_LIB=/usr/local/Cellar/scalapack/2.1.0_3/lib/libscalapack.dylib

pip install --upgrade --user -r requirements_mac.txt

cd $GPROOT
mkdir -p build
cd build
cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_CXX_COMPILER=$CCCPP -DCMAKE_C_COMPILER=$CCC -DCMAKE_Fortran_COMPILER=$FTN -DTPL_BLAS_LIBRARIES=$BLAS_LIB -DTPL_LAPACK_LIBRARIES=$LAPACK_LIB -DTPL_SCALAPACK_LIBRARIES=$SCALAPACK_LIB make


Then, I got the following errors.

(gptune) ➜ build (master) ✔ cmake .. -DBUILD_SHARED_LIBS=ON -DCMAKE_CXX_COMPILER=$CCCPP -DCMAKE_C_COMPILER=$CCC -DCMAKE_Fortran_COMPILER=$FTN -DTPL_BLAS_LIBRARIES=$BLAS_LIB -DTPL_LAPACK_LIBRARIES=$LAPACK_LIB -DTPL_SCALAPACK_LIBRARIES=$SCALAPACK_LIB -- The C compiler identification is AppleClang 12.0.0.12000032 -- The CXX compiler identification is AppleClang 12.0.0.12000032 -- The Fortran compiler identification is GNU 11.2.0 -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working C compiler: /usr/local/bin/mpicc - skipped -- Detecting C compile features -- Detecting C compile features - done -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Check for working CXX compiler: /usr/local/bin/mpic++ - skipped -- Detecting CXX compile features -- Detecting CXX compile features - done -- Checking whether Fortran compiler has -isysroot -- Checking whether Fortran compiler has -isysroot - yes -- Checking whether Fortran compiler supports OSX deployment target flag -- Checking whether Fortran compiler supports OSX deployment target flag - yes -- Detecting Fortran compiler ABI info -- Detecting Fortran compiler ABI info - done -- Check for working Fortran compiler: /usr/local/bin/mpif90 - skipped -- Include /Users/jkoo/research/github/GPTune/cmake/setup_external_macros.cmake -- gptuneclcm will be built as a shared library. -- Performing Test qoptmatmul -- Performing Test qoptmatmul - Failed -- Could NOT find OpenMP_C (missing: OpenMP_C_FLAGS OpenMP_C_LIB_NAMES) -- Found OpenMP_CXX: -Xclang -fopenmp (found version "4.5") -- Found OpenMP_Fortran: -fopenmp (found version "4.5") -- Could NOT find OpenMP (missing: OpenMP_C_FOUND) (found version "4.5") -- Using TPL_BLAS_LIBRARIES='/usr/local/Cellar/openblas/0.3.19/lib/libblas.dylib' -- Using TPL_LAPACK_LIBRARIES='/usr/local/Cellar/lapack/3.10.0/lib/liblapack.dylib' -- Using TPL_SCALAPACK_LIBRARIES='/usr/local/Cellar/scalapack/2.1.0_3/lib/libscalapack.dylib' -- Could NOT find MPI_C (missing: MPI_C_WORKS) -- Found MPI_CXX: /usr/local/bin/mpic++ (found version "3.1") -- Found MPI_Fortran: /usr/local/bin/mpif90 (found version "3.1") -- Could NOT find MPI (missing: MPI_C_FOUND) (found version "3.1") -- Found Python3: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 (found version "3.9.9") found components: Interpreter Development Development.Module Development.Embed -- Python3 interpreter: /usr/local/Frameworks/Python.framework/Versions/3.9/bin/python3.9 -- Python3 installation directory: /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages -- GPTune will be installed in the default location: /usr/local/opt/python@3.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages -- Configuring done -- Generating done -- Build files have been written to: /Users/jkoo/research/github/GPTune/build

(gptune) ➜ build (master) ✗ make [ 7%] Building C object CMakeFiles/_gptuneclcm.dir/gptuneclcm/lcm.c.o error: invalid argument '-std=c++11' not allowed with 'C' make[2]: [CMakeFiles/_gptuneclcm.dir/gptuneclcm/lcm.c.o] Error 1 make[1]: [CMakeFiles/_gptuneclcm.dir/all] Error 2 make: *** [all] Error 2

liuyangzhuan commented 2 years ago

Setting the following 
export CCC=/usr/local/bin/mpicc
 export CCCPP=/usr/local/bin/mpic++ 
export FTN=/usr/local/bin/mpif90
 will pick clang for c and c++ and gfortran for fortran. I think it's always more consistent to use GNU for all of them. And you do need to install openmpi for source using GNU compilers. I see you commented the installation of openmpi, which was causing failure of your two trials. More specifically, you don't have /Users/jkoo/research/github/GPTune/openmpi-4.0.1/bin and /Users/jkoo/research/github/GPTune/openmpi-4.0.1/lib installed correctly.

I would suggest fixing the openmpi build first. Indeed, building openmpi from source using gnu compilers is slow. But you only need to do it once. Once it's done, you can set MPIFromSource=0 at line 13, and then edit line 62-70 accordingly, and then run config_macbook.zsh.

jaehoonkoo commented 2 years ago

Thank you for your comments.

I was able to fix it on my macOS Monterey 12.1 (21C52), such as:

brew reinstall gcc
brew reinstall gfortran
add /Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/lib to PATH, LD_LIBRARY_PATH, LIBRARY_PATH, DYLD_LIBRARY_PATH