search

Argonne-National-Laboratory / DSP

An open-source parallel optimization solver for structured mixed-integer programming
Other
81 stars 24 forks source link

issue when running make during installation #278

Closed ChenxiH closed 3 months ago

ChenxiH commented 3 months ago

Hi! I am trying to install DSP on linux after running cmake.. I run make but got the following error:

$ make Scanning dependencies of target CoinUtils [ 1%] Creating directories for 'CoinUtils' [ 2%] No download step for 'CoinUtils' [ 3%] No update step for 'CoinUtils' [ 4%] No patch step for 'CoinUtils' [ 5%] Performing configure step for 'CoinUtils' checking build system type... x86_64-unknown-linux-gnu checking whether we want to compile in debug mode... no checking for C compiler default output file name... configure: error: C compiler cannot create executables See `config.log' for more details. make[2]: [extra/CMakeFiles/CoinUtils.dir/build.make:125: extra/CoinUtils-prefix/src/CoinUtils-stamp/CoinUtils-configure] Error 77 make[1]: [CMakeFiles/Makefile2:275: extra/CMakeFiles/CoinUtils.dir/all] Error 2 make: *** [Makefile:160: all] Error 2

I also try it on mac but got the same error, can you help me with it?

kibaekkim commented 3 months ago

Based on the log you shared, I suspect that CoinUtils is not available when you run make.

Did you follow the command line(s) described at the documentation (https://dsp.readthedocs.io/en/stable/install/)? In particular, you need to use the following to clone all the necessary sub-repositories:

git clone --recursive https://github.com/Argonne-National-Laboratory/DSP.git

If you still have issues, please share the detailed steps of your build.

ChenxiH commented 3 months ago

yes I just follow the documentation. But after cmake .., there is no /lib folder. I also tried "make" command but just got an error:

cmake ..
-- The C compiler identification is GNU 8.5.0
-- The CXX compiler identification is GNU 8.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/cc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The Fortran compiler identification is GNU 8.5.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /usr/bin/gfortran - skipped
-- Checking whether /usr/bin/gfortran supports Fortran 90
-- Checking whether /usr/bin/gfortran supports Fortran 90 - yes
-- The system information: Linux-4.18.0-425.3.1.el8.x86_64.x86_64
-- Build type: Release
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - not found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - found
-- Found BLAS: /usr/lib64/libblas.so
-- Looking for Fortran cheev
-- Looking for Fortran cheev - not found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- Found LAPACK: /usr/lib64/liblapack.so;/usr/lib64/libblas.so
-- Found ZLIB: /usr/lib64/libz.so (found version "1.2.11")
-- Found BZip2: /usr/lib64/libbz2.so (found version "1.0.6")
-- Looking for BZ2_bzCompressInit
-- Looking for BZ2_bzCompressInit - found
-- Found MPI_C: /share1/anaconda/3.11/lib/libmpi.so (found version "4.0")
-- Found MPI_CXX: /share1/anaconda/3.11/lib/libmpicxx.so (found version "4.0")
-- Could NOT find MPI_Fortran (missing: MPI_Fortran_WORKS)
-- Could NOT find MPI (missing: MPI_Fortran_FOUND) (found version "4.0")
-- Found /home/usrx/gurobi912/linux64/lib/libgurobi.so.9.1.2
-- Could not find ma27 (disabled OOQP solver)
-- SCIP is required for Benders decomposition. Benders decomposition is disabled with SCIP library.
-- Code coverage: OFF
-- Configuring done
CMake Warning at src/CMakeLists.txt:219 (add_library):
  Cannot generate a safe runtime search path for target Dsp because files in
  some directories may conflict with libraries in implicit directories:

    runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
      /share1/anaconda/3.11/lib

  Some of these libraries may not be found correctly.

CMake Warning at src/CMakeLists.txt:222 (add_executable):
  Cannot generate a safe runtime search path for target runDsp because files
  in some directories may conflict with libraries in implicit directories:

    runtime library [libz.so.1] in /usr/lib64 may be hidden by files in:
      /share1/anaconda/3.11/lib

  Some of these libraries may not be found correctly.

-- Generating done
-- Build files have been written to: /home/usrx/DSP/build

Then I run make and got the error

$ make
Scanning dependencies of target CoinUtils
[  1%] Creating directories for 'CoinUtils'
[  2%] No download step for 'CoinUtils'
[  3%] No update step for 'CoinUtils'
[  4%] No patch step for 'CoinUtils'
[  5%] Performing configure step for 'CoinUtils'
checking build system type... x86_64-unknown-linux-gnu
checking whether we want to compile in debug mode... no
checking for C compiler default output file name... configure: error: C compiler cannot create executables
See `config.log' for more details.
make[2]: *** [extra/CMakeFiles/CoinUtils.dir/build.make:125: extra/CoinUtils-prefix/src/CoinUtils-stamp/CoinUtils-configure] Error 77
make[1]: *** [CMakeFiles/Makefile2:275: extra/CMakeFiles/CoinUtils.dir/all] Error 2
make: *** [Makefile:160: all] Error 2

The config.log shows that

configure: failed program was:
| /* confdefs.h.  */
|
| #define PACKAGE_NAME "CoinUtils"
| #define PACKAGE_TARNAME "coinutils"
| #define PACKAGE_VERSION "2.11"
| #define PACKAGE_STRING "CoinUtils 2.11"
| #define PACKAGE_BUGREPORT "coinutils@lists.coin-or.org"
| /* end confdefs.h.  */
|
| int
| main ()
| {
|
|   ;
|   return 0;
| }
configure:2104: error: C compiler cannot create executables
kibaekkim commented 3 months ago

Your cmake looks OK. There are warnings, but the error message does not seem related to the warning.

Did you start from the scratch? That is, remove the old clone and follow the instructions in the documentation?

The error message indicates that your C compiler (GNU 8.5.0) cannot compile the simple test code. Can you test your C compiler with a simple test file?

Can you also run make VERBOSE=1 instead of make and share the output?

ChenxiH commented 3 months ago

I have tried to start from the scratch several times but the error still occurs. I have tested my C compiler and it seems it works well. I have also tried on a Mac but got similar issue: cmake:

The C compiler identification is AppleClang 14.0.3.14030022
The CXX compiler identification is AppleClang 14.0.3.14030022
Detecting C compiler ABI info
Detecting C compiler ABI info - done
Check for working C compiler: /Library/Developer/CommandLineTools/usr/bin/cc - skipped
Detecting C compile features
Detecting C compile features - done
Detecting CXX compiler ABI info
Detecting CXX compiler ABI info - done
Check for working CXX compiler: /Library/Developer/CommandLineTools/usr/bin/c++ - skipped
Detecting CXX compile features
Detecting CXX compile features - done
The Fortran compiler identification is GNU 14.1.0
Checking whether Fortran compiler has -isysroot
Checking whether Fortran compiler has -isysroot - yes
Checking whether Fortran compiler supports OSX deployment target flag
Checking whether Fortran compiler supports OSX deployment target flag - yes
Detecting Fortran compiler ABI info
Detecting Fortran compiler ABI info - done
Check for working Fortran compiler: /opt/homebrew/bin/gfortran - skipped
The system information: Darwin-22.6.0.arm64
Build type: Release
Found Library/gurobi1102/macos_universal2/lib
Could not find ma27 (disabled OOQP solver)
SCIP is required for Benders decomposition. Benders decomposition is disabled with SCIP library.
Code coverage: OFF
Configuring done (6.5s)
Generating done (0.1s)

but when running make: % make VERBOSE=1

/Applications/CMake.app/Contents/bin/cmake -S/Users/Work/DSP -B/Users/Work/DSP/build --check-build-system CMakeFiles/Makefile.cmake 0
/Applications/CMake.app/Contents/bin/cmake -E cmake_progress_start /Users/Work/DSP/build/CMakeFiles /Users/Work/DSP/build//CMakeFiles/progress.marks
/Library/Developer/CommandLineTools/usr/bin/make  -f CMakeFiles/Makefile2 all
/Library/Developer/CommandLineTools/usr/bin/make  -f extra/CMakeFiles/CoinUtils.dir/build.make extra/CMakeFiles/CoinUtils.dir/depend
cd /Users/Work/DSP/build && /Applications/CMake.app/Contents/bin/cmake -E cmake_depends "Unix Makefiles" /Users/Work/DSP /Users/Work/DSP/extra /Users/Work/DSP/build /Users/Work/DSP/build/extra /Users/Work/DSP/build/extra/CMakeFiles/CoinUtils.dir/DependInfo.cmake "--color="
/Library/Developer/CommandLineTools/usr/bin/make  -f extra/CMakeFiles/CoinUtils.dir/build.make extra/CMakeFiles/CoinUtils.dir/build
[  1%] Performing configure step for 'CoinUtils'
cd /Users/Work/DSP/build/extra/CoinUtils-prefix/src/CoinUtils-build && /Users/Work/DSP/extra/CoinUtils/configure --prefix=/Users/Work/DSP/build --enable-static --disable-shared CC=/Library/Developer/CommandLineTools/usr/bin/cc CXX=/Library/Developer/CommandLineTools/usr/bin/c++ "ADD_CXXFLAGS=-fPIC -Wno-sign-conversion -Wno-float-conversion -Wno-unused-variable"
checking build system type... arm-apple-darwin22.6.0
checking whether we want to compile in debug mode... no
checking for C compiler default output file name... configure: error: C compiler cannot create executables
See `config.log' for more details.
make[2]: *** [extra/CoinUtils-prefix/src/CoinUtils-stamp/CoinUtils-configure] Error 77
make[1]: *** [extra/CMakeFiles/CoinUtils.dir/all] Error 2
make: *** [all] Error 2

I also tried to run "./configure" and "make" under CoinUtils directory but they can both process successfully.

kibaekkim commented 3 months ago

I could not reproduce the error on linux and mac. Can you confirm that you are cloning release branch? Here is the log from the cloning:

▶ git clone --recursive https://github.com/Argonne-National-Laboratory/DSP.git DSP-temp
Cloning into 'DSP-temp'...
remote: Enumerating objects: 14501, done.
remote: Counting objects: 100% (2784/2784), done.
remote: Compressing objects: 100% (794/794), done.
remote: Total 14501 (delta 2059), reused 2679 (delta 1986), pack-reused 11717
Receiving objects: 100% (14501/14501), 65.59 MiB | 6.33 MiB/s, done.
Resolving deltas: 100% (10608/10608), done.
Updating files: 100% (521/521), done.
Submodule 'extra/Alps' (https://github.com/kibaekkim/CHiPPS-ALPS.git) registered for path 'extra/Alps'
Submodule 'extra/Clp' (https://github.com/coin-or/Clp.git) registered for path 'extra/Clp'
Submodule 'extra/CoinUtils' (https://github.com/coin-or/CoinUtils.git) registered for path 'extra/CoinUtils'
Submodule 'extra/OOQP' (https://github.com/emgertz/OOQP.git) registered for path 'extra/OOQP'
Submodule 'extra/Osi' (https://github.com/kibaekkim/Osi.git) registered for path 'extra/Osi'
Submodule 'extra/Smi' (https://github.com/kibaekkim/Smi.git) registered for path 'extra/Smi'
Cloning into '/home/kimk/DSP-temp/extra/Alps'...
remote: Enumerating objects: 5562, done.
remote: Counting objects: 100% (2042/2042), done.
remote: Compressing objects: 100% (62/62), done.
remote: Total 5562 (delta 2012), reused 1980 (delta 1980), pack-reused 3520
Receiving objects: 100% (5562/5562), 1.71 MiB | 5.61 MiB/s, done.
Resolving deltas: 100% (4330/4330), done.
Cloning into '/home/kimk/DSP-temp/extra/Clp'...
remote: Enumerating objects: 25593, done.
remote: Counting objects: 100% (6080/6080), done.
remote: Compressing objects: 100% (1566/1566), done.
remote: Total 25593 (delta 3781), reused 5913 (delta 3687), pack-reused 19513
Receiving objects: 100% (25593/25593), 73.65 MiB | 6.99 MiB/s, done.
Resolving deltas: 100% (18677/18677), done.
Cloning into '/home/kimk/DSP-temp/extra/CoinUtils'...
remote: Enumerating objects: 25018, done.
remote: Counting objects: 100% (2350/2350), done.
remote: Compressing objects: 100% (447/447), done.
remote: Total 25018 (delta 2007), reused 2123 (delta 1885), pack-reused 22668
Receiving objects: 100% (25018/25018), 56.87 MiB | 6.87 MiB/s, done.
Resolving deltas: 100% (18149/18149), done.
Cloning into '/home/kimk/DSP-temp/extra/OOQP'...
remote: Enumerating objects: 1912, done.
remote: Total 1912 (delta 0), reused 0 (delta 0), pack-reused 1912
Receiving objects: 100% (1912/1912), 1.21 MiB | 5.26 MiB/s, done.
Resolving deltas: 100% (1086/1086), done.
Cloning into '/home/kimk/DSP-temp/extra/Osi'...
remote: Enumerating objects: 15277, done.
remote: Total 15277 (delta 0), reused 0 (delta 0), pack-reused 15277
Receiving objects: 100% (15277/15277), 5.86 MiB | 6.14 MiB/s, done.
Resolving deltas: 100% (11675/11675), done.
Cloning into '/home/kimk/DSP-temp/extra/Smi'...
remote: Enumerating objects: 2775, done.
remote: Counting objects: 100% (64/64), done.
remote: Compressing objects: 100% (25/25), done.
remote: Total 2775 (delta 43), reused 56 (delta 39), pack-reused 2711
Receiving objects: 100% (2775/2775), 1.03 MiB | 5.71 MiB/s, done.
Resolving deltas: 100% (1916/1916), done.
Submodule path 'extra/Alps': checked out 'd635ae3e2a044ce32eca8902cc0e38644a3db733'
Submodule path 'extra/Clp': checked out '134c3360ce71a54ed8ce9fdc68d3393f0897e383'
Submodule path 'extra/CoinUtils': checked out '731071b10b73e3a07983bb14c33f7fa33c6e447e'
Submodule path 'extra/OOQP': checked out '460ba0eb849eadaa29c4d4d88a9ce3554f3ce165'
Submodule path 'extra/Osi': checked out '1cc7dfc66ec60c60dc6ab8761d9aa99d90d7bca0'
Submodule path 'extra/Smi': checked out '771badf2defafe0f751c293e504e6b8b198843af'

Then, cmake gives this output:

▶ cmake ..
-- The C compiler identification is GNU 8.5.0
-- The CXX compiler identification is GNU 8.5.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.2.0/gcc-8.5.0-zpsjiod/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.2.0/gcc-8.5.0-zpsjiod/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- The Fortran compiler identification is GNU 8.5.0
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Check for working Fortran compiler: /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.2.0/gcc-8.5.0-zpsjiod/bin/gfortran - skipped
-- The system information: Linux-5.15.0-107-generic.x86_64
-- Build type: Release
-- Looking for Fortran sgemm
-- Looking for Fortran sgemm - not found
-- Looking for pthread.h
-- Looking for pthread.h - found
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Looking for Fortran dgemm
-- Looking for Fortran dgemm - found
-- Found BLAS: /usr/lib/x86_64-linux-gnu/libblas.so;/usr/lib/x86_64-linux-gnu/libf77blas.so;/usr/lib/x86_64-linux-gnu/libatlas.so
-- Looking for Fortran cheev
-- Looking for Fortran cheev - not found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - not found
-- Looking for Fortran cheev
-- Looking for Fortran cheev - found
-- Found LAPACK: /usr/lib/x86_64-linux-gnu/liblapack.so;/usr/lib/x86_64-linux-gnu/libblas.so;/usr/lib/x86_64-linux-gnu/libf77blas.so;/usr/lib/x86_64-linux-gnu/libatlas.so
-- Found ZLIB: /usr/lib/x86_64-linux-gnu/libz.so (found version "1.2.11")
-- Found BZip2: /usr/lib/x86_64-linux-gnu/libbz2.so (found version "1.0.8")
-- Looking for BZ2_bzCompressInit
-- Looking for BZ2_bzCompressInit - found
-- Found MPI_C: /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/openmpi-4.1.3-qrpnszy/lib/libmpi.so (found version "3.1")
-- Found MPI_CXX: /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/openmpi-4.1.3-qrpnszy/lib/libmpi.so (found version "3.1")
-- Found MPI_Fortran: /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.3.0/openmpi-4.1.3-qrpnszy/lib/libmpi_usempif08.so (found version "3.1")
-- Found MPI: TRUE (found version "3.1")
-- Found /home/kimk/REPOS/DSPThirdPartyLibs/lib/libscip.so
-- Found /nfs/gce/projects/ddo/software/ibm/ILOG/CPLEX_Studio201/cplex/lib/x86-64_linux/static_pic/libcplex.a
-- Found libgfortran: /nfs/gce/software/custom/linux-ubuntu22.04-x86_64/spack/opt/spack/linux-ubuntu22.04-x86_64/gcc-11.2.0/gcc-8.5.0-zpsjiod/lib64/libgfortran.so
-- Could not find ma27 (disabled OOQP solver)
-- Code coverage: OFF
-- Configuring done
-- Generating done
-- Build files have been written to: /home/kimk/DSP-temp/build

I was able to run make. Can you notice any difference?

ChenxiH commented 3 months ago

May I ask whether the /lib folder is created after running cmake, or does this folder only appear after a successful make install? I think there may be some problems with my environmental variable settings. I tried to modify the CMake configuration to specify make.exe manually, and CoinUtils was able to install successfully with the following command:

set(SH "/path/to/sh.exe")
set(MAKE "/path/to/make.exe")
ExternalProject_Add(CoinUtils
    SOURCE_DIR ${COINUTILS_DIR}
    CONFIGURE_COMMAND ${SH} -c ${COINUTILS_DIR}/configure
        --prefix=${DEPEND_DIR}
        --enable-static 
        --disable-shared
        CC=${CMAKE_C_COMPILER}
        CXX=${CMAKE_CXX_COMPILER}
        ADD_CXXFLAGS=${EXTRA_CXX_FLAGS}
    BUILD_COMMAND ${MAKE} -j1
    INSTALL_COMMAND ${MAKE} -j1 install)

However, the following dependencies still encounter errors because they cannot find CoinUtils, such like:

configure: error: Required package CoinUtils not available.
configure: error: /bin/sh './configure' failed for Osi

Anyway, thank you for your help! I will keep trying to figure out what's wrong with my computer.

kibaekkim commented 3 months ago

May I ask whether the /lib folder is created after running cmake, or does this folder only appear after a successful make install?

lib folder is not created after running cmake. It's created after make.

You seem to be using Windows this time. I have not tested this solver on Windows.

It's possible that the dependency was not identified if you compile the package separately not through the cmake script. If you choose one machine (ideally linux or macosx), I may be able to help better.

Thanks for your patience.

ChenxiH commented 3 months ago

You just reminded me that I had installed the CMake app on my Mac, which may have caused the dependency not to be identified when compiling the package. I reinstalled CMake using brew, and it finally worked! Thank you so much for your help !

By the way, when I test DSPopt, the following fault occurs:

optimize!: ExtensiveForm: Test Failed at /Users/cxh/.julia/packages/DSPopt/Ys5uR/test/dcap.jl:103
  Expression: status == MathOptInterface.OPTIMAL
   Evaluated: MathOptInterface.ALMOST_OPTIMAL == MathOptInterface.OPTIMAL

Stacktrace:
 [1] macro expansion
   @ /opt/homebrew/Cellar/julia/1.10.3/share/julia/stdlib/v1.10/Test/src/Test.jl:672 [inlined]
 [2] macro expansion
   @ ~/.julia/packages/DSPopt/Ys5uR/test/dcap.jl:103 [inlined]
 [3] macro expansion
   @ /opt/homebrew/Cellar/julia/1.10.3/share/julia/stdlib/v1.10/Test/src/Test.jl:1669 [inlined]
 [4] top-level scope
   @ ~/.julia/packages/DSPopt/Ys5uR/test/dcap.jl:1690
Test Summary:              | Pass  Fail  Total  Time
dcap                       |   11     1     12  1.7s
  Parent model             |    2            2  0.1s
  Child model 2            |    3            3  0.0s
  Child model 3            |    3            3  0.0s
  Child model 1            |    3            3  0.0s
  optimize!: ExtensiveForm |          1      1  0.2s
ERROR: LoadError: Some tests did not pass: 11 passed, 1 failed, 0 errored, 0 broken.
in expression starting at /Users/cxh/.julia/packages/DSPopt/Ys5uR/test/runtests.jl:64
ERROR: Package DSPopt errored during testing

I also ran "ctest" for DSP and passed all test. I am not sure if this related to the failure of the DSPopt test? It is still OK for me to use DSPopt?

I am using cplex 22.1.1 and scip 9.0.1 on MacOS

kibaekkim commented 3 months ago

Great that you were able to compile the code and run the test!

The test failure came from the extensive form, most likely due to the definition of optimality. Our solution says the solution is almost optimal vs. OPTIMAL. I would think you can safely ignore that. I may need to update the JuMP interface of DSPopt.