search

trilinos / xSDKTrilinos

BSD 3-Clause "New" or "Revised" License
6 stars 6 forks source link

failure on Apple with latest xcode. #8

Open BarrySmith opened 7 years ago

BarrySmith commented 7 years ago 
/Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 preinstall
make[1]: Nothing to be done for `preinstall'.
Install the project...
/usr/local/Cellar/cmake/3.6.2/bin/cmake -P cmake_install.cmake
-- Install configuration: "DEBUG"
-- Installing: /Users/barrysmith/Src/petsc/arch-xsdk/include/xSDKTrilinos_version.h
-- Installing: /Users/barrysmith/Src/petsc/arch-xsdk/include/Makefile.export.xSDKTrilinos
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/include/Makefile.export.xSDKTrilinos
-- Installing: /Users/barrysmith/Src/petsc/arch-xsdk/lib/cmake/xSDKTrilinos/xSDKTrilinosConfigVersion.cmake
-- Installing: /Users/barrysmith/Src/petsc/arch-xsdk/lib/libxsdkpetsc.12.6.2.dylib
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/lib/libxsdkpetsc.12.dylib
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/lib/libxsdkpetsc.dylib
-- Installing: /Users/barrysmith/Src/petsc/arch-xsdk/include/xSDKTrilinos_config.hpp
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/include/BelosPETScSolMgr.hpp
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/include/Tpetra_PETScAIJGraph.hpp
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/include/Tpetra_PETScAIJMatrix.hpp
-- Installing: /Users/barrysmith/Src/petsc/arch-xsdk/lib/libxsdkhypre.12.6.2.dylib
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/lib/libxsdkhypre.12.dylib
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/lib/libxsdkhypre.dylib
-- Up-to-date: /Users/barrysmith/Src/petsc/arch-xsdk/include/Ifpack2_Hypre.hpp/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: liblast_lib.a(last_lib_dummy.c.o) has no symbols
/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: file: liblast_lib.a(last_lib_dummy.c.o) has no symbols
warning: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ranlib: warning for library: liblast_lib.a the table of contents is empty (no object file members in the library define global symbols)
In file included from /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/petsc/example/PETSc_Amesos2Ex.cpp:60:
In file included from /Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2.hpp:45:
In file included from /Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2_Factory.hpp:102:
In file included from /Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2_Superludist.hpp:47:
In file included from /Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2_Superludist_decl.hpp:58:
/Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2_Superludist_FunctionMap.hpp:285:17: warning: comparison of constant 67 with expression of type 'SLUD::DiagScale_t' is always false [-Wtautological-constant-out-of-range-compare]
      char eq = AMESOS2_SLUD_GET_EQUED(*equed);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2_Superludist_FunctionMap.hpp:71:98: note: expanded from macro 'AMESOS2_SLUD_GET_EQUED'
#define AMESOS2_SLUD_GET_EQUED(ds) (((ds)==SLUD::NOEQUIL) ? 'N' : ((ds)==SLUD::ROW) ? 'R' : ((ds)=='C') ? SLUD::COL : SLUD::BOTH)
                                                                                             ~~~~^ ~~~
/Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2_Superludist_FunctionMap.hpp:306:17: warning: comparison of constant 67 with expression of type 'SLUD::DiagScale_t' is always false [-Wtautological-constant-out-of-range-compare]
      char eq = AMESOS2_SLUD_GET_EQUED(*equed);
                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/barrysmith/Src/petsc/arch-xsdk/include/Amesos2_Superludist_FunctionMap.hpp:71:98: note: expanded from macro 'AMESOS2_SLUD_GET_EQUED'
#define AMESOS2_SLUD_GET_EQUED(ds) (((ds)==SLUD::NOEQUIL) ? 'N' : ((ds)==SLUD::ROW) ? 'R' : ((ds)=='C') ? SLUD::COL : SLUD::BOTH)
                                                                                             ~~~~^ ~~~
2 warnings generated.
error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/install_name_tool: for: /Users/barrysmith/Src/petsc/arch-xsdk/lib/libxsdkpetsc.12.6.2.dylib (for architecture x86_64) option "-add_rpath /Users/barrysmith/Src/petsc/arch-xsdk/lib" would duplicate path, file already has LC_RPATH for: /Users/barrysmith/Src/petsc/arch-xsdk/lib
********End of Output of running make on XSDKTRILINOS *******
                ===============================================================================
                    Testing xSDKTrilinos; this may take several minutes
                ===============================================================================

Executing: cd /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build && /usr/local/bin/ctest
stdout:
Test project /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build
      Start  1: xSDKTrilinos_PETScAIJMatrix
 1/10 Test  #1: xSDKTrilinos_PETScAIJMatrix ..........***Exception: Other  0.00 sec
      Start  2: xSDKTrilinos_PETSc_Amesos2_example
 2/10 Test  #2: xSDKTrilinos_PETSc_Amesos2_example ...***Exception: Other  0.00 sec
      Start  3: xSDKTrilinos_PETSc_Anasazi_example
 3/10 Test  #3: xSDKTrilinos_PETSc_Anasazi_example ...***Exception: Other  0.00 sec
      Start  4: xSDKTrilinos_PETSc_Ifpack2_example
 4/10 Test  #4: xSDKTrilinos_PETSc_Ifpack2_example ...***Exception: Other  0.00 sec
      Start  5: xSDKTrilinos_PETSc_MueLu_example
 5/10 Test  #5: xSDKTrilinos_PETSc_MueLu_example .....***Exception: Other  0.00 sec
      Start  6: xSDKTrilinos_example_TpetraKSP
 6/10 Test  #6: xSDKTrilinos_example_TpetraKSP .......***Exception: Other  0.00 sec
      Start  7: xSDKTrilinos_example_EpetraKSP
 7/10 Test  #7: xSDKTrilinos_example_EpetraKSP .......***Exception: Other  0.00 sec
      Start  8: xSDKTrilinos_HypreTest
 8/10 Test  #8: xSDKTrilinos_HypreTest ...............***Exception: Other  0.00 sec
      Start  9: xSDKTrilinos_Hypre_Belos_example
 9/10 Test  #9: xSDKTrilinos_Hypre_Belos_example .....***Exception: Other  0.00 sec
      Start 10: xSDKTrilinos_Hypre_Solve_example
10/10 Test #10: xSDKTrilinos_Hypre_Solve_example .....***Exception: Other  0.00 sec
0% tests passed, 10 tests failed out of 10
Label Time Summary:
xSDKTrilinos    =   0.02 sec (10 tests)
Total Test time (real) =   0.03 sec
The following tests FAILED:
      1 - xSDKTrilinos_PETScAIJMatrix (OTHER_FAULT)
      2 - xSDKTrilinos_PETSc_Amesos2_example (OTHER_FAULT)
      3 - xSDKTrilinos_PETSc_Anasazi_example (OTHER_FAULT)
      4 - xSDKTrilinos_PETSc_Ifpack2_example (OTHER_FAULT)
      5 - xSDKTrilinos_PETSc_MueLu_example (OTHER_FAULT)
      6 - xSDKTrilinos_example_TpetraKSP (OTHER_FAULT)
      7 - xSDKTrilinos_example_EpetraKSP (OTHER_FAULT)
      8 - xSDKTrilinos_HypreTest (OTHER_FAULT)
      9 - xSDKTrilinos_Hypre_Belos_example (OTHER_FAULT)
     10 - xSDKTrilinos_Hypre_Solve_example (OTHER_FAULT)
**** Configure header /var/folders/c1/ldz_dt8n2r3dtwv_chp5pfr40000gn/T/petsc-vz4kpx/confdefs.h ****

Looks like some Cmaky type thing might need to be updated in the install process

jwillenbring commented 7 years ago

@amklinv Do you have a machine that would let you reproduce this?

amklinv commented 7 years ago

I do not, unless you can think of one. If it's a CMake issue, should we notify @bartlettroscoe?

bartlettroscoe commented 7 years ago

@BarrySmith, can you attach the CTest log file? You should be able to find it at:

/Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/Testing/Temporary/LastTest.log

See https://trilinos.org/docs/files/TrilinosBuildReference.html#testing-with-ctest

BarrySmith commented 7 years ago

Sorry cannot attach because it does not accept .log files. Man that always drives me nuts. Having to rename the file is a royal pain.

Ross.txt

bartlettroscoe commented 7 years ago

Just rename it TestLog.out.txt

BarrySmith commented 7 years ago

I did upload it. You should see it in my previous text box named Ross.txt

bartlettroscoe commented 7 years ago

The attached CTest log file show that all of the tests are failing with the same error:

dyld: Library not loaded: /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/petsc/src/libxsdkpetsc.12.dylib
  Referenced from: /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/petsc/test/xSDKTrilinos_PETScAIJMatrix.exe
  Reason: no suitable image found.  Did find:
    /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/petsc/src/libxsdkpetsc.12.dylib: malformed mach-o: load commands size (81296) > 32768

That seems to suggest that a PETSc library is too big for the dynamic library load commands on thsi system? The only thing that I could find on this was:

I don't develop on Macs so I am not sure how these .dylib libraries work on that system. But those links suggest that it might be due to a particular version of Mac OS?

BarrySmith commented 7 years ago

Ross,

It is complaining about a xskdtrilinos library:

/libxsdkpetsc.12.dylib: malformed mach-o: load commands size (81296) > 32768

not a PETSc library. libxsdkpetsc.12.dylib is not part of PETSc. I think that the cmake code that it is building the xsdktrilinos libraries might not work with the latest Mac Xcode and thus produces a bad library.

Barry

On Oct 12, 2016, at 8:59 AM, Roscoe A. Bartlett notifications@github.com wrote:

The attached CTest log file show that all of the tests are failing with the same error:

dyld: Library not loaded: /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/petsc/src/libxsdkpetsc.12.dylib Referenced from: /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/petsc/test/xSDKTrilinos_PETScAIJMatrix.exe Reason: no suitable image found. Did find: /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/petsc/src/libxsdkpetsc.12.dylib: malformed mach-o: load commands size (81296) > 32768

That seems to suggest that a PETSc library is too big for the dynamic library load commands on thsi system? The only thing that I could find on this was:

• http://stackoverflow.com/questions/39646221/stack-gives-dyld-malformed-mach-o • commercialhaskell/stack#2577 • https://ghc.haskell.org/trac/ghc/ticket/12479 I don't develop on Macs so I am not sure how these .dylib libraries work on that system. But those links suggest that it might be due to a particular version of Mac OS?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

bartlettroscoe commented 7 years ago

Sandia now has a contract with Kitware. I have asked for some help since I don't have any experience with this platform myself personally.

Otherwise, what happens when you use the Makefile generator instead of the XCode generator on this Mac?

BarrySmith commented 7 years ago

On Oct 12, 2016, at 12:59 PM, Roscoe A. Bartlett notifications@github.com wrote:

Sandia now has a contract with Kitware. I have asked for some help since I don't have any experience with this platform myself personally.

Otherwise, what happens when you use the Makefile generator instead of the XCode generator on this Mac?

I don't know what you mean. I am just using the cmake commands in petsc-dir/config/BuildSystem/config/packages/xsdktrilinos.py to build xsdktrilinos. So I think it is just using the "Makefile generator". 

Apple makes life confusing because they refer to Xcode as both the GUI IDE and the compiler suite that. In my email I am only talking about the compiler suite not the IDE.

Barry

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

bradking commented 7 years ago

On macOS 10.12 Sierra the dynamic loader has a new/lower limit of some kind that in practice is triggered by a large number of entries (or large total length) of the runtime search path (RPATH). What is the output of otool -l libxsdkpetsc.12.dylib?

BarrySmith commented 7 years ago

Here it is attached. otool.txt

bradking commented 7 years ago

According to the otool output, the total size of all LC_LOAD_DYLIB entries is 11568 and the total size of all LC_RPATH entries is 67984. The former is for dylib dependencies and the latter is for the runtime search path to find them.

It appears there are many duplicate LC_RPATH entries and only 7 unique ones out of 828 entries. IIRC CMake is supposed to de-duplicate the rpath entries, so we need to figure out why so many entries are appearing. What version of CMake are you using?

Also under /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build/petsc/src/ there should be a file called CMakeFiles/xsdkpetsc.dir/link.txt or something like that. That has the link command that was used to build the dylib. Does it list many rpath entries?

BarrySmith commented 7 years ago

$ cmake --version cmake version 3.6.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

Uploaded link.txt link.txt

bradking commented 7 years ago

From the link.txt I can see that many extra -Wl,-rpath,... link flags appear. CMake always generates a single block of these before all the libraries start. CMake also does de-duplicate the list during construction. However, link.txt shows many -Wl,-rpath,... flags interspersed among the libraries. These must be coming from the TriBITS or Trilinos infrastructure through target_link_libraries calls that expliclty specify the rpath flags. @bartlettroscoe?

bartlettroscoe commented 7 years ago

These must be coming from the TriBITS or Trilinos infrastructure through target_link_libraries calls that expliclty specify the rpath flags

The only thing that the TriBITS CMake build for xSDKTrilinos does in this use case is to read in the file TrilinosConfig.cmake and link the libraries listed there to the xSDKTrilinos libraries.

@BarrySmith, can you please attach the file TrilinosConfig.cmake that is installed with Trilinos on this machine? That will show if it is listing a bunch of -Wl,-rpath, entries and where they are coming from.

BarrySmith commented 7 years ago

Here it is

TrilinosConfig.cmake.txt

bartlettroscoe commented 7 years ago

It looks like the PETSc configure of Trilinos is likely explicitly passing through all of these -Wl,-rpath paths when it configures Trilinos. That is because they show up in the variables Trilinos_EXTRA_LD_FLAGS and Trilinos_TPL_LIBRARIES. These variables are directly provided by the user who configures Trilinos. TriBITS just takes these and says "yes sir" and uses them.

## Extra link flags (e.g., specification of fortran libraries)
SET(Trilinos_EXTRA_LD_FLAGS "-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib [...] ")

[...]

## The project tpl libraries
#SET(Trilinos_TPL_LIBRARIES "-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib [...]")

@BarrySmith, to make sure this is the case, can you please attach one of the <Package>Config.cmake files this is included by TrilinosConfig.cmake like StratimikosConfig.cmake? That will show what paths CMake/TriBITS is trying to pass through.

Also, where is the Python code in PETSc that configures Trilinos and xSDKTrilinos? That might show what is happening and how to resolve this.

BarrySmith commented 7 years ago

I've added four files: the two python scripts for trilinos and xsdktrilinos and the exact input generated from each of these passed to their cmake.

Curse the .txt suffix requirement yet again 👎 pkg.conf.trilinos.txt pkg.conf.xsdktrilinos.txt Trilinos.py.txt xSDKTrilinos.py.txt

also attaching StratimikosConfig.cmake StratimikosConfig.cmake.txt

bartlettroscoe commented 7 years ago

You can see these -Wl,-rpath entries in many of these files. Again, they look to mostly be coming in through the the variables <Name>_EXTRA_LD_FLAGS and <Name>_TPL_LIBRARIES which are just passed on from the configure command for Trilinos.

@BarrySmith, is there some way to see the log of the exact cmake command that is being used to configure Trilinos and xSDKTrilinos on this machine? Is this captured in a log file somewhere?

BarrySmith commented 7 years ago

Here it is for Trilinos

Executing: cd /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.trilinos/build && /usr/local/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/Users/barrysmith/Src/petsc/arch-xsdk -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="/Users/barrysmith/Src/petsc/arch-xsdk/bin/mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-Qunused-arguments -g3" -DCMAKE_CXX_COMPILER="/Users/barrysmith/Src/petsc/arch-xsdk/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-g" -DCMAKE_Fortran_COMPILER="/Users/barrysmith/Src/petsc/arch-xsdk/bin/mpif90" -DCMAKE_Fortran_FLAGS:STRING="-ffree-line-length-0 -g" -DBUILD_SHARED_LIBS=on -DUSE_XSDK_DEFAULTS=YES -DCMAKE_BUILD_TYPE=DEBUG -DTrilinos_ENABLE_DEBUG=YES -DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON -DTrilinos_EXTRA_LINK_FLAGS="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lmpifort -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpicxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -ldl -lmpi -lpmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -lclang_rt.osx -ldl " -DTeuchos_ENABLE_FLOAT=OFF -DTeuchos_ENABLE_COMPLEX=OFF -DTpetra_INST_FLOAT=OFF -DTpetra_INST_COMPLEX_FLOAT=OFF -DTpetra_INST_COMPLEX_DOUBLE=OFF -DCMAKE_INSTALL_NAME_DIR:STRING="/Users/barrysmith/Src/petsc/arch-xsdk/lib" -DTPL_ENABLE_Boost=ON -DTPL_Boost_INCLUDE_DIRS="/Users/barrysmith/Src/petsc/arch-xsdk/include" -DTPL_Boost_LIBRARIES= -DTPL_ENABLE_MPI=ON -DTrilinos_ENABLE_Epetra=ON -DTrilinos_ENABLE_AztecOO=ON -DTrilinos_ENABLE_Ifpack=ON -DTrilinos_ENABLE_Amesos2=ON -DTrilinos_ENABLE_Tpetra=ON -DTrilinos_ENABLE_Sacado=ON -DTrilinos_ENABLE_Zoltan=ON -DTrilinos_ENABLE_Stratimikos=ON -DTrilinos_ENABLE_Thyra=ON -DTrilinos_ENABLE_Isorropia=ON -DTrilinos_ENABLE_ML=ON -DTrilinos_ENABLE_Belos=ON -DTrilinos_ENABLE_Anasazi=ON -DTrilinos_ENABLE_Zoltan2=ON -DTrilinos_ENABLE_Ifpack2=ON -DTrilinos_ENABLE_ShyLU=ON -DTrilinos_ENABLE_NOX=ON -DTrilinos_ENABLE_MueLu=ON -DTrilinos_ENABLE_Stokhos=ON -DTrilinos_ENABLE_ROL=ON -DTrilinos_ENABLE_Piro=ON -DTrilinos_ENABLE_Pike=ON -DTrilinos_ENABLE_TrilinosCouplings=ON -DTrilinos_ENABLE_Panzer=ON -DTrilinos_ENABLE_SEACAS=ON -DTPL_ENABLE_Matio=OFF -DTPL_ENABLE_GLM=OFF -DTPL_ENABLE_X11=OFF -DTrilinos_ENABLE_FEI=OFF -DTrilinos_ENABLE_STK=OFF -DTPL_BLAS_LIBRARIES="-llapack -lblas -Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lmpifort -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lm" -DTPL_LAPACK_LIBRARIES="-llapack -lblas -Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lmpifort -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lm" -DTrilinos_ASSERT_MISSING_PACKAGES=OFF -DTPL_ENABLE_TPL_SuperLU:BOOL=OFF -DTPL_ENABLE_SuperLUDist:BOOL=ON -DTPL_SuperLUDist_INCLUDE_DIRS="/Users/barrysmith/Src/petsc/arch-xsdk/include" -DTPL_SuperLUDist_LIBRARIES="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lsuperlu_dist" -DTPL_ENABLE_HYPRE:BOOL=ON -DTPL_HYPRE_INCLUDE_DIRS="/Users/barrysmith/Src/petsc/arch-xsdk/include" -DTPL_HYPRE_LIBRARIES="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lHYPRE -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lclang_rt.osx -lmpicxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -lclang_rt.osx" -DTPL_ENABLE_TPL_PARDISO_MKL:BOOL=OFF -DTPL_ENABLE_METIS:BOOL=ON -DTPL_METIS_INCLUDE_DIRS="/Users/barrysmith/Src/petsc/arch-xsdk/include" -DTPL_METIS_LIBRARIES="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lmetis" -DTPL_ENABLE_ParMETIS:BOOL=ON -DTPL_ParMETIS_INCLUDE_DIRS="/Users/barrysmith/Src/petsc/arch-xsdk/include" -DTPL_ParMETIS_LIBRARIES="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lparmetis" -DTPL_ENABLE_Scotch:BOOL=OFF -DTPL_ENABLE_HDF5:BOOL=ON -DTPL_HDF5_INCLUDE_DIRS="/Users/barrysmith/Src/petsc/arch-xsdk/include" -DTPL_HDF5_LIBRARIES="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5" -DTPL_ENABLE_Netcdf:BOOL=ON -DTPL_Netcdf_INCLUDE_DIRS="/Users/barrysmith/Src/petsc/arch-xsdk/include" -DTPL_Netcdf_LIBRARIES="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lnetcdf -lhdf5hl_fortran -lhdf5_fortran -lhdf5_hl -lhdf5" -DTPL_ENABLE_ExodusII:BOOL=OFF

Here it is for xsdktrilinos

Executing: cd /Users/barrysmith/Src/petsc/arch-xsdk/externalpackages/git.xsdktrilinos/build && /usr/local/bin/cmake .. -DCMAKE_INSTALL_PREFIX=/Users/barrysmith/Src/petsc/arch-xsdk -DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="/Users/barrysmith/Src/petsc/arch-xsdk/bin/mpicc" -DCMAKE_AR=/usr/bin/ar -DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-Qunused-arguments -g3" -DCMAKE_CXX_COMPILER="/Users/barrysmith/Src/petsc/arch-xsdk/bin/mpicxx" -DCMAKE_CXX_FLAGS:STRING="-g" -DCMAKE_Fortran_COMPILER="/Users/barrysmith/Src/petsc/arch-xsdk/bin/mpif90" -DCMAKE_Fortran_FLAGS:STRING="-ffree-line-length-0 -g" -DBUILD_SHARED_LIBS=on -DUSE_XSDK_DEFAULTS=YES -DTRILINOS_INSTALL_DIR=/Users/barrysmith/Src/petsc/arch-xsdk -DTrilinos_INSTALL_DIR=/Users/barrysmith/Src/petsc/arch-xsdk -DTPL_ENABLE_HYPRE=ON -DTPL_HYPRE_LIBRARIES="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lHYPRE -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lclang_rt.osx -lmpicxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -lclang_rt.osx" -DTPL_HYPRE_INCLUDE_DIRS=/Users/barrysmith/Src/petsc/arch-xsdk/include -DTPL_ENABLE_PETSC=ON -DTPL_PETSC_LIBRARIES="-L/Users/barrysmith/Src/petsc/arch-xsdk/lib -lpetsc -Wl,-rpath,/usr/local/opt/openssl/lib -L/usr/local/opt/openssl/lib -lssl -lcrypto -Wl,-rpath,/opt/X11/lib -L/opt/X11/lib -lX11" -DTPL_PETSC_INCLUDE_DIRS=/Users/barrysmith/Src/petsc/include -DCMAKE_BUILD_TYPE=DEBUG -DxSDKTrilinos_ENABLE_DEBUG=YES -DxSDKTrilinos_EXTRA_LINK_FLAGS="-Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/8.0.0/lib/darwin -lmpifort -lgfortran -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6/gcc/x86_64-apple-darwin16.0.0/6.2.0 -Wl,-rpath,/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -L/usr/local/Cellar/gcc/6.2.0/lib/gcc/6 -lgfortran -lgcc_ext.10.5 -lquadmath -lm -lclang_rt.osx -lmpicxx -lc++ -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -lclang_rt.osx -Wl,-rpath,/Users/barrysmith/Src/petsc/arch-xsdk/lib -L/Users/barrysmith/Src/petsc/arch-xsdk/lib -ldl -lmpi -lpmpi -lSystem -Wl,-rpath,/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -L/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/8.0.0/lib/darwin -lclang_rt.osx -ldl " -DxSDKTrilinos_ENABLE_TESTS=ON

They are also what was previously contained in the previously attached pk.conf.* files

bartlettroscoe commented 7 years ago

I think explicitly passing all of these -Wl,-rpath, arguments to the extra link flags as well as for every TPL is the source of the problem. The way this is supposed to work is that if you want CMake to to put in RPATH for the libraries it links to, you just pass in the the path to the shared libraries and then set CMake options to have CMake put the RPATHs in automatically in the build and install dirs. Then it will put in other the minimal RPATH entries that are needed.

What I would like to try to resolve this once and for all is to implement TriBITSPub/TriBITS#126 and then test this out with Trilinos on Linux and OSX. Then I think PETSc can strip all of the -Wl,-rpath, arguments out if the configure line for Trilinos and everything should work correctly.

@bradking, does this sound like the best course of action?

bradking commented 7 years ago

@bartlettroscoe yes, getting INSTALL_RPATH properties set correctly should solve all this. That is what it is meant to do.

BarrySmith commented 7 years ago

Ok, let me know when I should try the new API.

bartlettroscoe commented 7 years ago

One question: What versions for Trilinos and xSDKTrilinos are being tested above? Is this the release version in the last xSDK release or the 'master' versions? I am asking because the Trilinos 'master' version should already have different behavior than the past release of Trilinos due to TriBITS setting CMAKE_POLICY(VERSION 3.3).

Anyway, I am working on TriBITSPub/TriBITS#126 now trying to understand the behaviors of CMake for RPATH and get this fixed once and for all.

bartlettroscoe commented 7 years ago

If the PETSc build removes the manually set -rpath for the TPL dirs for Trilinos, this should be fixed. The fixing Trilinos commit is:

commit f2b04f756935a55664f49d96aa24da3e4e688e70
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Fri Nov 4 15:34:03 2016 -0600

    Automatic snapshot commit from tribits at 4769da4

    Origin repo remote tracking branch: 'github/master'
    Origin repo remote repo URL: 'github = git@github.com:TriBITSPub/TriBITS.git'

    At commit:

    commit 4769da42b41ca12e295d829cd543d4c17a44417d
    Author:  Roscoe A. Bartlett <rabartl@sandia.gov>
    Date:    Thu Nov 3 07:36:18 2016 -0600
    Summary: Some edits after review from @ibaned (#126)

   8.5% cmake/tribits/core/modules/
  16.5% cmake/tribits/core/package_arch/
   5.4% cmake/tribits/core/utils/
   4.0% cmake/tribits/ctest_driver/
  20.3% cmake/tribits/doc/build_ref/
  18.9% cmake/tribits/doc/developers_guide/
  12.9% cmake/tribits/python_utils/
  13.1% cmake/tribits/

@BarrySmith, if someone on the PETSc team has time to strip out the manually set -rpath args then this should be fixed for OSX.

BarrySmith commented 7 years ago

@bartlettroscoe Thanks, I'm working on it now. One thing that would be useful, is there any easy scriptable way to pull out the Trilinos version at pre-cmake time? I can only strip out the rpath for for new Trilinos so need to know the version of Trilinos before building it. I can manually dig around and find it in Trilinos_version.h but do you have a trick to pull it out automatically?

bartlettroscoe commented 7 years ago

One thing that would be useful, is there any easy scriptable way to pull out the Trilinos version at pre-cmake time?

Yup. You can read it out of the Trilinos/Version.cmake file. The best way to do that is with a cmake -P script. If you are not sure how to do that, let me know and I can throw one together pretty fast.

Note that in the future I would like to configure the Version.cmake file by reading the version for the git tag with git described as described here. So in the future you can just run:

git describe --match=”`cat trilinos_tag_prefix.txt`*"

and get the version from that which will obey the semantic versioning standard.

BarrySmith commented 7 years ago

Googling cmake -P script did not help me at all. So yes if you could send me something that would be great

On Nov 30, 2016, at 2:49 PM, Roscoe A. Bartlett notifications@github.com wrote:

One thing that would be useful, is there any easy scriptable way to pull out the Trilinos version at pre-cmake time?

Yup. You can read it out of the Trilinos/Version.cmake file. The best way to do that is with a cmake -P script. If you are not sure how to do that, let me know and I can throw one together pretty fast.

Note that in the future I would like to configure the Version.cmake file by reading the version for the git tag with git described as described here. So in the future you can just run:

git describe --match=”cat trilinos_tag_prefix.txt*"

I may not have obtained cmake from git.

Barry

and get the version from that which will obey the semantic versioning standard.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

bartlettroscoe commented 7 years ago

Googling cmake -P script did not help me at all. So yes if you could send me something that would be great

@BarrySmith,

Copy the attached file GetProjectVersion.cmake.txt to GetProjectVersion.cmake and then run it as:

cmake -D PROJECT_BASE_DIR=<trilinos-base-dir> -P GetProjectVersion.cmake

When I ran it on the current Trilinos 'develop' version with:

cmake -DPROJECT_BASE_DIR=$HOME/Trilinos.base/Trilinos -P GetProjectVersion.cmake

I get the STDOUT:

Trilinos_VERSION = 12.9
Trilinos_MAJOR_VERSION = 12
Trilinos_MAJOR_MINOR_VERSION = 120900
Trilinos_VERSION_STRING = '12.9 (Dev)'

You should be able to parse that output pretty easy in Python.

Let me know if you have any issues with this.

Again, in the near future I hope to move to a git tag model based on git describe as I mention above.

BarrySmith commented 7 years ago

Almost there. I can build trilinos without the rpath. I can also build the xsdktrilinos EXACT for one problem

otool -L /Users/barrysmith/Src/petsc/arch-basic/externalpackages/git.xsdktrilinos/build/hypre/src/libxsdkhypre.00.dylib /Users/barrysmith/Src/petsc/arch-basic/externalpackages/git.xsdktrilinos/build/hypre/src/libxsdkhypre.00.dylib: @rpath/libxsdkhypre.00.dylib (compatibility version 0.0.0, current version 0.1.0) /Users/barrysmith/Src/petsc/arch-basic/lib/libpetsc.3.07.dylib (compatibility version 3.7.0, current version 3.7.4) /usr/local/opt/openssl@1.1/lib/libssl.1.1.dylib (compatibility version 1.1.0, current version 1.1.0) /usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib (compatibility version 1.1.0, current version 1.1.0) /opt/X11/lib/libX11.6.dylib (compatibility version 10.0.0, current version 10.0.0) /Users/barrysmith/Src/petsc/arch-basic/lib/libpike-blackbox.12.dylib (compatibility version 12.0.0, current version 12.9.0)

otool -L /Users/barrysmith/Src/petsc/arch-basic/lib/libxsdk*.dylib | grep rpath @rpath/libxsdkhypre.00.dylib (compatibility version 0.0.0, current version 0.1.0) @rpath/libxsdkhypre.00.dylib (compatibility version 0.0.0, current version 0.1.0) @rpath/libxsdkhypre.00.dylib (compatibility version 0.0.0, current version 0.1.0) @rpath/libxsdkpetsc.00.dylib (compatibility version 0.0.0, current version 0.1.0) @rpath/libxsdkpetsc.00.dylib (compatibility version 0.0.0, current version 0.1.0) @rpath/libxsdkpetsc.00.dylib (compatibility version 0.0.0, current version 0.1.0)

Presumably something needs to be fixed in the xsdktrilinos cmake @amklinv