trilinos / xSDKTrilinos

BSD 3-Clause "New" or "Revised" License
6 stars 6 forks source link

Fix build of xSDKTrilinos with static TPL libs #2

Open bartlettroscoe opened 8 years ago

bartlettroscoe commented 8 years ago

In the below email, Jason reports that xSDKTrilinos is putting the libraries in the wrong order on the link line.

This Issue ticket is to investigate this issue and try to resolve it.


From: Jason Sarich jason.sarich@gmail.com on behalf of "Sarich, Jason J." sarich@mcs.anl.gov Date: Tuesday, April 12, 2016 at 1:41 PM To: "Smith, Barry F." bsmith@mcs.anl.gov Cc: "Xiaoye (Sherry) Li" xsli@lbl.gov, "Balay, Satish" balay@mcs.anl.gov, "Willenbring, James M" jmwille@sandia.gov, Sherry Li xiaoye@nersc.gov, "Klinvex, Alicia Marie" amklinv@sandia.gov, Lois Curfman McInnes curfman@mcs.anl.gov Subject: Re: nearing final preparations for xSDK release

I'm still having some trouble with installxSDK.sh on mira, I was able to fix a few issues, but now I'm getting a bad link line for xsdktrilinos. libpetsc.a is not added until the very end, but it depends on previous libraries (hdf5, superlu, metis, hypre, ml, etc.), see link command below.

I'm hoping somebody more familiar with cmake and the xsdktrilinos package can figure out where the library list is coming from quicker than I can. There's also a '-rdynamic' in there that I don't think will work.

[sarich@miralac1 example]$ /soft/compilers/wrappers/gcc/mpicxx    -pedantic -Wall -Wno-long-long -Wwrite-strings   -g -std=c++11 -std=c++11\
 -g -O0   CMakeFiles/xSDKTrilinos_PETSc_MueLu_example.dir/PETSc_MueLuEx.cpp.o  -o xSDKTrilinos_PETSc_MueLu_example.exe  -L/projects/OSCon/s\
arich/xsdkinstall/lib -rdynamic ../src/libxsdkpetsc.a /projects/OSCon/sarich/xsdkinstall/lib/libpike-blackbox.a /projects/OSCon/sarich/xsdk\
install/lib/libtrilinoscouplings.a /projects/OSCon/sarich/xsdkinstall/lib/libpanzer-disc-fe.a /projects/OSCon/sarich/xsdkinstall/lib/libpan\
zer-dof-mgr.a /projects/OSCon/sarich/xsdkinstall/lib/libpanzer-core.a /projects/OSCon/sarich/xsdkinstall/lib/libphalanx.a /projects/OSCon/s\
arich/xsdkinstall/lib/libintrepid2.a /projects/OSCon/sarich/xsdkinstall/lib/libpiro.a /projects/OSCon/sarich/xsdkinstall/lib/libstokhos_mue\
lu.a /projects/OSCon/sarich/xsdkinstall/lib/libstokhos_ifpack2.a /projects/OSCon/sarich/xsdkinstall/lib/libstokhos_amesos2.a /projects/OSCo\
n/sarich/xsdkinstall/lib/libstokhos_tpetra.a /projects/OSCon/sarich/xsdkinstall/lib/libstokhos_sacado.a /projects/OSCon/sarich/xsdkinstall/\
lib/libstokhos.a /projects/OSCon/sarich/xsdkinstall/lib/libmuelu-adapters.a /projects/OSCon/sarich/xsdkinstall/lib/libmuelu-interface.a /pr\
ojects/OSCon/sarich/xsdkinstall/lib/libmuelu.a /projects/OSCon/sarich/xsdkinstall/lib/liblocathyra.a /projects/OSCon/sarich/xsdkinstall/lib\
/liblocaepetra.a /projects/OSCon/sarich/xsdkinstall/lib/liblocalapack.a /projects/OSCon/sarich/xsdkinstall/lib/libloca.a /projects/OSCon/sa\
rich/xsdkinstall/lib/libnoxepetra.a /projects/OSCon/sarich/xsdkinstall/lib/libnoxlapack.a /projects/OSCon/sarich/xsdkinstall/lib/libnox.a /\
projects/OSCon/sarich/xsdkinstall/lib/libteko.a /projects/OSCon/sarich/xsdkinstall/lib/libanasazitpetra.a /projects/OSCon/sarich/xsdkinstal\
l/lib/libModeLaplace.a /projects/OSCon/sarich/xsdkinstall/lib/libanasaziepetra.a /projects/OSCon/sarich/xsdkinstall/lib/libanasazi.a /proje\
cts/OSCon/sarich/xsdkinstall/lib/libstratimikos.a /projects/OSCon/sarich/xsdkinstall/lib/libstratimikosbelos.a /projects/OSCon/sarich/xsdki\
nstall/lib/libstratimikosaztecoo.a /projects/OSCon/sarich/xsdkinstall/lib/libstratimikosamesos.a /projects/OSCon/sarich/xsdkinstall/lib/lib\
stratimikosml.a /projects/OSCon/sarich/xsdkinstall/lib/libstratimikosifpack.a /projects/OSCon/sarich/xsdkinstall/lib/libml.a /projects/OSCo\
n/sarich/xsdkinstall/lib/libgaleri-xpetra.a /projects/OSCon/sarich/xsdkinstall/lib/libgaleri-epetra.a /projects/OSCon/sarich/xsdkinstall/li\
b/libifpack2-adapters.a /projects/OSCon/sarich/xsdkinstall/lib/libifpack2.a /projects/OSCon/sarich/xsdkinstall/lib/librol.a /projects/OSCon\
/sarich/xsdkinstall/lib/libamesos2.a /projects/OSCon/sarich/xsdkinstall/lib/libintrepid.a /projects/OSCon/sarich/xsdkinstall/lib/libsacado.\
a /projects/OSCon/sarich/xsdkinstall/lib/libshards.a /projects/OSCon/sarich/xsdkinstall/lib/librythmos.a /projects/OSCon/sarich/xsdkinstall\
/lib/liboptipack.a /projects/OSCon/sarich/xsdkinstall/lib/libglobipack.a /projects/OSCon/sarich/xsdkinstall/lib/libshylu.a /projects/OSCon/\
sarich/xsdkinstall/lib/libbelostpetra.a /projects/OSCon/sarich/xsdkinstall/lib/libbelosepetra.a /projects/OSCon/sarich/xsdkinstall/lib/libb\
elos.a /projects/OSCon/sarich/xsdkinstall/lib/libifpack.a /projects/OSCon/sarich/xsdkinstall/lib/libzoltan2.a /projects/OSCon/sarich/xsdkin\
stall/lib/libxpetra-sup.a /projects/OSCon/sarich/xsdkinstall/lib/libxpetra.a /projects/OSCon/sarich/xsdkinstall/lib/libthyratpetra.a /proje\
cts/OSCon/sarich/xsdkinstall/lib/libthyraepetraext.a /projects/OSCon/sarich/xsdkinstall/lib/libthyraepetra.a /projects/OSCon/sarich/xsdkins\
tall/lib/libthyracore.a /projects/OSCon/sarich/xsdkinstall/lib/librtop.a /projects/OSCon/sarich/xsdkinstall/lib/libamesos.a /projects/OSCon\
/sarich/xsdkinstall/lib/libaztecoo.a /projects/OSCon/sarich/xsdkinstall/lib/libisorropia.a /projects/OSCon/sarich/xsdkinstall/lib/libepetra\
ext.a /projects/OSCon/sarich/xsdkinstall/lib/libtriutils.a /projects/OSCon/sarich/xsdkinstall/lib/libtpetraext.a /projects/OSCon/sarich/xsd\
kinstall/lib/libtpetrainout.a /projects/OSCon/sarich/xsdkinstall/lib/libtpetra.a /projects/OSCon/sarich/xsdkinstall/lib/libkokkostsqr.a /pr\
ojects/OSCon/sarich/xsdkinstall/lib/libtpetrakernels.a /projects/OSCon/sarich/xsdkinstall/lib/libkokkosalgorithms.a /projects/OSCon/sarich/\
xsdkinstall/lib/libkokkoscontainers.a /projects/OSCon/sarich/xsdkinstall/lib/libtpetraclassiclinalg.a /projects/OSCon/sarich/xsdkinstall/li\
b/libtpetraclassicnodeapi.a /projects/OSCon/sarich/xsdkinstall/lib/libtpetraclassic.a /projects/OSCon/sarich/xsdkinstall/lib/libepetra.a /p\
rojects/OSCon/sarich/xsdkinstall/lib/libteuchoskokkoscomm.a /projects/OSCon/sarich/xsdkinstall/lib/libteuchoskokkoscompat.a /projects/OSCon\
/sarich/xsdkinstall/lib/libteuchosremainder.a /projects/OSCon/sarich/xsdkinstall/lib/libteuchosnumerics.a /projects/OSCon/sarich/xsdkinstal\
l/lib/libteuchoscomm.a /projects/OSCon/sarich/xsdkinstall/lib/libteuchosparameterlist.a /projects/OSCon/sarich/xsdkinstall/lib/libteuchosco\
re.a /projects/OSCon/sarich/xsdkinstall/lib/libkokkoscore.a /projects/OSCon/sarich/xsdkinstall/lib/libzoltan.a -lm /projects/OSCon/sarich/x\
sdkinstall/lib/libpamgen_extras.a /projects/OSCon/sarich/xsdkinstall/lib/libpamgen.a -Wl,-rpath,/projects/OSCon/sarich/xsdkinstall/lib -L/p\
rojects/OSCon/sarich/xsdkinstall/lib -lHYPRE -L/bgsys/drivers/V1R2M2/ppc64/comm/lib -L/bgsys/drivers/V1R2M2/ppc64/comm/lib64 -L/bgsys/drive\
rs/V1R2M2/ppc64/spi/lib -L/bgsys/drivers/V1R2M2/ppc64/comm/sys/lib -L/soft/compilers/gcc/4.8.4/lib/gcc/powerpc64-bgq-linux/4.8.4 -L/soft/co\
mpilers/gcc/4.8.4/lib/gcc -L/soft/compilers/gcc/4.8.4/powerpc64-bgq-linux/lib -lmpichcxx-gcc -L/soft/libraries/hdf5/1.8.14/cnk-gcc/V1R2M2-2\
0150515/lib -L/soft/libraries/alcf/current/xl/ZLIB/lib -lhdf5_hl -lhdf5 -lz -Wl,-rpath,/projects/OSCon/sarich/xsdkinstall/lib -L/projects/O\
SCon/sarich/xsdkinstall/lib -lflapack -Wl,-rpath,/projects/OSCon/sarich/xsdkinstall/lib -L/projects/OSCon/sarich/xsdkinstall/lib -lfblas -L\
/bgsys/drivers/V1R2M2/ppc64/comm/lib -L/bgsys/drivers/V1R2M2/ppc64/comm/lib64 -L/bgsys/drivers/V1R2M2/ppc64/spi/lib -L/bgsys/drivers/V1R2M2\
/ppc64/comm/sys/lib -L/soft/compilers/gcc/4.8.4/lib/gcc/powerpc64-bgq-linux/4.8.4 -L/soft/compilers/gcc/4.8.4/lib/gcc -L/soft/compilers/gcc\
/4.8.4/powerpc64-bgq-linux/lib -lmpichf90-gcc -lgfortran -lm -lgfortran -lm -lm -Wl,-rpath,/projects/OSCon/sarich/xsdkinstall/lib -L/projec\
ts/OSCon/sarich/xsdkinstall/lib -lsuperlu_dist -Wl,-rpath,/projects/OSCon/sarich/xsdkinstall/lib -L/projects/OSCon/sarich/xsdkinstall/lib -\
lparmetis -Wl,-rpath,/projects/OSCon/sarich/xsdkinstall/lib -L/projects/OSCon/sarich/xsdkinstall/lib -lmetis -Wl,-rpath,/projects/OSCon/sar\
ich/xsdkinstall/lib -L/projects/OSCon/sarich/xsdkinstall/lib -lexoIIv2for -lexodus /projects/OSCon/sarich/xsdkinstall/lib/libpetsc.a -lmpic\
hf90-gcc -lgfortran -Wl,-rpath,/projects/OSCon/sarich/xsdkinstall/lib
bartlettroscoe commented 8 years ago

@BarrySmith , the current 'master' version of PETSc fails the download of SuperLUDist. Do you know what happened with this? See the detailed error below.

CC: @jwillenbring


Detailed Notes:

Trying to produce a static build of xSDK with xSDKTrilinos ...

I cloned the installxSDK repo with:

$ git clone git@github.com:xsdk-project/installxSDK.git

I then attempted to do the static build with:

$ ./installxSDK.sh --prefix=../install/mpi_debug_static --enable-shared=0

but it got the following error in the configure of NetCDF:

checking for library containing H5Fflush... noconfigure: error: Can't find or link to the hdf5 library. Use --disable-netcdf-4, or see config.log for errors.

I sent Barry and Jason an email with the detailed configure.log file.

I just realized that the configure script for PETSc that Alicia and I were using did not download and install NetCDF or ExodusII or Metis or Boost.

Barry said that the installxSDK.sh script is actaully currently cloning PETSc from bitbucket master, cloning Trilinos on the 12.6 branch and then cloning xSDKTrilinos on its master branch.

Therefore, I will go back and try doing a static build of PETSc then Trilinos, then xSDKTrilinos individually and see if I can reproduce this problem.

1) Install PETSc and TPLs with static libs:

Using PETSc on master:

$ cd PETSC_BASE=/home/8vt/localhome/PROJECTS/PETSc.base/petsc/
$ git checkout master
$ git pull
$ git log-short --name-status -1

1736e08 "Added --download-xxx-configure-arguments for passing additional configure options"
Author: Barry Smith <bsmith@mcs.anl.gov>
Date:   Tue Apr 12 14:34:26 2016 -0500 (2 hours ago)

M       config/BuildSystem/config/package.py

Configure, build, and install PETSc and its TPLs:

$ export PETSC_BASE=/home/8vt/localhome/PROJECTS/PETSc.base
$ cd $PETSC_BASE/
$ cd petsc/
$ git clean -fdx
$ export PETSC_INSTALL_DIR=$PETSC_BASE/install/mpi-debug-static
$ time ./configure \
  --prefix=${PETSC_INSTALL_DIR} --enable-shared=0 \
  --with-cxx-dialect=C++11 --with-c++-support --with-clanguage=cxx \
  --download-superlu_dist --download-hypre --download-metis --download-parmetis \
  &> configure.out

real    2m5.420s
user    3m40.979s
sys 1m3.290s

This failed to configure, showing the failure:

Unable to checkout commit: v5.0.0 in repository: /localhome/8vt/PROJECTS/PETSc.base/petsc/arch-linux2-cxx-debug/externalpackages/git.superlu_dist.
Perhaps its a remote branch, if so - use: origin/v5.0.0

It appears as though that tag does not exist in that git repo:

$ cd /localhome/8vt/PROJECTS/PETSc.base/petsc/arch-linux2-cxx-debug/externalpackages/git.superlu_dist
[8vt@th232 git.superlu_dist ((v4.3-p1))]$ git tag
v1
v2
v3.0
v3.1
v3.2
v3.3
v3.3-p1
v4.0
v4.0-p1
v4.0-p2
v4.0-p3
v4.1
v4.2
v4.3
v4.3-p1

The remote is actually a PETSc mirror:

$ git remote -v
origin  https://bitbucket.org/petsc/pkg-superlu_dist.git (fetch)
origin  https://bitbucket.org/petsc/pkg-superlu_dist.git (push)

I did a fetch:

$ git fetch origin

and it shows:

[8vt@th232 git.superlu_dist (master)]$ git ls-remote --tags
From https://bitbucket.org/petsc/pkg-superlu_dist.git
d181701f7656437fa799700461595494f1cf0d7a    refs/tags/v1
4cd7e0bc4ab418fbd4e9edd299d4d5e9cf2d07df    refs/tags/v2
e489440123f5ad750bd552d4f198014411d2c3b6    refs/tags/v3.0
f3b65b5bdb7ac6f05dc550dde796f7a183cd3132    refs/tags/v3.1
be7e97dc9404f5b817ca14f51f806c5b232559e9    refs/tags/v3.2
749f33d8104157767d443ff1a1d151642751486d    refs/tags/v3.3
477ac4cb42240655cde9eb1e077f03af87b6c9fa    refs/tags/v3.3-p1
01fcf8aed07ff59fddfec8a624a9a079dec480a4    refs/tags/v4.0
bdddaa1c55e208b48f96d6281b8713b55f836c6e    refs/tags/v4.0-p1
27583c7ca1d2b1b0760a8f20033d92bf3b0163c1    refs/tags/v4.0-p2
dad206a611c05e60b3b5dd58030aae0941dc93ed    refs/tags/v4.0-p3
1e387bb0c07bec2b746fd388e6c40892b608a8ef    refs/tags/v4.1
c66af245f0c24df3aff80d98f58215f069dfd2a1    refs/tags/v4.2
7b6195f2dcd9383d5cf296be5d18b015d48fb977    refs/tags/v4.3
934ad8f128722c600221150ae0108f7c5b3568de    refs/tags/v4.3-p1

What is up with that? Why is PETSc 'master' pointing to a tag that does not even exist?

It seems this change got made 3 days ago in the commit:

commit b8a3fda2eec00bbbfad0cf0db17d1d1f04b07f29
Author: Barry Smith <bsmith@mcs.anl.gov>
Date:   Sat Apr 9 19:36:18 2016 -0500

    Updated to the release tarballs and commits for SuperLU and SuperLU_Dist

diff --git a/config/BuildSystem/config/packages/SuperLU_DIST.py b/config/BuildSystem/config/packages/SuperLU_DIST.py
index b01ff94..76af43e 100644
--- a/config/BuildSystem/config/packages/SuperLU_DIST.py
+++ b/config/BuildSystem/config/packages/SuperLU_DIST.py
@@ -4,9 +4,8 @@ import os
 class Configure(config.package.CMakePackage):
   def __init__(self, framework):
     config.package.CMakePackage.__init__(self, framework)
-    self.gitcommit        = '892d677'
-#    self.download         = ['git://https://github.com/petsc/superlu_dist']
-    self.download         = ['git://https://github.com/xiaoyeli/superlu_dist']
+    self.gitcommit         = 'v5.0.0'
+    self.download         = ['http://crd-legacy.lbl.gov/~xiaoye/SuperLU/superlu_dist_5.0.0.tar.gz','git://https://github.com/xiaoyeli/superlu_dist']
     self.functions        = ['set_default_options_dist']
     self.includes         = ['superlu_ddefs.h']
     self.liblist          = [['libsuperlu_dist.a']]

I don't understand how this works for anyone else.

bartlettroscoe commented 8 years ago

@BarrySmith, can someone please point me to a version of PETSc that is known to work for the arguments:

  --enable-shared=0 \
  --with-cxx-dialect=C++11 --with-c++-support --with-clanguage=cxx \
  --download-superlu_dist --download-hypre --download-metis --download-parmetis \

?

The problem is the that download (git clone then checkout at a commit or tag) fails.


Detailed Notes:

since I can't configure and build the current 'master' version of PETSc, I am going to try to get an older version to work. How about the version right before b8a3fda2eec0?

1) Install PETSc and TPLs with static libs:

Using PETSc on master:

$ cd PETSC_BASE=/home/8vt/localhome/PROJECTS/PETSc.base/petsc/
$ git checkout b8a3fda2eec0^
$ git log-short --name-status -1

1736e08 "Added --download-xxx-configure-arguments for passing additional configure options"
Author: Barry Smith <bsmith@mcs.anl.gov>
Date:   Tue Apr 12 14:34:26 2016 -0500 (2 hours ago)

M       config/BuildSystem/config/package.py

Configure, build, and install PETSc and its TPLs:

$ export PETSC_BASE=/home/8vt/localhome/PROJECTS/PETSc.base
$ cd $PETSC_BASE/
$ cd petsc/
$ git clean -fdx
$ export PETSC_INSTALL_DIR=$PETSC_BASE/install/mpi-debug-static
$ time ./configure \
  --prefix=${PETSC_INSTALL_DIR} --enable-shared=0 \
  --with-cxx-dialect=C++11 --with-c++-support --with-clanguage=cxx \
  --download-superlu_dist --download-hypre --download-metis --download-parmetis \
  &> configure.out

$ time make PETSC_DIR=$PETSC_BASE/petsc PETSC_ARCH=arch-linux2-cxx-debug \
  MAKE_NP=12 all &> make.out

real    2m3.785s
user    3m36.466s
sys 1m1.727s

This time I got the failure:

*******************************************************************************
         UNABLE to CONFIGURE with GIVEN OPTIONS    (see configure.log for details):
-------------------------------------------------------------------------------
Unable to checkout commit: 892d677 in repository: /localhome/8vt/PROJECTS/PETSc.base/petsc/arch-linux2-cxx-debug/externalpackages/git.superlu_dist.
Perhaps its a remote branch, if so - use: origin/892d677
*******************************************************************************

Darn, this could be harder that I imagined.

BarrySmith commented 8 years ago

Sorry about this. I will check why the SuperLU_Dist is not working. First it should be defaulting to using the tarball so I am not sure why the it is using the git version. I may have made a mistake in the tag name.

BarrySmith commented 8 years ago

Hmm, I just got the superlu_dist repository and it definitely has the tag v5.0.0

$ git clone https://github.com/xiaoyeli/superlu_dist.git Cloning into 'superlu_dist'... remote: Counting objects: 1869, done. remote: Compressing objects: 100% (164/164), done. remote: Total 1869 (delta 78), reused 0 (delta 0), pack-reused 1705 Receiving objects: 100% (1869/1869), 3.37 MiB | 4.28 MiB/s, done. Resolving deltas: 100% (1270/1270), done. Checking connectivity... done. ~/Src $ cd superlu_dist/ ~/Src/superlu_dist (master=) $ git tag v5.0.0 ~/Src/superlu_dist (master=) $ git checkout v5.0.0 Note: checking out 'v5.0.0'.

~/Src/superlu_dist ((v5.0.0))

Can you try deleting the entire directory under installxSDK.sh and run it again. If that fails email the configure.log from the petsc directory to petsc-maint@mcs.anl.gov

BarrySmith commented 8 years ago

@bartlettroscoe

I was able to run with a clean build and get the failure in xSDKTrilinos as you predicted due to library link order issues. Here is the relevent information. If you are able to update the xSDKTrilinos cmake stuff I can try again. Just let me know.

config.log.gz

bartlettroscoe commented 8 years ago

Barry,

Okay, I know what happened. When I ran git clean -xdf is did not remove the existing git repos:

$ git clean -xdfn
...
Would skip repository arch-linux2-cxx-debug/externalpackages/SuperLU_DIST
...

Generally, I consider this to be a feature of git clean (and we rely on this with CASL VERA when cleaning all of the *.pyc files before creating a release tarball).

So that means to completely clean you can't just run git clean -xdf since git will not remove slave git repos by default. It appears you have to pass in -f twice! That seems to do the trick:

$ git clean -xdffn
...
Would remove arch-linux2-c-debug/
Would remove arch-linux2-cxx-debug/
...

(This is an example of why it is generally desirable to put build artifacts under a different directory not under the git repo. That way, you can safely use rm -rf <build-dir> and there is no question that the build artifacts are gone but that source tree is unaffected. You generally only want to put things under the base source dir/repo that would be very hard to put elsewhere.)

Anyway, I am sure it will work fine now (i.e. I should be able to reproduce the error on my machine).

Thanks!

bartlettroscoe commented 8 years ago

What is likely happening is described in TriBITSPub/TriBITS#115. The fix will be a simple hack (i.e. list all of the TPL dependencies, even the indirect ones, in xSDKTrilinos/cmake/Dependencies.cmake).

The correct fix will come when I can work TriBITSPub/TriBITS#115.

BarrySmith commented 8 years ago

Do you have write access to xSDKTrilinos or do we need to make fork to put in the fix?

Barry

On Apr 12, 2016, at 9:53 PM, Roscoe A. Bartlett notifications@github.com wrote:

What is likely happening is described in TriBITSPub/TriBITS#115. The fix will be a simple hack (i.e. list all of the TPL dependencies, even the indirect ones, in xSDKTrilinos/cmake/Dependencies.cmake).

The correct fix will come when I can work TriBITSPub/TriBITS#115.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

bartlettroscoe commented 8 years ago

Do you have write access to xSDKTrilinos or do we need to make fork to put in the fix?

I have write access. Once I have this reproduced, I will put in the fix and push. Will be tomorrow before I get that finished though.

bartlettroscoe commented 8 years ago

Barry,

I did a full static build of PETSc, Trilinos, and xSDKTrilinos manually as described below. Other than having to update my configure scripts for SuperLUDist v5.0.0 and having to add -lXll and -lssl to the link lines, all of xSDKTrilinos built and ran all of the tests ran successfully. Therefore, I was not able to reproduce a problem with Trilinos or xSDKTrilinos.

The issue may be in how Trilinos and/or xSDKTrilinos are being configured by installxSDK.sh or PETSc when static libs are used. Unfortunately, I can't reproduce the problem with installxSDK.sh on my own machine because it bombs in the configure of NetCDF on my machine as described above (which is perhaps an issue that should be looked at separately because it impacts the portability of xSDK).

Is there a way to turn off the download/install of NetCDF and other problemantic libraries with installxSDK.sh? Do I do this using --with-netcdf=0? I will give that a try.


Detailed Notes:

Reproducing failed static build ...

1) Install PETSc with static libs:

Using PETSc on master:

$ cd PETSC_BASE=/home/8vt/localhome/PROJECTS/PETSc.base/petsc/
$ git checkout master
$ git pull
$ git log-short --name-status -1

1736e08 "Added --download-xxx-configure-arguments for passing additional configure options"
Author: Barry Smith <bsmith@mcs.anl.gov>
Date:   Tue Apr 12 14:34:26 2016 -0500 (2 hours ago)

M       config/BuildSystem/config/package.py

Configure, build, and install PETSc and its TPLs:

$ export PETSC_BASE=/home/8vt/localhome/PROJECTS/PETSc.base
$ cd $PETSC_BASE/
$ cd petsc/
$ git clean -fdx
$ export PETSC_INSTALL_DIR=$PETSC_BASE/install/mpi-debug-static
$ time ./configure \
  --prefix=${PETSC_INSTALL_DIR} --enable-shared=0 \
  --with-cxx-dialect=C++11 --with-c++-support --with-clanguage=cxx \
  --download-superlu_dist --download-hypre --download-metis --download-parmetis \
  &> configure.out

real    3m5.988s
user    4m56.414s
sys 1m11.047s

$ time make PETSC_DIR=$PETSC_BASE/petsc PETSC_ARCH=arch-linux2-cxx-debug \
  MAKE_NP=12 all &> make.out

real    1m50.860s
user    17m36.296s
sys 2m35.365s

$ time make PETSC_DIR=$PETSC_BASE/petsc PETSC_ARCH=arch-linux2-cxx-debug \
  MAKE_NP=12 install &> make.install.out 

real    0m3.213s
user    0m0.436s
sys 0m1.093s

This produced the install tree:

$ ls -w 1 /home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/
libHYPRE.a
libmetis.a
libparmetis.a
libpetsc.a
libsuperlu_dist.a
petsc
pkgconfig

2) Install Trilinos packages needed by xSDKTrilinos with static libs:

The version of Trilinos on the branch trilinos-release-12-6-branch is:

[8vt@th232 Trilinos (trilinos-release-12-6-branch)]$ git log-short --name-status -1

1176a8d "Move install of TriBITS to end, add optional skip install (SDK-79)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Wed Apr 6 08:23:02 2016 -0400 (7 days ago)

M       CMakeLists.txt
M       cmake/CallbackSetupExtraOptions.cmake

Configure, build and install Trilinos packages needed by xSDKTrilinos:

$ cd home/8vt/localhome/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/
$ mkdir MPI_DEBUG_DEBUG_STATIC
$ cd MPI_DEBUG_DEBUG_STATIC/
$ ln -s \
  ../../../Trilinos/packages/xSDKTrilinos/scripts/do-configure-trilinos-fissile4 \
  do-configure
$ rm -r CMake*
$ export PETSC_INSTALL_DIR=$HOME/localhome/PROJECTS/PETSc.base/install/mpi-debug-static
$ export \
  TRILINOS_INSTALL_DIR=$HOME/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static
$ time ./do-configure \
  -DBUILD_SHARED_LIBS=OFF \
  -DTrilinos_ENABLE_Teuchos=ON \
  -DTrilinos_ENABLE_Tpetra=ON \
  -DTrilinos_ENABLE_Belos=ON \
  -DTrilinos_ENABLE_Ifpack2=ON \
  -DTrilinos_ENABLE_MueLu=ON \
  -DTrilinos_ENABLE_Anasazi=ON \
  -DTrilinos_ENABLE_Amesos2=ON \
  -DTrilinos_ENABLE_Epetra=ON \
  -DTrilinos_ENABLE_EpetraExt=ON \
  -DTrilinos_ENABLE_TESTS=OFF \
  -DCMAKE_INSTALL_PREFIX=${TRILINOS_INSTALL_DIR} \
  &> configure.out

real    0m23.184s
user    0m16.060s
sys 0m5.160s

$ time make -j12 install &> make.install.out

real    12m35.811s
user    85m58.969s
sys 12m36.579s

This installed:

 ~/tree.py -c --depth=2 ${TRILINOS_INSTALL_DIR}
  mpi_debug_debug_static/
  +-include/
  | +-Cuda/
  | +-OpenMP/
  | +-Qthread/
  | +-Threads/
  +-lib/

3) Build xSDKTrilinos wtih static libs:

The xSDKTrilinos version of 'master' is:

[8vt@th232 xSDKTrilinos (master)]$ git log-short --name-status -1

ea46243 "Fix SuperLUDist lib name, require PETSC and TRILINOS install dirs (#2)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Wed Apr 13 11:26:29 2016 -0400 (78 minutes ago)

M       scripts/do-configure-fissile4
M       scripts/do-configure-trilinos-fissile4

Configure and build xSDKTrilinos against installed Trilinos:

$ cd ???
$ cd /home/8vt/localhome/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/
$ mkdir XSDKTRILINOS_MPI_DEBUG_DEBUG_STATIC
$ cd XSDKTRILINOS_MPI_DEBUG_DEBUG_STATIC/
$ ln -s \
  ../../../Trilinos/packages/xSDKTrilinos/scripts/do-configure-fissile4 \
  do-configure

$ export PETSC_INSTALL_DIR=$HOME/localhome/PROJECTS/PETSc.base/install/mpi-debug-static
$ export \
  TRILINOS_INSTALL_DIR=$HOME/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static

$ time ./do-configure -DBUILD_SHARED_LIBS=OFF &> configure.out

real    0m11.485s
user    0m6.747s
sys 0m3.350s

$ time make -j12 &> make.out

real    1m16.029s
user    4m0.332s
sys 0m34.815s

This resulted in the build failure:

[ 42%] Linking CXX executable xSDKTrilinos_PETSc_Amesos2_example.exe
cd /home/8vt/localhome/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/XSDKTRILINOS_MPI_DEBUG_DEBUG_STATIC/petsc/example && /projects/vera/common_tools/cmake-3.3.2/bin/cmake -E cmake_link_script CMakeFiles/xSDKTrilinos_PETSc_Amesos2_example.dir/link.txt --verbose=1
/projects/vera/gcc-4.8.3/toolset/mpich-3.1.3/bin/mpicxx    -pedantic -Wall -Wno-long-long -Wwrite-strings  -std=c++11 -g -O0   CMakeFiles/xSDKTrilinos_PETSc_Amesos2_example.dir/PETSc_Amesos2Ex.cpp.o  -o xSDKTrilinos_PETSc_Amesos2_example.exe  -L/home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib -rdynamic ../src/libxsdkpetsc.a ../../liblast_lib.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libmuelu-adapters.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libmuelu-interface.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libmuelu.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteko.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libstratimikos.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libstratimikosbelos.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libstratimikosaztecoo.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libstratimikosamesos.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libstratimikosml.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libstratimikosifpack.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libifpack2-adapters.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libifpack2.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libanasazitpetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libModeLaplace.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libanasaziepetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libanasazi.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libamesos2.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libbelostpetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libbelosepetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libbelos.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libml.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libifpack.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libzoltan2.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libamesos.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libgaleri-xpetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libgaleri-epetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libaztecoo.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libisorropia.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libxpetra-sup.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libxpetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libthyratpetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libthyraepetraext.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libthyraepetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libthyracore.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libepetraext.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtpetraext.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtpetrainout.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtpetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libkokkostsqr.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtpetrakernels.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtpetraclassiclinalg.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtpetraclassicnodeapi.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtpetraclassic.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libtriutils.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libzoltan.a -lm /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libepetra.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/librtop.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteuchoskokkoscomm.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteuchoskokkoscompat.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteuchosremainder.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteuchosnumerics.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteuchoscomm.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteuchosparameterlist.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libteuchoscore.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libkokkosalgorithms.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libkokkoscontainers.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libkokkoscore.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libpamgen_extras.a /home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib/libpamgen.a /home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a /home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libHYPRE.a /home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libsuperlu_dist.a /home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libparmetis.a /home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libmetis.a /projects/vera/gcc-4.8.3/tpls/opt_static/lapack-3.3.1/lib/liblapack.a /projects/vera/gcc-4.8.3/tpls/opt_static/lapack-3.3.1/lib/libblas.a -lgomp -Wl,-rpath,/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib64 -lmpifort -lgfortran -lquadmath -Wl,-rpath,/home/8vt/localhome/PROJECTS/Trilinos.base/install/mpi_debug_debug_static/lib 
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xioerr.o): In function `PetscSetXIOErrorHandler':
xioerr.c:(.text+0x21): undefined reference to `XSetIOErrorHandler'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(client.o): In function `PetscSSLInitializeContext':
client.c:(.text+0xa0): undefined reference to `SSLv23_method'
client.c:(.text+0xa8): undefined reference to `SSL_CTX_new'
client.c:(.text+0xbf): undefined reference to `SSL_CTX_ctrl'
client.c:(.text+0x1b1): undefined reference to `SSL_library_init'
client.c:(.text+0x1b6): undefined reference to `SSL_load_error_strings'
client.c:(.text+0x1c4): undefined reference to `BIO_new_fp'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(client.o): In function `PetscSSLDestroyContext':
client.c:(.text+0x255): undefined reference to `SSL_CTX_free'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(client.o): In function `PetscHTTPSRequest':
client.c:(.text+0xe91): undefined reference to `SSL_write'
client.c:(.text+0xe9e): undefined reference to `SSL_get_error'
client.c:(.text+0xf7d): undefined reference to `SSL_read'
client.c:(.text+0xf8d): undefined reference to `SSL_get_error'
client.c:(.text+0x10b7): undefined reference to `SSL_free'
client.c:(.text+0x116a): undefined reference to `SSL_shutdown'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(client.o): In function `PetscHTTPSConnect':
client.c:(.text+0x1791): undefined reference to `SSL_new'
client.c:(.text+0x179e): undefined reference to `BIO_new_socket'
client.c:(.text+0x17ac): undefined reference to `SSL_set_bio'
client.c:(.text+0x17b4): undefined reference to `SSL_connect'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawString_X(_p_PetscDraw*, double, double, int, char const*)':
xops.c:(.text+0x1597): undefined reference to `XSetForeground'
xops.c:(.text+0x1615): undefined reference to `XDrawString'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawPointPixel_X(_p_PetscDraw*, int, int, int)':
xops.c:(.text+0x1928): undefined reference to `XSetForeground'
xops.c:(.text+0x194b): undefined reference to `XDrawPoint'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawLine_X(_p_PetscDraw*, double, double, double, double, int)':
xops.c:(.text+0x1b4c): undefined reference to `XSetForeground'
xops.c:(.text+0x1c57): undefined reference to `XDrawLine'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawRectangle_X(_p_PetscDraw*, double, double, double, double, int, int, int, int)':
xops.c:(.text+0x1e62): undefined reference to `XSetForeground'
xops.c:(.text+0x1f8b): undefined reference to `XFillRectangle'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawEllipse_X(_p_PetscDraw*, double, double, double, double, int)':
xops.c:(.text+0x2188): undefined reference to `XSetForeground'
xops.c:(.text+0x22eb): undefined reference to `XFillArc'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawArrow_X(_p_PetscDraw*, double, double, double, double, int)':
xops.c:(.text+0x24e0): undefined reference to `XSetForeground'
xops.c:(.text+0x25f7): undefined reference to `XDrawLine'
xops.c:(.text+0x26dc): undefined reference to `XDrawLine'
xops.c:(.text+0x2704): undefined reference to `XDrawLine'
xops.c:(.text+0x27d0): undefined reference to `XDrawLine'
xops.c:(.text+0x27f6): undefined reference to `XDrawLine'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o):xops.c:(.text+0x2849): more undefined references to `XDrawLine' follow
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawTriangle_X(_p_PetscDraw*, double, double, double, double, double, double, int, int, int)':
xops.c:(.text+0x2995): undefined reference to `XSetForeground'
xops.c:(.text+0x2b21): undefined reference to `XFillPolygon'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawPoint_X(_p_PetscDraw*, double, double, int)':
xops.c:(.text+0x2f63): undefined reference to `XSetForeground'
xops.c:(.text+0x2f9a): undefined reference to `XDrawPoint'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawStringVertical_X(_p_PetscDraw*, double, double, int, char const*)':
xops.c:(.text+0x3265): undefined reference to `XSetForeground'
xops.c:(.text+0x32ac): undefined reference to `XDrawString'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawSetTitle_X(_p_PetscDraw*, char const*)':
xops.c:(.text+0x44af): undefined reference to `XGetWMName'
xops.c:(.text+0x44bc): undefined reference to `XFree'
xops.c:(.text+0x44ed): undefined reference to `XSetWMName'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawSetViewport_X(_p_PetscDraw*, double, double, double, double)':
xops.c:(.text+0x4d83): undefined reference to `XSetClipRectangles'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawClear_X(_p_PetscDraw*)':
xops.c:(.text+0x5523): undefined reference to `XSync'
xops.c:(.text+0x5ee5): undefined reference to `XSetForeground'
xops.c:(.text+0x5f18): undefined reference to `XFillRectangle'
xops.c:(.text+0x5f22): undefined reference to `XSync'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawFlush_X(_p_PetscDraw*)':
xops.c:(.text+0x69e7): undefined reference to `XSync'
xops.c:(.text+0x7a8e): undefined reference to `XCopyArea'
xops.c:(.text+0x7aa6): undefined reference to `XSync'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawGetMouseButton_X(_p_PetscDraw*, PetscDrawButton*, double*, double*, double*, double*)':
xops.c:(.text+0xba3e): undefined reference to `XCreateFontCursor'
xops.c:(.text+0xba5f): undefined reference to `XDefineCursor'
xops.c:(.text+0xba71): undefined reference to `XSelectInput'
xops.c:(.text+0xba8a): undefined reference to `XCheckTypedEvent'
xops.c:(.text+0xbaa9): undefined reference to `XMaskEvent'
xops.c:(.text+0xbafa): undefined reference to `XQueryPointer'
xops.c:(.text+0xbb4d): undefined reference to `XGetGeometry'
xops.c:(.text+0xbb5c): undefined reference to `XSelectInput'
xops.c:(.text+0xbb69): undefined reference to `XUndefineCursor'
xops.c:(.text+0xbb75): undefined reference to `XFreeCursor'
xops.c:(.text+0xbb80): undefined reference to `XSync'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xops.o): In function `PetscDrawCreate_X':
xops.c:(.text+0xc2f9): undefined reference to `XOpenDisplay'
xops.c:(.text+0xc32e): undefined reference to `XCloseDisplay'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xtext.o): In function `PetscDrawXiFontFixed':
xtext.c:(.text+0x3a3): undefined reference to `XLoadFont'
xtext.c:(.text+0x3b2): undefined reference to `XQueryFont'
xtext.c:(.text+0x3e1): undefined reference to `XFreeFontInfo'
xtext.c:(.text+0x406): undefined reference to `XChangeGC'
xtext.c:(.text+0x6ab): undefined reference to `XListFontsWithInfo'
xtext.c:(.text+0x748): undefined reference to `XFreeFontInfo'
xtext.c:(.text+0x871): undefined reference to `XListFontsWithInfo'
xtext.c:(.text+0x943): undefined reference to `XFreeFontInfo'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xtone.o): In function `PetscDrawInterpolatedTriangle_X':
xtone.c:(.text+0x4b1): undefined reference to `XSetForeground'
xtone.c:(.text+0x4db): undefined reference to `XDrawPoint'
xtone.c:(.text+0x703): undefined reference to `XSetForeground'
xtone.c:(.text+0x72f): undefined reference to `XDrawPoint'
xtone.c:(.text+0x7b1): undefined reference to `XSetForeground'
xtone.c:(.text+0x7d9): undefined reference to `XDrawPoint'
xtone.c:(.text+0x81a): undefined reference to `XSetForeground'
xtone.c:(.text+0x83c): undefined reference to `XDrawPoint'
xtone.c:(.text+0x963): undefined reference to `XSetForeground'
xtone.c:(.text+0x98f): undefined reference to `XDrawPoint'
xtone.c:(.text+0x9ce): undefined reference to `XSetForeground'
xtone.c:(.text+0x9f8): undefined reference to `XDrawPoint'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiDisplayWindow(PetscDraw_X*, char*, int, int, int, int)':
xinit.c:(.text+0x14c): undefined reference to `XGetWindowAttributes'
xinit.c:(.text+0x23e): undefined reference to `XCreateWindow'
xinit.c:(.text+0x276): undefined reference to `XStringListToTextProperty'
xinit.c:(.text+0x29f): undefined reference to `XStringListToTextProperty'
xinit.c:(.text+0x352): undefined reference to `XSetWMProperties'
xinit.c:(.text+0x35c): undefined reference to `XFree'
xinit.c:(.text+0x369): undefined reference to `XFree'
xinit.c:(.text+0x37a): undefined reference to `XSelectInput'
xinit.c:(.text+0x386): undefined reference to `XMapWindow'
xinit.c:(.text+0x403): undefined reference to `XMaskEvent'
xinit.c:(.text+0x4a6): undefined reference to `XSelectInput'
xinit.c:(.text+0x7bb): undefined reference to `XStringListToTextProperty'
xinit.c:(.text+0x7d6): undefined reference to `XStringListToTextProperty'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiClose':
xinit.c:(.text+0xead): undefined reference to `XFreeGC'
xinit.c:(.text+0xeb5): undefined reference to `XCloseDisplay'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiInit':
xinit.c:(.text+0x100d): undefined reference to `XOpenDisplay'
xinit.c:(.text+0x114d): undefined reference to `XCreateGC'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiQuickWindow':
xinit.c:(.text+0x1599): undefined reference to `XSetWindowBackground'
xinit.c:(.text+0x15a5): undefined reference to `XClearWindow'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiQuickWindowFromWindow':
xinit.c:(.text+0x177b): undefined reference to `XGetWindowAttributes'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiQuickPixmap':
xinit.c:(.text+0x1959): undefined reference to `XCreatePixmap'
xinit.c:(.text+0x1979): undefined reference to `XSetForeground'
xinit.c:(.text+0x19a2): undefined reference to `XFillRectangle'
xinit.c:(.text+0x19ac): undefined reference to `XSync'
xinit.c:(.text+0x1a64): undefined reference to `XFreePixmap'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiResizeWindow':
xinit.c:(.text+0x1b29): undefined reference to `XSelectInput'
xinit.c:(.text+0x1b3a): undefined reference to `XResizeWindow'
xinit.c:(.text+0x1b50): undefined reference to `XWindowEvent'
xinit.c:(.text+0x1b5e): undefined reference to `XSelectInput'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xinit.o): In function `PetscDrawXiGetGeometry':
xinit.c:(.text+0x1e43): undefined reference to `XGetGeometry'
xinit.c:(.text+0x1e7f): undefined reference to `XTranslateCoordinates'
xinit.c:(.text+0x1e9b): undefined reference to `XGetWindowAttributes'
xinit.c:(.text+0x1ed0): undefined reference to `XTranslateCoordinates'
xinit.c:(.text+0x1f10): undefined reference to `XGetGeometry'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(ximage.o): In function `PetscDrawGetImage_X':
ximage.c:(.text+0xa17): undefined reference to `XSync'
ximage.c:(.text+0x1136): undefined reference to `XGetGeometry'
ximage.c:(.text+0x1179): undefined reference to `XGetImage'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xcolor.o): In function `PetscDrawSetUpColormap_Shared(_XDisplay*, int, Visual*, unsigned long)':
xcolor.c:(.text+0xec): undefined reference to `XAllocNamedColor'
xcolor.c:(.text+0x1f2): undefined reference to `XAllocColor'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xcolor.o): In function `PetscDrawSetUpColormap_Private(_XDisplay*, int, Visual*, unsigned long)':
xcolor.c:(.text+0x4e8): undefined reference to `XStoreColor'
xcolor.c:(.text+0x533): undefined reference to `XParseColor'
xcolor.c:(.text+0x543): undefined reference to `XAllocColor'
xcolor.c:(.text+0x673): undefined reference to `XAllocColor'
xcolor.c:(.text+0x6b0): undefined reference to `XStoreColor'
xcolor.c:(.text+0x806): undefined reference to `XCreateColormap'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xcolor.o): In function `PetscDrawSetUpColormap_X(_XDisplay*, int, Visual*, unsigned long)':
xcolor.c:(.text+0x99c): undefined reference to `XMatchVisualInfo'
xcolor.c:(.text+0xa95): undefined reference to `XMatchVisualInfo'
xcolor.c:(.text+0xab6): undefined reference to `XMatchVisualInfo'
xcolor.c:(.text+0xad7): undefined reference to `XMatchVisualInfo'
xcolor.c:(.text+0xaf8): undefined reference to `XMatchVisualInfo'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xcolor.o):xcolor.c:(.text+0xb19): more undefined references to `XMatchVisualInfo' follow
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xcolor.o): In function `PetscDrawXiSetColormap(PetscDraw_X*)':
xcolor.c:(.text+0x168d): undefined reference to `XSetWindowColormap'
/home/8vt/localhome/PROJECTS/PETSc.base/install/mpi-debug-static/lib/libpetsc.a(xcolor.o): In function `PetscDrawXiGetPalette(PetscDraw_X*, unsigned char (*) [3])':
xcolor.c:(.text+0x181d): undefined reference to `XQueryColors'
collect2: error: ld returned 1 exit status
make[2]: *** [petsc/example/xSDKTrilinos_PETSc_Amesos2_example.exe] Error 1
make[2]: Leaving directory `/localhome/8vt/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/XSDKTRILINOS_MPI_DEBUG_DEBUG_STATIC'
make[1]: *** [petsc/example/CMakeFiles/xSDKTrilinos_PETSc_Amesos2_example.dir/all] Error 2
make[1]: Leaving directory `/localhome/8vt/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/XSDKTRILINOS_MPI_DEBUG_DEBUG_STATIC'
make: *** [all] Error 2

The listed order of the TPL libraries listed above is libpetsc.a, libHYPRE.a, libsuperlu_dist.a, libparmetis.a, libmetis.a, and liblapack.a. Is that not the currect order?

Looking at the link errors, what library is supposed to provide XSetIOErrorHandler? It looks like this is coming from the X11 library. Does PETSc pick up X11 and use it if it is there? If so, why?

What library provides SSLv23_method? That appears to be the OpenSSL library. What libraries is PETSc getting these from?

... So adding -lx11 and -lssl to the link line seems to solve that link failure. So I think that I just need to add those to the link line and we should be good to go. So I will add:

-DxSDKTrilinos_EXTRA_LINK_FLAGS="-lX11 -lssl -lgomp -Wl,-rpath,/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib64"

I did this in the xSDKTrilinos commit:

3d49a09 "Add -lX11 and -lssl for linking with PETSc (#2)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Wed Apr 13 12:53:14 2016 -0400 (4 minutes ago)

M       scripts/do-configure-fissile4

With that change, all of xSDKTrilinos links and passes all of the tests:

8vt@th232 XSDKTRILINOS_MPI_DEBUG_DEBUG_STATIC]$ ctest -j12
Test project /home/8vt/localhome/PROJECTS/Trilinos.base/BUILDS/GCC-4.8.3/XSDKTRILINOS_MPI_DEBUG_DEBUG_STATIC
      Start  1: xSDKTrilinos_PETScAIJMatrix_MPI_4
      Start  2: xSDKTrilinos_PETSc_Amesos2_example_MPI_4
      Start  3: xSDKTrilinos_PETSc_Anasazi_example_MPI_4
 1/10 Test  #1: xSDKTrilinos_PETScAIJMatrix_MPI_4 ..........   Passed    0.11 sec
 2/10 Test  #2: xSDKTrilinos_PETSc_Amesos2_example_MPI_4 ...   Passed    0.10 sec
      Start  4: xSDKTrilinos_PETSc_Ifpack2_example_MPI_4
      Start  5: xSDKTrilinos_PETSc_MueLu_example_MPI_4
 3/10 Test  #4: xSDKTrilinos_PETSc_Ifpack2_example_MPI_4 ...   Passed    0.20 sec
      Start  6: xSDKTrilinos_example_TpetraKSP_MPI_4
 4/10 Test  #6: xSDKTrilinos_example_TpetraKSP_MPI_4 .......   Passed    0.20 sec
      Start  7: xSDKTrilinos_example_EpetraKSP_MPI_4
 5/10 Test  #5: xSDKTrilinos_PETSc_MueLu_example_MPI_4 .....   Passed    0.57 sec
 6/10 Test  #7: xSDKTrilinos_example_EpetraKSP_MPI_4 .......   Passed    0.16 sec
      Start  8: xSDKTrilinos_HypreTest_MPI_4
      Start  9: xSDKTrilinos_Hypre_Belos_example_MPI_4
 7/10 Test  #8: xSDKTrilinos_HypreTest_MPI_4 ...............   Passed    0.10 sec
 8/10 Test  #9: xSDKTrilinos_Hypre_Belos_example_MPI_4 .....   Passed    0.10 sec
      Start 10: xSDKTrilinos_Hypre_Solve_example_MPI_4
 9/10 Test #10: xSDKTrilinos_Hypre_Solve_example_MPI_4 .....   Passed    0.10 sec
10/10 Test  #3: xSDKTrilinos_PETSc_Anasazi_example_MPI_4 ...   Passed    1.74 sec

100% tests passed, 0 tests failed out of 10

Label Time Summary:
xSDKTrilinos    =   3.39 sec

Total Test time (real) =   1.75 sec

What this means is that I can't find a problem with the link order of the libraries. My guess is that the problem is with how Trilinos and/or xSDKTrilinos are being configured.

BarrySmith commented 8 years ago

Yes --with-netcdf=0 and --with-exodusii=0 will turn off their installs but I am not sure if Trilinos will build on without those packages.

You can email the petsc/configure.log to petsc-maint@mcs.anl.gov for the failed netcdf case so we can try to figure out why it failed to build. We can usually figure out failed builds from configure.log

bartlettroscoe commented 8 years ago

You can email the petsc/configure.log to petsc-maint@mcs.anl.gov for the failed netcdf case so we can try to figure out why it failed to build. We can usually figure out failed builds from configure.log

Barry, I sent two emails to the list petsc-maint@mcs.anl.gov. One for a --download-netcdf failure and one for a --download-boost failure.

Yes --with-netcdf=0 and --with-exodusii=0 will turn off their installs but I am not sure if Trilinos will build on without those packages.

What would be nice is if PETSc would react to --with-netcdf=0 by configuring Trilinos with -DTPL_ENABLE_NetCDF=OFF. If you also configure with -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON, then all of the Trilinos packages that require NetCDF will be disabled automatically (see https://trilinos.org/docs/files/TrilinosBuildReference.html#disabling-support-for-a-third-party-lib).

You can do the same thing with the Boost and BoostLib TPLs too if --with-boost=0 is passed in. That is, if --with-boost=0 is passed in, then configure Trilinos with -DTPL_ENABLE_Boost=OFF, -DTPL_ENABLE_BoostLib=OFF, and -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON.

The issue is that we not need NetCDF or Boost in order to build and test xSDKTrilinos. We don't want to present Trilinos as one huge glob of software that is take it or leave it (because people will leave it if they run into these problems). That is why Trilinos is partitioned into a bunch of smaller package with carefully managed dependencies.

Does that seem reasonable?

Also, why is PETSc building with external ExodusII? Exodus is actually built as part of the of the Trilinos package SEACAS. Does that mean the two version of Exodus are being installed?

balay commented 8 years ago

For one - netcdf is already listed as optional for trilinos.

I can switch boost to be optional. per the above option.

you've listed -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON with both netcdf and boost. What happens if this flag is always set? [irrespective of enabling/disabling netcdf/boost] i.e is it ok to always set this flag?

BarrySmith commented 8 years ago

Roscoe,

Thanks for letting us know about these options. I am adding a bunch of them and testing them now. This definitely will make live easier.

Satish,

 Don't make any changes now since I am changing them.

Barry

On Apr 13, 2016, at 3:59 PM, Satish Balay notifications@github.com wrote:

For one - netcdf is already listed as optional for trilinos.

I can switch boost to be optional. per the above option.

you've listed -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON with both netcdf and boost. What happens if this flag is always set? [irrespective of enabling/disabling netcdf/boost]

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

BarrySmith commented 8 years ago

Also, why is PETSc building with external ExodusII? Exodus is actually built as part of the of the Trilinos package SEACAS. Does that mean the two version of Exodus are being installed?

We have used Exodusii directly for many years.

If the Exodusii in SEACAS is identical to the independent Exodusii I can add some code to insure only one is included.

Barry

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

bartlettroscoe commented 8 years ago

you've listed -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON with both netcdf and boost. What happens if this flag is always set? [irrespective of enabling/disabling netcdf/boost] i.e is it ok to always set this flag?

Yes, it is fine to set this always, unconditionally. This just determines what happens when you have inconsistent enables and disables as described here. In my opinion, I think that this should be on by default for all TriBITS projects, and it is with CASL VERA. It just massively simplifies dependency management with very large and complex dependency structures. It was just that some Trilinos users got confused by this behavior and wanted the default to error out, which is fine.

BarrySmith commented 8 years ago

Ross,

I've updated the Trilinos.py to support not requiring all of these packages.

Barry

On Apr 13, 2016, at 2:10 PM, Roscoe A. Bartlett notifications@github.com wrote:

You can email the petsc/configure.log to petsc-maint@mcs.anl.gov for the failed netcdf case so we can try to figure out why it failed to build. We can usually figure out failed builds from configure.log

Barry, I sent two emails to the list petsc-maint@mcs.anl.gov. One for a --download-netcdf failure and one for a --download-boost failure.

Yes --with-netcdf=0 and --with-exodusii=0 will turn off their installs but I am not sure if Trilinos will build on without those packages.

What would be nice is if PETSc would react to --with-netcdf=0 by configuring Trilinos with -DTPL_ENABLE_NetCDF=OFF. If you also configure with -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON, then all of the Trilinos packages that require NetCDF will be disabled automatically (see https://trilinos.org/docs/files/TrilinosBuildReference.html#disabling-support-for-a-third-party-lib).

You can do the same thing with the Boost and BoostLib TPLs too if --with-boost=0 is passed in. That is, if --with-boost=0 is passed in, then configure Trilinos with -DTPL_ENABLE_Boost=OFF, -DTPL_ENABLE_BoostLib=OFF, and -DTrilinos_DISABLE_ENABLED_FORWARD_DEP_PACKAGES=ON.

The issue is that we not need NetCDF or Boost in order to build and test xSDKTrilinos. We don't want to present Trilinos as one huge glob of software that is take it or leave it (because people will leave it if they run into these problems). That is why Trilinos is partitioned into a bunch of smaller package with carefully managed dependencies.

Does that seem reasonable?

Also, why is PETSc building with external ExodusII? Exodus is actually built as part of the of the Trilinos package SEACAS. Does that mean the two version of Exodus are being installed?

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

bartlettroscoe commented 8 years ago

I've updated the Trilinos.py to support not requiring all of these packages.

Thanks! I will give this a try right away!

bartlettroscoe commented 8 years ago

Barry,

I have reproduced the link failures with xSDKTrilinos through installxSDK.sh. I believe that I know how to resolve the ordering issues by just making changes to xSDKTrilinos. However, there are a few things that will need to be changed on the PETSc side to get xSDKTrilinos to work with static libraries for the way the installxSDK.sh/PETSc is configuring and building everything.

First, when PETSc builds ParMETIS against METIS (i.e. --download-parmetis --download-metis), then PETSc needs to pass in the link info for METIS along with ParMETIS. If PETSc were to enable any of the Trilinos tests or examples that depend (directly or indirectly) on ParMETIS, then you would see link failures just in Trilinos with static libs. One way to pass in METIS link info is to just pass it in through TPL_ParMETIS_LIBRARIES (i.e. add -L<path> -lmetis). The other way is to enable the Trilinos "METIS" TPL and pass in the link info. That is, set -DTPL_ENABLE_METIS=ON, -DTPL_METIS_LIBRARIES="..." and -DTPL_METIS_INCLUDE_DIRS="..." like is done for ParMETIS. That will ensure that the METIS library will be listed on the link line along with the ParMETIS TPL (but get listed after the ParMETIS lib). I will test this to be sure in my own local source and build dirs for PETSc, Trilinos, and xSDKTrilinos but I am about 99% sure that this will solve that problem.

Second, since PETSc is building against the X11 and SSL libraries on my system (and likely other systems), it needs to specify these system libraries as well in order to work with static libs. Currently, PETSc is configuring xSDKTrilinos with:

-DTPL_ENABLE_PETSC=ON \
-DPETSC_LIBRARY_DIRS=<petsc-install-prefix>/lib \
-DPETSC_INCLUDE_DIRS=<petsc-install-prefix>/include \

By default, this will only result in the CMake/TriBITS configure of xSDKTrilinos finding libpetsc.a (when static libs are used). The PETSC configure of xsDKTrilinos can resolve this in one of two ways. Either PETSc can pass:

-DPETSC_LIBRARY_NAMES="petsc;X11;ssl" \

or can pass these extra libraries (since they are system libraries) as:

-DxSDKTrilinos_EXTRA_LINK_FLAGS="-lX11 -lssl ..."

Passing common system libraries through -D<Project>_EXTRA_LINK_FLAGS is generally how we resolve these dependencies (commonly with -ldl, -lgfortran, -lomp, etc.).

Of course, PETSc will need to pass all of its linked against system libraries in this way.

I will make the necessary changes to xSDKTrilinos on my side to fix static libraries (and verify it works with the PETSc way of configuring Trilinos and xSDKTrilinos) and push to the 'master' branch in the xSDKTrilinos github repo. After that, if someone on the PETSc side could take care of passing in the METIS info to Trilinos and the system libraries for PETSc to xSDKTrilinos, then I think we will have the static library build of xSDK fixed!

balay commented 8 years ago

On Thu, 14 Apr 2016, Roscoe A. Bartlett wrote:

Barry,

I have reproduced the link failures with xSDKTrilinos through installxSDK.sh. I believe that I know how to resolve the ordering issues by just making changes to xSDKTrilinos. However, there are a few things that will need to be changed on the PETSc side to get xSDKTrilinos to work with static libraries for the way the installxSDK.sh/PETSc is configuring and building everything.

First, when PETSc builds ParMETIS against METIS (i.e. --download-parmetis --download-metis), then PETSc needs to pass in the link info for METIS along with ParMETIS. If PETSc were to enable any of the Trilinos tests or examples that depend (directly or indirectly) on ParMETIS, then you would see link failures just in Trilinos with static libs. One way to pass in METIS link info is to just pass it in through TPL_ParMETIS_LIBRARIES (i.e. add -L<path> -lmetis). The other way is to enable the Trilinos "METIS" TPL and pass in the link info. That is, set -DTPL_ENABLE_METIS=ON, -DTPL_METIS_LIBRARIES="..." and -DTPL_METIS_INCLUDE_DIRS="..." like is done for ParMETIS. That will ensure that the METIS library will be listed on the link line along with the ParMETIS TPL (but get listed after the ParMETIS lib). I will test this to be sure in my own local source and build dirs for PETSc, Trilinos, and xSDKTrilinos but I am about 99% sure that this will solve that problem.

Second, since PETSc is building against the X11 and SSL libraries on my system (and likely other systems), it needs to specify these system libraries as well in order to work with static libs. Currently, PETSc is configuring xSDKTrilinos with:

-DTPL_ENABLE_PETSC=ON \
-DPETSC_LIBRARY_DIRS=<petsc-install-prefix>/lib \
-DPETSC_INCLUDE_DIRS=<petsc-install-prefix>/include \

By default, this will only result in the CMake/TriBITS configure of xSDKTrilinos finding libpetsc.a (when static libs are used). The PETSC configure of xsDKTrilinos can resolve this in one of two ways. Either PETSc can pass:

-DPETSC_LIBRARY_NAMES="petsc;X11;ssl" \

or can pass these extra libraries (since they are system libraries) as:

-DxSDKTrilinos_EXTRA_LINK_FLAGS="-lX11 -lssl ..."

Passing common system libraries through -D<Project>_EXTRA_LINK_FLAGS is generally how we resolve these dependencies (commonly with -ldl, -lgfortran, -lomp, etc.).

Of course, PETSc will need to pass all of its linked against system libraries in this way.

Perhaps petsc and all dependent libraries aka PETSC_LIB should be passed in via -DxSDKTrilinos_EXTRA_LINK_FLAGS [including trilinos, metis libs etc].

There is no reason there should be 2 code paths that resolve package libraries and dependencies

Satish

I will make the necessary changes to xSDKTrilinos on my side to fix static libraries (and verify it works with the PETSc way of configuring Trilinos and xSDKTrilinos) and push to the 'master' branch in the xSDKTrilinos github repo. After that, if someone on the PETSc side could take care of passing in the METIS info to Trilinos and the system libraries for PETSc to xSDKTrilinos, then I think we will have the static library build of xSDK fixed!


You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/trilinos/xSDKTrilinos/issues/2#issuecomment-210133417

bartlettroscoe commented 8 years ago

Barry,

A few other things that I noticed when looking into this:

bartlettroscoe commented 8 years ago

Perhaps petsc and all dependent libraries aka PETSC_LIB should be passed in via -DxSDKTrilinos_EXTRA_LINK_FLAGS [including trilinos, metis libs etc]. There is no reason there should be 2 code paths that resolve package libraries and dependencies

Yes, but you have huge duplication if you list the same system libraries over and over again for each TPL separately. It is much cleaner, IMHO, to pass universal system libraries and things like RPATH to the GCC libraries through -D<Project>_EXTRA_LINK_FLAGS. Just my opinion.

balay commented 8 years ago

On Thu, 14 Apr 2016, Roscoe A. Bartlett wrote:

Perhaps petsc and all dependent libraries aka PETSC_LIB should be passed in via -DxSDKTrilinos_EXTRA_LINK_FLAGS [including trilinos, metis libs etc]. There is no reason there should be 2 code paths that resolve package libraries and dependencies

Yes, but you have huge duplication if you list the same system libraries over and over again for each TPL separately. It is much cleaner, IMHO, to pass universal system libraries and things like RPATH to the GCC libraries through -D<Project>_EXTRA_LINK_FLAGS. Just my opinion.

Any given package should support the following configure modes [perhaps in autoconf configure terms]: a) --with-pkg1-dir=PKG1DIR --with-pkg2-dir=PKG2DIR b) --with-pkg1-include=PKG1DIR/include --with-pkg1-lib="-LPKG1DIR/lib -lpkg1" --with-pkg2-include=PKG2DIR/include --with-pkg2lib="-LPKG2DIR/lib -lpkg2" c) --enable-pkg1 --enable-pkg2 CFLAGS=-IPKG1DIR/include -IPKG2DIR/include" [LIBS? or]LDFLAGS="-LPKG1DIR/lib -lpkg1 -LPKG2DIR/lib -lpkg2"

Normally users would be able to use any of these 3 modes. But in xsdk case - with a configure wrapper over another configure - we still have the option of using all 3 modes.

currently xSDKTrilinos.py appears to use mode (a) - which asks xSDKTrilinos to detect libraries & dependencies.

With (b) - the libraries detected by xsdk wrapper could be passed to xSDKTrilinos - and avoid mismatch in this library detection.

With (c) - both the library detection and dependency info thats already done in xsdk wrapper and it can be passed to xSDKTrilinos Note: There would be no duplicate library list in this case.

Satish

BarrySmith commented 8 years ago

Ross,

Thanks for all your suggestions. We'll get started on them.

Satish,

 Have you started on trying these changes or do you want me to do it? Doesn't make sense for us both be changing the same things.

Barry

On Apr 14, 2016, at 3:23 PM, Roscoe A. Bartlett notifications@github.com wrote:

Barry,

I have reproduced the link failures with xSDKTrilinos through installxSDK.sh. I believe that I know how to resolve the ordering issues by just making changes to xSDKTrilinos. However, there are a few things that will need to be changed on the PETSc side to get xSDKTrilinos to work with static libraries for the way the installxSDK.sh/PETSc is configuring and building everything.

First, when PETSc builds ParMETIS against METIS (i.e. --download-parmetis --download-metis), then PETSc needs to pass in the link info for METIS along with ParMETIS. If PETSc were to enable any of the Trilinos tests or examples that depend (directly or indirectly) on ParMETIS, then you would see link failures just in Trilinos with static libs. One way to pass in METIS link info is to just pass it in through TPL_ParMETIS_LIBRARIES (i.e. add -L -lmetis). The other way is to enable the Trilinos "METIS" TPL and pass in the link info. That is, set -DTPL_ENABLE_METIS=ON, -DTPL_METIS_LIBRARIES="..." and -DTPL_METIS_INCLUDE_DIRS="..." like is done for ParMETIS. That will ensure that the METIS library will be listed on the link line along with the ParMETIS TPL (but get listed after the ParMETIS lib). I will test this to be sure in my own local source and build dirs for PETSc, Trilinos, and xSDKTrilinos b ut I am about 99% sure that this will solve that problem.

Second, since PETSc is building against the X11 and SSL libraries on my system (and likely other systems), it needs to specify these system libraries as well in order to work with static libs. Currently, PETSc is configuring xSDKTrilinos with:

-DTPL_ENABLE_PETSC=ON \ -DPETSC_LIBRARY_DIRS=/lib \ -DPETSC_INCLUDE_DIRS=/include \

By default, this will only result in the CMake/TriBITS configure of xSDKTrilinos finding libpetsc.a (when static libs are used). The PETSC configure of xsDKTrilinos can resolve this in one of two ways. Either PETSc can pass:

-DPETSC_LIBRARY_NAMES="petsc;X11;ssl" \

or can pass these extra libraries (since they are system libraries) as:

-DxSDKTrilinos_EXTRA_LINK_FLAGS="-lX11 -lssl ..."

Passing common system libraries through -D_EXTRA_LINK_FLAGS is generally how we resolve these dependencies (commonly with -ldl, -lgfortran, -lomp, etc.).

Of course, PETSc will need to pass all of its linked against system libraries in this way.

I will make the necessary changes to xSDKTrilinos on my side to fix static libraries (and verify it works with the PETSc way of configuring Trilinos and xSDKTrilinos) and push to the 'master' branch in the xSDKTrilinos github repo. After that, if someone on the PETSc side could take care of passing in the METIS info to Trilinos and the system libraries for PETSc to xSDKTrilinos, then I think we will have the static library build of xSDK fixed!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

BarrySmith commented 8 years ago

Ross,

   If we very carefully pass all the external package library information to Trilinos cmake (such as Metis, HDF5, ....) in the appropriate TPL variables will xSDKTrilinos cmake be able to automatically reuse that information? Or will we need to pass it all again to xSDKTrilinos cmake  and can/should  we pass it again in the individual TPL variables or do we have to pass it all in xSDKTrilinos_EXTRA_LINK_FLAGS variable?

Thanks

Barry

It seems if we pass it again then there will be tons of overlinking where the same library appears a bunch of times in the same link line; that can work but is ugly.

On Apr 14, 2016, at 3:23 PM, Roscoe A. Bartlett notifications@github.com wrote:

Barry,

I have reproduced the link failures with xSDKTrilinos through installxSDK.sh. I believe that I know how to resolve the ordering issues by just making changes to xSDKTrilinos. However, there are a few things that will need to be changed on the PETSc side to get xSDKTrilinos to work with static libraries for the way the installxSDK.sh/PETSc is configuring and building everything.

First, when PETSc builds ParMETIS against METIS (i.e. --download-parmetis --download-metis), then PETSc needs to pass in the link info for METIS along with ParMETIS. If PETSc were to enable any of the Trilinos tests or examples that depend (directly or indirectly) on ParMETIS, then you would see link failures just in Trilinos with static libs. One way to pass in METIS link info is to just pass it in through TPL_ParMETIS_LIBRARIES (i.e. add -L -lmetis). The other way is to enable the Trilinos "METIS" TPL and pass in the link info. That is, set -DTPL_ENABLE_METIS=ON, -DTPL_METIS_LIBRARIES="..." and -DTPL_METIS_INCLUDE_DIRS="..." like is done for ParMETIS. That will ensure that the METIS library will be listed on the link line along with the ParMETIS TPL (but get listed after the ParMETIS lib). I will test this to be sure in my own local source and build dirs for PETSc, Trilinos, and xSDKTrilinos b ut I am about 99% sure that this will solve that problem.

Second, since PETSc is building against the X11 and SSL libraries on my system (and likely other systems), it needs to specify these system libraries as well in order to work with static libs. Currently, PETSc is configuring xSDKTrilinos with:

-DTPL_ENABLE_PETSC=ON \ -DPETSC_LIBRARY_DIRS=/lib \ -DPETSC_INCLUDE_DIRS=/include \

By default, this will only result in the CMake/TriBITS configure of xSDKTrilinos finding libpetsc.a (when static libs are used). The PETSC configure of xsDKTrilinos can resolve this in one of two ways. Either PETSc can pass:

-DPETSC_LIBRARY_NAMES="petsc;X11;ssl" \

or can pass these extra libraries (since they are system libraries) as:

-DxSDKTrilinos_EXTRA_LINK_FLAGS="-lX11 -lssl ..."

Passing common system libraries through -D_EXTRA_LINK_FLAGS is generally how we resolve these dependencies (commonly with -ldl, -lgfortran, -lomp, etc.).

Of course, PETSc will need to pass all of its linked against system libraries in this way.

I will make the necessary changes to xSDKTrilinos on my side to fix static libraries (and verify it works with the PETSc way of configuring Trilinos and xSDKTrilinos) and push to the 'master' branch in the xSDKTrilinos github repo. After that, if someone on the PETSc side could take care of passing in the METIS info to Trilinos and the system libraries for PETSc to xSDKTrilinos, then I think we will have the static library build of xSDK fixed!

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

bartlettroscoe commented 8 years ago

If we very carefully pass all the external package library information to Trilinos cmake (such as Metis, HDF5, ....) in the appropriate TPL variables will xSDKTrilinos cmake be able to automatically reuse that information?

It should reuse it and it does from what I have seen today. IT is just that the librareis were included in the wrong order (which I am fixing now). PETSc should only need to pass unique info to xSDKTrilinos for PETSc and HYPRE. All of the static libs that are already being used by Trilinos packages like ParMETIS, SuperLUDist, etc. will already be listed on the link line for xSDKTrilinos.

BTW, it looks like CMake is cleaning up all the duplicate libraries, link directories etc. Therefore, it does not impact the final compile and link lines that the compiler and linker sees (one of the built-in features of CMake).

BarrySmith commented 8 years ago

On Apr 14, 2016, at 6:25 PM, Roscoe A. Bartlett notifications@github.com wrote:

If we very carefully pass all the external package library information to Trilinos cmake (such as Metis, HDF5, ....) in the appropriate TPL variables will xSDKTrilinos cmake be able to automatically reuse that information?

It should reuse it and it does from what I have seen today. IT is just that the librareis were included in the wrong order (which I am fixing now). PETSc should only need to pass unique info to xSDKTrilinos for PETSc and HYPRE. All of the static libs that are already being used by Trilinos packages like ParMETIS, SuperLUDist, etc. will already be listed on the link line for xSDKTrilinos.

Ah, then we should be able to do something pretty clean.

Thanks Barry

BTW, it looks like CMake is cleaning up all the duplicate libraries, link directories etc. Therefore, it does not impact the final compile and link lines that the compiler and linker sees (one of the built-in features of CMake).

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

bartlettroscoe commented 8 years ago

Barry and Satish,

I just pushed the commits:

3c80b11 "Read compilers, compile flags, etc from TrilinosConfig.cmake (xSDKTrilinos #2)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Thu Apr 14 20:12:31 2016 -0400 (5 minutes ago)

M       CMakeLists.txt
M       scripts/do-configure-trilinos-fissile4
A       scripts/do-configure-universal

c3f9cbe "Split TrilinosTpl into Pkgs and Tpls TPL to fix static libs (xSDKTrilinos #2)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Thu Apr 14 19:54:07 2016 -0400 (23 minutes ago)

M       CMakeLists.txt
M       TPLsList.cmake
M       cmake/Dependencies.cmake
A       cmake/tpls/FindTPLTrilinosPkgsTpl.cmake
D       cmake/tpls/FindTPLTrilinosTpl.cmake
A       cmake/tpls/FindTPLTrilinosTplsTpl.cmake

to the 'master' branch of the xSDKTrilinos GitHub repo git@github.com:trilinos/xSDKTrilinos.git.

Using this version of xSDKTrilinos, I was was able to run the PETSc generated configure command and then build. It showed the libraries in the correct order. It failed to build because of the missing -lmetis -lX11 -lssl. I then manually configured again hacking on:

-DxSDKTrilinos_EXTRA_LINK_FLAGS="-lmetis -lX11 -lssl -lgomp -Wl,-rpath,/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib64"

and I was able to get all of xSDKTrilinos to link and pass the tests.

See details below.

Now, if PETSc can take care of -lmetis and -lX11 -lssl, then installxSDK.sh should be able to build xSDKTrilinos with all static libs.


Detailed Notes:

Fixing the issues with xSDKTrilinos ...

1) Split the TrilinosTpl TPL into TrilinosTplsTpl and TrilinosPkgsTpl TPLs and order these TrilinosTplsTpl, HYPRE, TrilinosPkgsTpl, PETSC (because PETSc may be statically linked against some Trilinos packages; test this to verify this is true):

I made the local commits:

3c80b11 "Read compilers, compile flags, etc from TrilinosConfig.cmake (xSDKTrilinos #2)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Thu Apr 14 20:12:31 2016 -0400 (5 minutes ago)

M       CMakeLists.txt
M       scripts/do-configure-trilinos-fissile4
A       scripts/do-configure-universal

c3f9cbe "Split TrilinosTpl into Pkgs and Tpls TPL to fix static libs (xSDKTrilinos #2)"
Author: Roscoe A. Bartlett <rabartl@sandia.gov>
Date:   Thu Apr 14 19:54:07 2016 -0400 (23 minutes ago)

M       CMakeLists.txt
M       TPLsList.cmake
M       cmake/Dependencies.cmake
A       cmake/tpls/FindTPLTrilinosPkgsTpl.cmake
D       cmake/tpls/FindTPLTrilinosTpl.cmake
A       cmake/tpls/FindTPLTrilinosTplsTpl.cmake

2) Move changes for xSDKTrilinos over to the installxSDK.sh git source dir for xSDKTrilinos and test over there with the PETSc configure (manually run it adding in -lX11 -lssl):

Now to just try this out manually in the installxSDK.sh directory by getting the branch:

$ cd /localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/arch-linux2-c-debug/externalpackages/git.xsdktrilinos/

$ git remote add local \
  /home/8vt/localhome/PROJECTS/Trilinos.base/Trilinos/packages/xSDKTrilinos

$ git fetch local

$ git checkout -b fix-static local/master

Now to run the configure command generated by PETSc gotten from the petsc/configure.log file:

$ cd /localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/arch-linux2-c-debug/externalpackages/git.xsdktrilinos/build
$ /projects/vera/common_tools/cmake-3.3.2/bin/cmake .. \
-DCMAKE_INSTALL_PREFIX=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static \
-DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="mpicc" -DCMAKE_AR=/usr/bin/ar \
-DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-g3" -DCMAKE_CXX_COMPILER="mpicxx" \
-DCMAKE_CXX_FLAGS:STRING="-g -std=c++11" -DCMAKE_Fortran_COMPILER="mpif90" \
-DCMAKE_Fortran_FLAGS:STRING="-ffree-line-length-0 -g" -DBUILD_SHARED_LIBS=off \
-DUSE_XSDK_DEFAULTS=YES \
-DTRILINOS_INSTALL_DIR=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static \
-DTrilinos_INSTALL_DIR=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static \
-DTPL_ENABLE_HYPRE=ON \
-DTPL_HYPRE_LIBRARIES="-Wl,-rpath,/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static/lib -L/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static/lib -lHYPRE -L/projects/vera/gcc-4.8.3/toolset/mpich-3.1.3/lib -L/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib64 -L/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib -Wl,-rpath,/projects/vera/gcc-4.8.3/toolset/mpich-3.1.3/lib -lmpicxx -lstdc++" \
-DTPL_HYPRE_INCLUDE_DIRS=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static/include \
-DTPL_ENABLE_PETSC=ON \
-DPETSC_LIBRARY_DIRS=/home/8vt/localhome/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/lib \
-DPETSC_INCLUDE_DIRS=/home/8vt/localhome/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/include \
-DxSDKTrilinos_ENABLE_TESTS=ON

Now I get link failures like:

/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static/lib/libparmetis.a(move.c.o): In function `libparmetis__ProjectInfoBack':
move.c:(.text+0x136f): undefined reference to `gk_mcorePush'
move.c:(.text+0x13bd): undefined reference to `libmetis__iset'

I am going to cheat a little and add -lmetis to the extra link flags to make this link ...

$ cd /localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/arch-linux2-c-debug/externalpackages/git.xsdktrilinos/build
$ /projects/vera/common_tools/cmake-3.3.2/bin/cmake .. \
-DCMAKE_INSTALL_PREFIX=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static \
-DCMAKE_VERBOSE_MAKEFILE=1 -DCMAKE_C_COMPILER="mpicc" -DCMAKE_AR=/usr/bin/ar \
-DCMAKE_RANLIB=/usr/bin/ranlib -DCMAKE_C_FLAGS:STRING="-g3" -DCMAKE_CXX_COMPILER="mpicxx" \
-DCMAKE_CXX_FLAGS:STRING="-g -std=c++11" -DCMAKE_Fortran_COMPILER="mpif90" \
-DCMAKE_Fortran_FLAGS:STRING="-ffree-line-length-0 -g" -DBUILD_SHARED_LIBS=off \
-DUSE_XSDK_DEFAULTS=YES \
-DTRILINOS_INSTALL_DIR=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static \
-DTrilinos_INSTALL_DIR=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static \
-DTPL_ENABLE_HYPRE=ON \
-DTPL_HYPRE_LIBRARIES="-Wl,-rpath,/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static/lib -L/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static/lib -lHYPRE -L/projects/vera/gcc-4.8.3/toolset/mpich-3.1.3/lib -L/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib/gcc/x86_64-unknown-linux-gnu/4.8.3 -L/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib64 -L/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib -Wl,-rpath,/projects/vera/gcc-4.8.3/toolset/mpich-3.1.3/lib -lmpicxx -lstdc++" \
-DTPL_HYPRE_INCLUDE_DIRS=/localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/install/mpi_debug_static/include \
-DTPL_ENABLE_PETSC=ON \
-DPETSC_LIBRARY_DIRS=/home/8vt/localhome/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/lib \
-DPETSC_INCLUDE_DIRS=/home/8vt/localhome/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/include \
-DxSDKTrilinos_ENABLE_TESTS=ON \
-DxSDKTrilinos_EXTRA_LINK_FLAGS="-lmetis -lX11 -lssl -lgomp -Wl,-rpath,/projects/vera/gcc-4.8.3/toolset/gcc-4.8.3/lib64"

After configuring with that, I was able to build and run the xSDKTrilinos tests:

$ ctest -j12

Test project /localhome/8vt/PROJECTS/xSDK.base/installxSDK/xsdk/petsc/arch-linux2-c-debug/externalpackages/git.xsdktrilinos/build
      Start  1: xSDKTrilinos_PETScAIJMatrix
      Start  2: xSDKTrilinos_PETSc_Amesos2_example
      Start  3: xSDKTrilinos_PETSc_Anasazi_example
      Start  4: xSDKTrilinos_PETSc_Ifpack2_example
      Start  5: xSDKTrilinos_PETSc_MueLu_example
      Start  6: xSDKTrilinos_example_TpetraKSP
      Start  7: xSDKTrilinos_example_EpetraKSP
      Start  8: xSDKTrilinos_HypreTest
      Start  9: xSDKTrilinos_Hypre_Belos_example
      Start 10: xSDKTrilinos_Hypre_Solve_example
 1/10 Test  #4: xSDKTrilinos_PETSc_Ifpack2_example ...   Passed    0.11 sec
 2/10 Test  #6: xSDKTrilinos_example_TpetraKSP .......   Passed    0.10 sec
 3/10 Test  #7: xSDKTrilinos_example_EpetraKSP .......   Passed    0.10 sec
 4/10 Test  #8: xSDKTrilinos_HypreTest ...............   Passed    0.17 sec
 5/10 Test  #9: xSDKTrilinos_Hypre_Belos_example .....   Passed    0.17 sec
 6/10 Test #10: xSDKTrilinos_Hypre_Solve_example .....   Passed    0.17 sec
 7/10 Test  #2: xSDKTrilinos_PETSc_Amesos2_example ...   Passed    0.28 sec
 8/10 Test  #1: xSDKTrilinos_PETScAIJMatrix ..........   Passed    0.67 sec
 9/10 Test  #5: xSDKTrilinos_PETSc_MueLu_example .....   Passed    0.77 sec
10/10 Test  #3: xSDKTrilinos_PETSc_Anasazi_example ...   Passed    2.55 sec

100% tests passed, 0 tests failed out of 10

Label Time Summary:
xSDKTrilinos    =   5.09 sec

Total Test time (real) =   2.95 sec

So once I push these changes and PETSc passes -lmetis -lX11 and -lssl, then this should work for static libraries, as well as for shared libraries.

bartlettroscoe commented 8 years ago

Satish,

c) --enable-pkg1 --enable-pkg2 CFLAGS=-IPKG1DIR/include -IPKG2DIR/include" [LIBS? or]LDFLAGS="-LPKG1DIR/lib -lpkg1 -LPKG2DIR/lib -lpkg2"

Actually, you can't use that approach in general because some of TPL libraries may be incompatible. For example, for several years in CASL VERA, we had to support two versions of PETSc and Trilinos at the same time. An older version of PETSc built with ML (from a very old version of Trilinos) was being used by the Hydra-TH code but a newer version of PETSc was being used by the core simulator codes COBRA-TF and MPACT and a very new version of Trilinos (integrated with repo syncs) was being used by the Exnihilo codes. This was all done under one cmake configure for "VERA" which make it easy to run automated builds for testing, posting to CDash, doing deployments, etc. This was no problem because these two versions of PETSc and Trilinos never got linked into the same libs or executables. Therefore, a general meta-build system (like TriBITS is almost) can't assume that all the libraries can just be lumped together. It just happened that TriBITS already supported that use case well.

Because if this, it made sense to split out the unique libraries into different TriBITS TPLs and then put all of the common system libraries into a single variable to avoid duplication.

Does that make sense?

bartlettroscoe commented 8 years ago

a) --with-pkg1-dir=PKG1DIR --with-pkg2-dir=PKG2DIR

BTW, TriBITS is currently missing this mode, but it will be added at some point soon:

https://docs.google.com/document/d/1WKs8rJhI3037yKGnEVMhIx9dPN7a7uFRM5isdNAhAXI/edit#bookmark=id.ns1z60jetuc

It has been requested a couple of times.

balay commented 8 years ago

On Thu, 14 Apr 2016, Roscoe A. Bartlett wrote:

Satish,

c) --enable-pkg1 --enable-pkg2 CFLAGS=-IPKG1DIR/include -IPKG2DIR/include" [LIBS? or]LDFLAGS="-LPKG1DIR/lib -lpkg1 -LPKG2DIR/lib -lpkg2"

Actually, you can't use that approach in general because some of TPL libraries may be incompatible. For example, for several years in CASL VERA, we had to support two versions of PETSc and Trilinos at the same time. An older version of PETSc built with ML (from a very old version of Trilinos) was being used by the Hydra-TH code but a newer version of PETSc was being used by the core simulator codes COBRA-TF and MPACT and a very new version of Trilinos (integrated with repo syncs) was being used by the Exnihilo codes. This was all done under one cmake configure for "VERA" which make it easy to run automated builds for testing, posting to CDash, doing deployments, etc. This was no problem because these two versions of PETSc and Trilinos never got linked into the same libs or executables. Therefore, a general meta-build system (like TriBITS is almost) can't assume that all the libraries can just be lumped together. It just happened that TriBITS already supported that use c

ase well .

Because if this, it made sense to split out the unique libraries into different TriBITS TPLs and then put all of the common system libraries into a single variable to avoid duplication.

Does that make sense?

Perhaps I'm misreading the above example. Is your suggestion that the above example can be avoided by using interfaces (a),(b) - but not (c)? In the examples I provided - all have single version of PKG1 and PKG2.

Yes - its possible to abuse and sneek in multiple versions of packages [either through the configure interface or through autodetection configure code in some of the packages] but thats possible with all 3 modes listed. [not just 'c']

I agree linking multiple versions of packages is not desired. And these can sneak up in prebuilt packages. This can happen with all 3 modes listed. If any checks can be added to detect/avoid this issue - all 3 modes will equally benefit from it.

The premise I initially proposed is - if a top level configure script is processing package installation, dependencies, and liblist - it should tell each package it installs exactly what it should use - and not let it autodetect or autoenable things that can conflict with what the top level script figured out already..

Satish


You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/trilinos/xSDKTrilinos/issues/2#issuecomment-210228854

bartlettroscoe commented 8 years ago

Perhaps I'm misreading the above example. Is your suggestion that the above example can be avoided by using interfaces (a),(b) - but not (c)? In the examples I provided - all have single version of PKG1 and PKG2.

In the case of CASL VERA, the older PETSc and Trilinos TPL was named "HydraTHTPLs" so to TriBITS it appeared to be a completely separate TPL.

But you can also have cases where two different TPLs have clashing header files and/or libraries. That is the case right now, for example, with the FEI in Trilinos and HYPRE (see SDK-54). If the include dirs (and perhaps the libraries) for HYPRE were added globally, you could never build the version of FEI actually in Trilinos.

Just making the point that in a more general meta-build env, you can have some code that is not well namespaced and you need mechanisms to avoid clashes. That was only point in arguing against option-c.

it should tell each package it installs exactly what it should use - and not let it autodetect or autoenable things that can conflict with what the top level script figured out already.

Agreed. Autodetection and autoenable is about the number one reason why meta-builds of different software using a heterogeneous set of build systems are not portable. For example, PETSc found X11 and SSL on my system and decided to build against it. Then the link failed in xSDKTrilinos because it was not being passed in. I am not picking on PETSc, there are examples in Trilinos that do the same thing. The issue is that when you have a bunch of different builds systems all making their own decisions, then porting becomes much more difficult because of autodetection and autoenable. This also was/is a problem with libmesh/MOOSE inside of CASL VERA. On a new system the autotools libmesh system suddenly found something new on some other system (TBB in one case I can think of) and decided to use it and it killed the build.

sarich commented 8 years ago

Update on mira:

The xSDKTrilinos test problems are getting built now, but executing them fails. I'm assuming that I can run a test with: qsub -A OSCon -t 20 -n 16 xSDKTrilinos_PETScAIJMatrix.exe

For output I'm getting: Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 0 of 16 (0,0,0,0,0,0) R2F-M0-N00-J00 and rank 0! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 4 of 16 (0,0,0,2,0,0) R2F-M0-N02-J12 and rank 4! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 11 of 16 (0,0,1,1,1,0) R2F-M0-N00-J10 and rank 11! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 15 of 16 (0,0,1,3,1,0) R2F-M0-N02-J06 and rank 15! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 9 of 16 (0,0,1,0,1,0) R2F-M0-N00-J06 and rank 9! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 12 of 16 (0,0,1,2,0,0) R2F-M0-N02-J13 and rank 12! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 8 of 16 (0,0,1,0,0,0) R2F-M0-N00-J01 and rank 8! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 7 of 16 (0,0,0,3,1,0) R2F-M0-N02-J07 and rank 7! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 2 of 16 (0,0,0,1,0,0) R2F-M0-N00-J12 and rank 2! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 14 of 16 (0,0,1,3,0,0) R2F-M0-N02-J01 and rank 14! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 10 of 16 (0,0,1,1,0,0) R2F-M0-N00-J13 and rank 10! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 5 of 16 (0,0,0,2,1,0) R2F-M0-N02-J11 and rank 5! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 6 of 16 (0,0,0,3,0,0) R2F-M0-N02-J00 and rank 6! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 13 of 16 (0,0,1,2,1,0) R2F-M0-N02-J10 and rank 13! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 1 of 16 (0,0,0,0,1,0) R2F-M0-N00-J07 and rank 1! Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name Task 3 of 16 (0,0,0,1,1,0) R2F-M0-N00-J11 and rank 3!


*\ Unit test suite ...


stderr is: 2016-04-19 16:39:51.284 (INFO ) [0x40001aebde0] ibm.runjob.AbstractOptions: using properties file /bgsys/local/etc/bg.properties 2016-04-19 16:39:51.286 (INFO ) [0x40001aebde0] ibm.runjob.AbstractOptions: max open file descriptors: 65536 2016-04-19 16:39:51.286 (INFO ) [0x40001aebde0] ibm.runjob.AbstractOptions: core file limit: 18446744073709551615 2016-04-19 16:39:51.288 (INFO ) [0x40001aebde0] 34635:tatu.runjob.client: scheduler job id is 794917 2016-04-19 16:39:51.308 (INFO ) [0x400016334e0] 34635:tatu.runjob.monitor: monitor started 2016-04-19 16:39:57.078 (INFO ) [0x400016334e0] 34635:tatu.runjob.monitor: task record 1746226 created 2016-04-19 16:39:57.079 (INFO ) [0x40001aebde0] MIR-488C0-7BBF1-512:34635:ibm.runjob.client.options.Parser: set local socket to runjob_mux from properti\ es file 2016-04-19 16:40:00.290 (INFO ) [0x40001aebde0] MIR-488C0-7BBF1-512:1947633:ibm.runjob.client.Job: job 1947633 started 2016-04-19 16:40:04.211 (WARN ) [0x40001aebde0] MIR-488C0-7BBF1-512:1947633:ibm.runjob.client.Job: terminated by signal 11 2016-04-19 16:40:04.212 (WARN ) [0x40001aebde0] MIR-488C0-7BBF1-512:1947633:ibm.runjob.client.Job: abnormal termination by signal 11 from rank 12 2016-04-19 16:40:04.212 (INFO ) [0x40001aebde0] tatu.runjob.client: task terminated by signal 11 2016-04-19 16:40:23.369 (INFO ) [0x400016334e0] 34635:tatu.runjob.monitor: tracklib completed 2016-04-19 16:40:23.369 (INFO ) [0x400016334e0] 34635:tatu.runjob.monitor: monitor terminating 2016-04-19 16:40:23.374 (INFO ) [0x40001aebde0] tatu.runjob.client: monitor completed

The link line is now: linkline.txt

bartlettroscoe commented 8 years ago

The xSDKTrilinos test problems are getting built now, but executing them fails. I'm assuming that I can run a test with:

@sarich, are you sure this is due to static library issues? Do all of the xSDKTrilinos tests fail like this? If you build and run native Trilinos tests (like for Epetra with -DEpetra_ENABLE_TESTS=ON) do they also fail? The error message just says that a segfault is occurring but not really giving any clues that I can see for what is causing this.

sarich commented 8 years ago

The xsdk trilinos tests are now working when run with a small number of processes (say 16 processes on 1 node), I haven't done much experimentation with when they run and when they fail. I also haven't looked at the source code to determine if this is expected or not.

On Wed, Apr 20, 2016 at 5:39 PM, Roscoe A. Bartlett < notifications@github.com> wrote:

The xSDKTrilinos test problems are getting built now, but executing them fails. I'm assuming that I can run a test with:

@sarich https://github.com/sarich, are you sure this is due to static library issues? Do all of the xSDKTrilinos tests fail like this? If you build and run native Trilinos tests (like for Epetra with -DEpetra_ENABLE_TESTS=ON) do they also fail? The error message just says that a segfault is occurring but not really giving any clues that I can see for what is causing this.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/trilinos/xSDKTrilinos/issues/2#issuecomment-212640348