Teuchos: Address issus with calling Fortran LAPACK rountines from C++ on Power8 with GCC/gfortran

[mhoemmen]

Next Action Status:

Looks like it was a problem with BLAS and LAPACK after all (see https://github.com/trilinos/Trilinos/issues/2454#issuecomment-386271621).

Blocks: #1191 Duplicates: #347 Analogous to: #1210 CC: @trilinos/teuchos @trilinos/framework

Description:

NOTE: The below original description does not pin-point the actual problem that was causing the failures on 'ride'. The real problem is what appears to be a mixed language calling defect summarized below.

Original Description:

BLAS and LAPACK libraries offer a Fortran 77 interface, but we want to call them from C++ code. This matters both in terms of Fortran function name mangling, and in terms of how to pass arguments into Fortran functions (the "application binary interface" or ABI). Most arguments just go into Fortran from the C or C++ world by pointer. However, some Fortran compilers have a different ABI for passing strings (CHARACTER(*)). Some compilers just take char*, but others may take char*, unsigned int or even char*, unsigned int* (or not unsigned). Fortran programmers don't see this; only programmers working in other languages (like C or C++), who need to know the Fortran ABI, may see this.

Teuchos' BLAS and LAPACK wrappers have two macros that govern the Fortran string ABI: CHAR_MACRO(c) and Teuchos_fcd.

CHAR_MACRO(c), defined redundantly in teuchos/numerics/src/Teuchos_LAPACK.cpp and Teuchos_BLAS.cpp, takes a const char c and does the right thing with it to pass it into a Fortran function. Teuchos offers two definitions of CHAR_MACRO:

#if defined (INTEL_CXML)
#define CHAR_MACRO(char_var) &char_var, one
#else
#define CHAR_MACRO(char_var) &char_var
#endif

namespace {
#if defined (INTEL_CXML)
        unsigned int one=1;
#endif
// ...
}

Per discussion in #1191, it looks like we need another definition for OpenBLAS on POWER, that passes in the constant one by address rather than by value.

Teuchos_fcd, defined redundantly in teuchos/numerics/src/Teuchos_LAPACK_wrappers.hpp and Teuchos_BLAS_wrappers.hpp, has the following definition:

#if defined(INTEL_CXML)
#  define PREFIX __stdcall
#  define Teuchos_fcd const char *, unsigned int
#elif defined(INTEL_MKL)
#  define PREFIX
#  define Teuchos_fcd const char *
#else /* Not CRAY_T3X or INTEL_CXML or INTEL_MKL */
#  define PREFIX
#  define Teuchos_fcd const char *
#endif

In the comments, CRAY_T3X refers to the Cray T3D and T3E, two mid-'90s distributed-memory computer architectures. This should give you an idea of the age of this code. In any case, we would need to add a new definition of Teuchos_fcd:

#  define Teuchos_fcd const char*, const int*

(See comments below for why it's an int and not unsigned int. It would be better to use int32_t, but Teuchos cannot assume C++11 / C99 types.)

[mhoemmen]

@nmhamster Would you happen to know where I could find documentation on the IBM (Open)POWER Fortran ABI?

[mhoemmen]

@ambrad might know this too...

[mhoemmen]

I was going to say that this was a POWER ABI with GCC issue, in which case, using the __powerpc64__ predefined GCC macro would help. However, it looks like this is a general issue with gfortran:

https://gcc.gnu.org/onlinedocs/gcc-6.3.0/gfortran/Argument-passing-conventions.html#Argument-passing-conventions

https://gcc.gnu.org/onlinedocs/gcc-4.9.2/gfortran/Argument-passing-conventions.html

Thus, the name of this issue really is justified: this comes about because we built OpenBLAS with gfortran, and gfortran passes character(*) as char*, int32_t* (unless it's a deferred-length string, in which case the length gets passed by value). Now I also know that the length is a signed 32-bit integer, which helps :-) . "Native" BLAS and LAPACK libraries usually get compiled such that character(*)' just gets passed aschar*`. The issue is a function both of the BLAS and LAPACK libraries, and of the compiler.

This implies the following solution:

1. Add CMake variable that tells us what string ABI scheme to use for the BLAS and LAPACK.
2. Consider automatic detection (if possible), but make sure that users can change the CMake variable manually.
3. Use the CMake variable to generate macros equivalent to CHAR_MACRO and Teuchos_fcd.

Other packages that have their own BLAS and LAPACK wrappers, like Epetra and ML, have the same two macros or the equivalent compile-time logic, so it should be easy to transfer this solution over to other packages.

[jwillenbring]

@krcb Could you look at this ticket and be ready to discuss it at the meeting tomorrow? If it would be better for @MicheldeMessieres to do the initial pass at the information that is fine too.

[mhoemmen]

This blocks #1191, among other issues. We haven't figured out the work-around for #1191 yet, which is to use ESSL with gcc on POWER, instead of OpenBLAS. If we can get ESSL to work with gcc on that platform, then hopefully that reduces the urgency. Currently, this issue blocks use of gcc on POWER.

[mhoemmen]

Also, the system BLAS is not an option on that platform. See e.g., the following configuration-time error: http://testing.sandia.gov/cdash/viewConfigure.php?buildid=2829720

[jjellio]

@mhoemmen I haven't worked with Power8+GCC in a few months. I did have this working with IBM XL+ESSL+OpenBLAS around November. The trick was to link esslsmp +XL required libs, then OpenBLAS. I seem to remember that ESSL did not provide a full BLAS implementation. I also used the SMP version of ESSL, not the serial version. I can check my link line tomorrow, there were a few required libs from XL that I had to manually provide paths for.

[nmhamster]

@mhoemmen @jjellio IBM does not provide a full BLAS or full LAPACK implementation in ESSL. They implement functions on a per-customer, per-request basis prioritizing the ones for larger installs as they get them. This means we are required to link we a second BLAS/LAPACK pack provider and insert ESSL first in the link line to get the behavior we would like.

[nmhamster]

@mhoemmen @jjellio - OK here you go for a simple DGEMM benchmark (note - I have modified the environment libraries to make all this work so you have definitely hit a bug in the testbed installs, we will need to fix this on other machines).

Module Load Configuration (you must follow this order):

(1) module load ibm/xl/13.1.5 (2) module load devpack/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44 (3) module load ibm/essl/5.4.0

Makefile Configuration:

CC=gfortran
CFLAGS=-O3 -fopenmp -DUSE_CBLAS
LDFLAGS=-L${ESSL_ROOT}/lib -L${XLF_ROOT}/lib -L${OPENBLAS_ROOT}/lib -L${XLF_ROOT}/../../xlsmp/4.1.5/lib \
    -lesslsmp -lopenblas -lxl -lxlopt -lxlfmath -lxlf90_r -lxlsmp

mt-dgemm: mt-dgemm.c
    $(CC) $(CFLAGS) -o mt-dgemm mt-dgemm.c $(LDFLAGS)

clean:
    rm mt-dgemm *.o

Run Line: OMP_NUM_THREADS=64 OMP_PLACES=threads OMP_PROC_BIND=close LD_LIBRARY_PATH=$XLF_ROOT/../../xlsmp/4.1.5/lib:$LD_LIBRARY_PATH ./mt-dgemm 2048

Output (single socket):

Matrix size input by command line: 2048
Repeat multiply defaulted to 8
Alpha =    1.000000
Beta  =    1.000000
Allocating Matrices...
Allocation complete, populating with values...
Performing multiplication...
Calculating matrix check...

===============================================================
Final Sum is:         2048.125000
Memory for Matrices:  96.000000 MB
Multiply time:        1.228393 seconds
FLOPs computed:       137506062336.000000
GFLOP/s rate:         111.939789 GF/s
===============================================================

[mhoemmen]

Thanks @nmhamster for posting this! :-)

@nmhamster wrote:

IBM does not provide a full BLAS or full LAPACK implementation in ESSL. They implement functions on a per-customer, per-request basis prioritizing the ones for larger installs as they get them. This means we are required to link we a second BLAS/LAPACK pack provider and insert ESSL first in the link line to get the behavior we would like.

Does this mean that Trilinos needs to support two different BLAS / LAPACK ABIs, one for the ESSL and one for OpenBLAS? Does it also mean that Trilinos needs to know which BLAS / LAPACK functions go with which ABI?

[mhoemmen]

Thanks @jjellio for responding :-) Was the ESSL complete enough that Trilinos mostly worked without needing to worry about missing functions?

[nmhamster]

@mhoemmen - I think the way to handle this is to specify the libraries in the order we want them (see my Makefile section) and then the linker will call the correct ones in ESSL first, if they don't exist it will drop through to OpenBLAS. That means the CMake options for BLAS and LAPACK just need to have all the libraries put in the correct order and I think you are good to go, the only issue is whether the calling conventions match up.

[bartlettroscoe]

I think the way to handle this is to specify the libraries in the order we want them (see my Makefile section) and then the linker will call the correct ones in ESSL first, if they don't exist it will drop through to OpenBLAS. That means the CMake options for BLAS and LAPACK just need to have all the libraries put in the correct order and I think you are good to go, the only issue is whether the calling conventions match up.

That will only work for static libraries? My guess is that this will not work for shared libs. It seems to make some assumptions about how the linker will work on that system and assumptions about how the libraries are put together and manifested as object code (i.e. there is no saved state from BLAS or LAPACK calls between the different function in the two library implementations). Is there precedent for this type of approach?

In any case, this approach should only be done on systems that absolutely need this. This would need to be a specialized case covered in the new Trilinos ATDM configure-related files (see TRIL-171).

[mhoemmen]

@nmhamster wrote:

...the only issue is whether the calling conventions match up.

That's my main concern :-) I'm asking because folks might be scoping out the task of refactoring the BLAS and LAPACK ABI logic in Trilinos right now, so it would be good to know whether they need to support two different ABIs. I really really hope not, because then Trilinos' CMake would need to know exactly which functions live in which library.

[nmhamster]

@mhoemmen - I think this is ultimately an issue that ESSL does not provide even a basic implementation of all functions. Having Trilinos figure out the exact calling convention across two libraries seems, to me at least, unreasonable. My gut feeling is that the best way to handle this in the end is to call into KokkosKernels, use the BLAS there and have it call into ESSL where we think there is a performance or correctness advantage and have us bypass OpenBLAS as a second library?

[mhoemmen]

@nmhamster wrote:

My gut feeling is that the best way to handle this in the end is to call into KokkosKernels, use the BLAS there . . . .

KokkosKernels doesn't have a complete BLAS. I don't see any push to write one any time soon.

This also doesn't solve the issue of missing LAPACK functions. Which ones are missing? We don't know yet. Does KokkosKernels need to implement dense eigensolvers? I'm certain they didn't sign up for that task.

Most importantly, lots of Trilinos packages do not use KokkosKernels to call BLAS or LAPACK functions. They use the Teuchos interface, or write their own interfaces. See e.g., #1210, #1211, #1212, #1213, and #1214. We can't make ML call KokkosKernels now; ML is a C package.

[mhoemmen]

@bartlettroscoe wrote:

. . . (i.e. there is no saved state from BLAS or LAPACK calls between the different function in the two library implementations).

LAPACK only recently started getting rid of saved state between calls, for functions like DLAMCH. Check out LAPACK 3.1.1's version of DLAMCH, with its SAVE and DATA statements:

http://www.netlib.org/lapack/explore-3.1.1-html/dlamch.f.html

It's entirely likely that ESSL hasn't moved with the times, unless it explicitly promises reentrant calls.

[nmhamster]

@mhoemmen @bartlettroscoe there are different versions of ESSL libraries available for various data types and threading scenarios.

@mhoemmen:

KokkosKernels doesn't have a complete BLAS. I don't see any push to write one any time soon. This also doesn't solve the issue of missing LAPACK functions. Which ones are missing? We don't know yet. Does KokkosKernels need to implement dense eigensolvers? I'm certain they didn't sign up for that task.

I know :-(, I just think that might be our longer term path. If we don't take a substrate kind of route, we will need to test on a per-function basis for everything we want to use in BLAS and LAPACK. That sounds potentially quite nasty. The result would be that we could then tailor each function calling ABI.

[mhoemmen]

@nmhamster Another option specific to OpenBLAS or other precompiled partial BLAS / LAPACK libraries, is that we can use Fortran 2003's ISO_C_BINDING module to build a shim library that presents an interface with whatever C ABI we want. All we need to do is build the shim with the same Fortran compiler as the original BLAS / LAPACK library. This approach should work with a partial library as well, as long as the shim knows what functions are in the partial library. Then, from Trilinos' perspective, it just sees the usual C ABI with whatever mangling we want.

[nmhamster]

@mhoemmen - we have all the compilers available to make this happen.

[mhoemmen]

@nmhamster Let me write an e-mail to people who normally interact with Tech-X, to see if they would consider this approach. Here are the advantages:

1. We would no longer need to detect or specify the ABI at configure time.
2. Due to (1), cross-compilation would Just Work.
3. We could strip out all the macro logic from all the BLAS and LAPACK interfaces in Trilinos. Trilinos would only see the mangling that it wants to see for BLAS and LAPACK functions.
4. The shim library could live in Trilinos/packages/common/auxiliarySoftware, and thus be available to all Trilinos packages.

Here are the disadvantages:

1. The shim library would need to wrap all BLAS and LAPACK functions that any Trilinos package calls.
2. The shim library would require a Fortran compiler that correctly implements the 2003 standard.
3. Trilinos currently has an option to build without a Fortran compiler. If we keep that, then we're back to the original problem of specifying or detecting the ABI for BLAS and LAPACK functions.

[kddevin]

One note: Not all platforms have Fortran compilers. For instance, I configure Trilinos on my laptop with -DTrilinos_ENABLE_Fortran=OFF. Please select a solution to this issue that allows builds without Fortran to continue to work. Thanks.

[jjellio]

@mhoemmen I'm updating my configure script now. I just built, GCC 5.4 + esslsmp + omp, and BLAS 32bit interface passed the TeuchosNumerics tests.

You have to do some hackery to get the rpath right, and you need a little more to make sure that Cmake does the right thing when it TryCompiles to figure out the Fortran interface.

I'll have to see if Epetra works. I'll post the script when I am done updating it.

[nmhamster]

Hey @jjellio this sounds great!

[krcb]

@jwillenbring, @MicheldeMessieres and I discussed this today. We are interested to find out the precise configuration that led to the error described above to see if we can reproduce. First, however, it seems we should wait for the outcome of @jjellio's testing.

[jjellio]

My script is a bit bloated (I hacked it from another).

I only module load devpack

jjellio@ride6:~/build$ cat do-configure-trilinos-gcc
#!/bin/bash

###
# ./do-configure [opt|dbg] [static|shared]
#  WARNING
# Sparc has a "remove duplicates" cmake function.
# this strips dupe items from the compiler flags...
# .. this will break the cuda front end (-Xcudafe ) flags.
# these are for suppressing annoying warnings.
# When sparc uses trilinos' flags, and strips the dupes,
# it will break these flags, so you will need to comment out
# sparc's remove duplcate stuff in their CMake file.
##

EXTRA_ARGS=$@

COMPILER=${CC_VENDOR}-${CXX_VERSION}
MPI=${MPI_VENDOR}-${MPI_VERSION}

TRILINOS_PATH=${HOME}/src/Trilinos

# pretend we have XL env
XLC_ROOT=/home/projects/pwr8-rhel73-lsf/ibm/xl/xlC/13.1.4
XLC_VERSION=13.1.4
XLF_ROOT=/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4
XLF_VERSION=15.1.4

OPT_FLAGS="-O3 -mtune=native -mcpu=native"
#REPORT_FLAGS="-qlist -qlistfmt=html -qreport -qphsinfo"

export REPORT_FLAGS=""
# optional : -qstrict=precision
EXTRA_F_FLAGS="-g -L${XLF_ROOT}/lib -L${XLC_ROOT}/lib -lopen-pal -lxl -lxlopt -lxlf90_r -lxlfmath -lm -libmc++ -lstdc++"
EXTRA_C_FLAGS="-g -L${XLF_ROOT}/lib -L${XLC_ROOT}/lib -lopen-pal -lxl -lxlopt -lxlf90_r -lxlfmath -lm -libmc++ -lstdc++"
EXTRA_CXX_FLAGS="-g -L${XLF_ROOT}/lib -L${XLC_ROOT}/lib -lopen-pal -lxl -lxlopt -lxlf90_r -lxlfmath -lm -libmc++ -lstdc++"
LINK_FLAGS="-g ${OPT_FLAGS} -L${XLF_ROOT}/lib -L${XLC_ROOT}/lib -lopen-pal -lxl -lxlopt -lxlf90_r -lxlfmath -lm -libmc++ -lstdc++ -L${MPIHOME}/lib -lmpi -Wl,-rpath,${XLC_ROOT}/lib -Wl,-rpath,${XLF_ROOT}/lib -Wl,-rpath,/ascldap/users/projects/pwr8-rhel72/ibm/xl/lib -Wl,-rpath,/home/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib"

#-lesslsmp
#-lxlf90_r
#-lxlfmath
#-lxl
#-lxlomp_ser

BLAS_DIR="${ESSL_ROOT}/lib64;${XLF_ROOT}/lib;${XLC_ROOT}/lib;/home/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib;/ascldap/users/projects/pwr8-rhel72/ibm/xl/lib;${OPENBLAS_ROOT}/lib"
LAPACK_DIR="${BLAS_DIR}"
LAPACK_BLAS_LIB="esslsmp;xlf90_r;xlfmath;xlopt;xl;xlsmp;xlomp_ser;openblas"

# Shouldn't need to change anything below this line
BUILD=${COMPILER}_${MPI}

LINK_DYNAMIC=ON
LINK_SUFFIX=dynamic

BUILD_TYPE=RELEASE
BUILD_SUFFIX=opt

CUDA=OFF
OPENMP=ON
PTHREAD=OFF
SERIAL=OFF
COMPLEX=OFF

TESTS=ON
EXAMPLES=ON

export MPI_DIR=${MPIHOME}
export MPI_ROOT=${MPI_DIR}

CXX_DETAILS=`mpicxx -showme:command`
C_DETAILS=`mpicc -showme:command`

echo CXX details: ${CXX_DETAILS}
echo C details: ${C_DETAILS}
echo OMPI_CXX: ${OMPI_CXX}
echo OMPI_CC:  ${OMPI_CC}

read -r -d '' MUELU_DETAILS <<- EOM
     -D Tpetra_ENABLE_MMM_Timings:BOOL=ON
     -D Trilinos_ENABLE_MueLu=ON
     -D KokkosKernels_ENABLE_Experimental:BOOL=ON
     -D TpetraKernels_ENABLE_Experimental:BOOL=ON
     -D MueLu_ENABLE_Experimental:BOOL=ON
     -D MueLu_ENABLE_Kokkos_Refactor:BOOL=ON
     -D Xpetra_ENABLE_Experimental:BOOL=ON
     -D Xpetra_ENABLE_Kokkos_Refactor:BOOL=ON
     -D Trilinos_ENABLE_Xpetra=ON
     -D Trilinos_ENABLE_Isorropia=ON
     -D Trilinos_ENABLE_Galeri=ON
     -D Trilinos_ENABLE_Teuchos=ON
     -D Trilinos_ENABLE_Epetra=ON
     -D Trilinos_ENABLE_EpetraExt=ON
     -D Trilinos_ENABLE_Anasazi=ON
     -D Anasazi_ENABLE_RBGen=ON
     -D Trilinos_ENABLE_Stokhos=OFF
     -D Trilinos_ENABLE_Teko=OFF
     -D Trilinos_ENABLE_Panzer=OFF
     -D Trilinos_ENABLE_Amesos2=ON
     -D Amesos2_ENABLE_KLU2=ON
     -D Trilinos_ENABLE_Ifpack2=ON
     -D Trilinos_ENABLE_Zoltan2=ON
EOM

echo "$MUELU_DETAILS"

MUELU_DETAILS=""

TRILINOS_HOME=${TRILINOS_PATH}
TRILINOS_INSTALL=${HOME}/install/Trilinos/${BUILD}_cuda-${CUDA_VERSION}_${LINK_SUFFIX}_${BUILD_SUFFIX}-omp_muelu-experimental

export MY_YAMLCPP_ROOT=${YAMLCPP_ROOT}

echo BUILD=${BUILD}
echo TRILINOS_INSTALL=${TRILINOS_INSTALL}
echo TRILINOS_HOME=${TRILINOS_HOME}
echo CUDA_ARCH_NAME=${CUDA_ARCH_NAME}
echo COMPILER_ROOT=
echo MPI_ROOT=${MPI_ROOT}
echo LAPACK_DIR=${LAPACK_DIR}
echo Using Lapack/BLAS lib: ${LAPACK_BLAS_LIB}
echo BLAS_ROOT=${BLAS_ROOT}
echo LAPACK_ROOT=${LAPACK_ROOT}
echo HDF5_ROOT=${HDF5_ROOT}
echo NETCDF_ROOT=${NETCDF_ROOT}
echo ZLIB_ROOT=${ZLIB_ROOT}
echo BOOST_ROOT=${BOOST_ROOT}
echo METIS_ROOT=${METIS_ROOT}
echo PARMETIS_ROOT=${PARMETIS_ROOT}
echo SUPERLUDIST_ROOT=${SUPERLUDIST_ROOT}

#rm -f CMakeCache.txt; rm -rf CMakeFiles

time \
cmake \
 -D Trilinos_ENABLE_DEBUG:BOOL=OFF \
\
 -D CMAKE_BUILD_TYPE=RELEASE \
 -D CMAKE_VERBOSE_MAKEFILE=OFF \
 -D Trilinos_ENABLE_ALL_PACKAGES=OFF \
 -D Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=ON \
 -D BUILD_SHARED_LIBS=${LINK_DYNAMIC} \
 -D DART_TESTING_TIMEOUT:STRING=200 \
 -D Trilinos_ENABLE_CXX11=ON \
  \
   -D CMAKE_INSTALL_PREFIX:PATH=${TRILINOS_INSTALL} \
   \
   -D CMAKE_C_COMPILER="mpicc" \
   -D CMAKE_CXX_COMPILER="mpicxx" \
   -D CMAKE_Fortran_COMPILER="mpif90" \
   -D CMAKE_C_FLAGS="$EXTRA_C_FLAGS" \
   -D CMAKE_CXX_FLAGS="$EXTRA_CXX_FLAGS" \
   -D CMAKE_Fortran_FLAGS="$EXTRA_F_FLAGS" \
   -D CMAKE_EXE_LINKER_FLAGS="$LINK_FLAGS" \
   -D Trilinos_CXX11_FLAGS:STRING=-std=c++11 \
   \
   -D Trilinos_VERBOSE_CONFIGURE=OFF \
   \
   -D Trilinos_ENABLE_TESTS=${TESTS}\
   -D Trilinos_ENABLE_EXAMPLES=${EXAMPLES} \
   -D DART_TESTING_TIMEOUT:STRING="200" \
   \
   -D Trilinos_ENABLE_EXPLICIT_INSTANTIATION=ON \
   -D Tpetra_INST_FLOAT=OFF \
   -D Tpetra_INST_DOUBLE=ON \
   -D Tpetra_INST_COMPLEX_FLOAT=OFF \
   -D Tpetra_INST_COMPLEX_DOUBLE=OFF \
   -D Tpetra_INST_INT_INT=ON \
   -D Tpetra_INST_INT_LONG=OFF \
   -D Tpetra_INST_INT_UNSIGNED=OFF \
   -D Tpetra_INST_INT_LONG_LONG=ON \
   -D Teuchos_ENABLE_LONG_LONG_INT=ON \
   -D Teuchos_ENABLE_COMPLEX=OFF \
   \
  -D Trilinos_ENABLE_Kokkos=ON \
  -D Kokkos_ENABLE_TESTS:BOOL=ON \
  -D Trilinos_ENABLE_OpenMP=${OPENMP} \
  -D Kokkos_ENABLE_Serial:BOOL=${SERIAL} \
  -D Kokkos_ENABLE_OpenMP:BOOL=${OPENMP} \
  -D Kokkos_ENABLE_Pthread:BOOL=${PTHREAD} \
  -D Kokkos_ENABLE_Cuda:BOOL=${CUDA} \
  -D TPL_ENABLE_Pthread=${PTHREAD} \
  -D TPL_ENABLE_CUDA=${CUDA} \
  -D Teuchos_ENABLE_COMPLEX=OFF \
\
-D Trilinos_ENABLE_Tpetra=ON \
  -D Tpetra_INST_SERIAL:BOOL=${SERIAL} \
  -D Tpetra_INST_OPENMP:BOOL=${OPENMP} \
  -D Tpetra_INST_PTHREAD:BOOL=${PTHREAD} \
  -D Tpetra_INST_CUDA:BOOL=${CUDA} \
  -D Tpetra_INST_COMPLEX_DOUBLE:BOOL=${COMPLEX} \
  -D Tpetra_ENABLE_TESTS:BOOL=${TESTS} \
  -D Tpetra_ENABLE_EXAMPLES:BOOL=${EXAMPLES} \
\
-D Trilinos_ENABLE_Belos=ON \
-D Trilinos_ENABLE_Teuchos=ON \
\
${MUELU_DETAILS} \
\
   -D TPL_ENABLE_MPI=ON \
   -D MPI_USE_COMPILER_WRAPPERS=ON \
   -D MPI_EXEC:PATH="mpirun" \
   -D MPI_EXEC_NUMPROCS_FLAG:STRING="--map-by;ppr:2:NUMA:pe=4;--display-map;--bind-to;core;-np" \
   \
   -D TPL_ENABLE_Zlib=ON \
   -D Zlib_LIBRARY_DIRS="${ZLIB_ROOT}/lib" \
   \
   -D TPL_ENABLE_DLlib=ON \
   \
   -D TPL_ENABLE_yaml-cpp=OFF \
   -D yaml-cpp_LIBRARY_DIRS:PATH=${MY_YAMLCPP_ROOT}/lib \
   -D yaml-cpp_INCLUDE_DIRS:PATH=${MY_YAMLCPP_ROOT}/include \
   \
   -D TPL_ENABLE_BLAS=ON \
   -D BLAS_LIBRARY_DIRS:PATH="${BLAS_DIR}" \
   -D BLAS_LIBRARY_NAMES:STRING="${LAPACK_BLAS_LIB}" \
   \
   -D TPL_ENABLE_LAPACK=ON \
   -D LAPACK_LIBRARY_DIRS:PATH="${LAPACK_DIR}" \
   -D LAPACK_LIBRARY_NAMES:STRING="${LAPACK_BLAS_LIB}" \
   \
   -D TPL_ENABLE_Boost=ON \
   -D Boost_INCLUDE_DIRS:PATH=${BOOST_ROOT}/include \
   \
   -D TPL_ENABLE_BoostLib=ON \
   -D BoostLib_INCLUDE_DIRS:PATH=${BOOST_ROOT}/include \
   -D BoostLib_LIBRARY_DIRS:PATH=${BOOST_ROOT}/lib \
   \
   -D TPL_ENABLE_Netcdf=ON \
   -D Netcdf_INCLUDE_DIRS:PATH="${NETCDF_ROOT}/include;${HDF5_ROOT}/include" \
   -D Netcdf_LIBRARY_DIRS:PATH="${NETCDF_ROOT}/lib;${PNETCDF_ROOT}/lib;${HDF5_ROOT}/lib;${ZLIB_ROOT}/lib" \
   -D Netcdf_LIBRARY_NAMES:STRING="netcdf;pnetcdf;hdf5_hl;hdf5;z" \
   \
   -D TPL_ENABLE_METIS=ON \
   -D METIS_INCLUDE_DIRS:PATH=${METIS_ROOT}/include \
   -D METIS_LIBRARY_DIRS:PATH=${METIS_ROOT}/lib \
   \
   -D TPL_ENABLE_ParMETIS=ON \
   -D ParMETIS_INCLUDE_DIRS:PATH="${PARMETIS_ROOT}/include;${METIS_ROOT}/include" \
   -D ParMETIS_LIBRARY_DIRS:PATH="${PARMETIS_ROOT}/lib;${METIS_ROOT}/lib" \
   \
\
$EXTRA_ARGS \
${TRILINOS_PATH}

jjellio@ride14:~/build/trilinos-gcc-pw8/packages/teuchos/numerics/test$ ctest
Test project /ascldap/users/jjellio/build/trilinos-gcc-pw8/packages/teuchos/numerics/test
      Start  1: TeuchosNumerics_BLAS_test_MPI_1
 1/11 Test  #1: TeuchosNumerics_BLAS_test_MPI_1 ....................   Passed    0.87 sec
      Start  2: TeuchosNumerics_BLAS_tmpl_test_MPI_1
 2/11 Test  #2: TeuchosNumerics_BLAS_tmpl_test_MPI_1 ...............   Passed    0.52 sec
      Start  3: TeuchosNumerics_BLAS_tmpl_comp_test_MPI_1
 3/11 Test  #3: TeuchosNumerics_BLAS_tmpl_comp_test_MPI_1 ..........   Passed    0.93 sec
      Start  4: TeuchosNumerics_BLAS_ROTG_test_MPI_1
 4/11 Test  #4: TeuchosNumerics_BLAS_ROTG_test_MPI_1 ...............   Passed    0.56 sec
      Start  5: TeuchosNumerics_DenseSolver_test_MPI_1
 5/11 Test  #5: TeuchosNumerics_DenseSolver_test_MPI_1 .............   Passed    0.26 sec
      Start  6: TeuchosNumerics_BandDenseSolver_test_MPI_1
 6/11 Test  #6: TeuchosNumerics_BandDenseSolver_test_MPI_1 .........   Passed    0.23 sec
      Start  7: TeuchosNumerics_SpdDenseSolver_test_MPI_1
 7/11 Test  #7: TeuchosNumerics_SpdDenseSolver_test_MPI_1 ..........   Passed    0.23 sec
      Start  8: TeuchosNumerics_QRDenseSolver_test_MPI_1
 8/11 Test  #8: TeuchosNumerics_QRDenseSolver_test_MPI_1 ...........   Passed    0.23 sec
      Start  9: TeuchosNumerics_LAPACK_test_MPI_1
 9/11 Test  #9: TeuchosNumerics_LAPACK_test_MPI_1 ..................   Passed    0.23 sec
      Start 10: TeuchosNumerics_Polynomial_test_MPI_1
10/11 Test #10: TeuchosNumerics_Polynomial_test_MPI_1 ..............   Passed    0.51 sec
      Start 11: TeuchosNumerics_MatrixMarket_Raw_InOutTest_MPI_1
11/11 Test #11: TeuchosNumerics_MatrixMarket_Raw_InOutTest_MPI_1 ...   Passed    0.22 sec

100% tests passed, 0 tests failed out of 11

[jjellio]

To bring the script up to date, you could probably carefully load modules related to XL, to get the LD_LIBRARY_PATH right.

I did this on RIDE btw.

I'll build Epetra now.

[jjellio]

EDIT: the failures listed are because of my cmake MPIRUN flags, not because of segfaults. EDIT2: que'd properly, all Epetra tests pass.

Epetra:

jjellio@ride14:~/build/trilinos-gcc-pw8/packages/epetra$ ctest 
Test project /ascldap/users/jjellio/build/trilinos-gcc-pw8/packages/epetra
      Start  1: Epetra_BlockMap_test_MPI_4
 1/61 Test  #1: Epetra_BlockMap_test_MPI_4 ..............................   Passed    0.23 sec
      Start  2: Epetra_BasicPerfTest_test_MPI_1
 2/61 Test  #2: Epetra_BasicPerfTest_test_MPI_1 .........................   Passed    1.03 sec
      Start  3: Epetra_Comm_test_MPI_1
 3/61 Test  #3: Epetra_Comm_test_MPI_1 ..................................   Passed    1.19 sec
      Start  4: Epetra_CrsGraph_test_unit_MPI_4
 4/61 Test  #4: Epetra_CrsGraph_test_unit_MPI_4 .........................   Passed    0.19 sec
      Start  5: Epetra_CrsMatrix_test_MPI_4
 5/61 Test  #5: Epetra_CrsMatrix_test_MPI_4 .............................***Failed  Required regular expression not found.Regex=[tests PASSED
]  0.18 sec
      Start  6: Epetra_CrsMatrix_memorytest_MPI_1
 6/61 Test  #6: Epetra_CrsMatrix_memorytest_MPI_1 .......................   Passed    0.88 sec
      Start  7: Epetra_RowMatrix_test_MPI_1
 7/61 Test  #7: Epetra_RowMatrix_test_MPI_1 .............................   Passed    2.08 sec
      Start  8: Epetra_CrsRectMatrix_test_MPI_1
 8/61 Test  #8: Epetra_CrsRectMatrix_test_MPI_1 .........................   Passed    1.18 sec
      Start  9: Epetra_Directory_test_MPI_1
 9/61 Test  #9: Epetra_Directory_test_MPI_1 .............................   Passed    0.54 sec
      Start 10: Epetra_FECrsGraph_test_MPI_1
10/61 Test #10: Epetra_FECrsGraph_test_MPI_1 ............................   Passed    0.92 sec
      Start 11: Epetra_FECrsMatrix_test_MPI_1
11/61 Test #11: Epetra_FECrsMatrix_test_MPI_1 ...........................   Passed    0.88 sec
      Start 12: Epetra_FEmatrix2_MPI_1
12/61 Test #12: Epetra_FEmatrix2_MPI_1 ..................................   Passed    0.23 sec
      Start 13: Epetra_FEVbrMatrix_test_MPI_1
13/61 Test #13: Epetra_FEVbrMatrix_test_MPI_1 ...........................   Passed    0.86 sec
      Start 14: Epetra_FEVector_test_MPI_1
14/61 Test #14: Epetra_FEVector_test_MPI_1 ..............................   Passed    0.89 sec
      Start 15: Epetra_FEvector_MPI_1
15/61 Test #15: Epetra_FEvector_MPI_1 ...................................   Passed    1.18 sec
      Start 16: Epetra_FusedImportExport_test_MPI_4
16/61 Test #16: Epetra_FusedImportExport_test_MPI_4 .....................***Failed  Required regular expression not found.Regex=[tests PASSED
]  0.18 sec
      Start 17: Epetra_ImportExport_test_MPI_4
17/61 Test #17: Epetra_ImportExport_test_MPI_4 ..........................   Passed    0.18 sec
      Start 18: Epetra_IntSerialDense_test_MPI_1
18/61 Test #18: Epetra_IntSerialDense_test_MPI_1 ........................   Passed    0.85 sec
      Start 19: Epetra_Map_test_MPI_1
19/61 Test #19: Epetra_Map_test_MPI_1 ...................................   Passed    0.87 sec
      Start 20: Epetra_MapColoring_test_MPI_1
20/61 Test #20: Epetra_MapColoring_test_MPI_1 ...........................   Passed    0.99 sec
      Start 21: Epetra_MultiVector_test_MPI_1
21/61 Test #21: Epetra_MultiVector_test_MPI_1 ...........................   Passed    1.79 sec
      Start 22: Epetra_Object_test_MPI_1
22/61 Test #22: Epetra_Object_test_MPI_1 ................................   Passed    0.86 sec
      Start 23: Epetra_RowMatrixTransposer_test_MPI_1
23/61 Test #23: Epetra_RowMatrixTransposer_test_MPI_1 ...................   Passed    5.37 sec
      Start 24: Epetra_SerialDense_test_MPI_1
24/61 Test #24: Epetra_SerialDense_test_MPI_1 ...........................   Passed    1.34 sec
      Start 25: Epetra_SerialSpdDense_test_MPI_1
25/61 Test #25: Epetra_SerialSpdDense_test_MPI_1 ........................   Passed    1.21 sec
      Start 26: Epetra_VbrMatrix_test_MPI_1
26/61 Test #26: Epetra_VbrMatrix_test_MPI_1 .............................   Passed   10.41 sec
      Start 27: Epetra_Vector_test_MPI_1
27/61 Test #27: Epetra_Vector_test_MPI_1 ................................   Passed    1.15 sec
      Start 28: Epetra_Bug_5988_FECrs_buildNonlocalGraph_MPI_4
28/61 Test #28: Epetra_Bug_5988_FECrs_buildNonlocalGraph_MPI_4 ..........   Passed    0.18 sec
      Start 29: Epetra_Bug_6079_DistObject_CombineMode_flags_MPI_4
29/61 Test #29: Epetra_Bug_6079_DistObject_CombineMode_flags_MPI_4 ......   Passed    0.18 sec
      Start 30: Epetra_SimpleLongLongTest_MPI_4
30/61 Test #30: Epetra_SimpleLongLongTest_MPI_4 .........................   Passed    0.18 sec
      Start 31: Epetra_BlockMap_test_LL_MPI_4
31/61 Test #31: Epetra_BlockMap_test_LL_MPI_4 ...........................   Passed    0.18 sec
      Start 32: Epetra_BasicPerfTest_test_LL_MPI_1
32/61 Test #32: Epetra_BasicPerfTest_test_LL_MPI_1 ......................   Passed    1.08 sec
      Start 33: Epetra_CrsGraph_test_LL_unit_MPI_4
33/61 Test #33: Epetra_CrsGraph_test_LL_unit_MPI_4 ......................   Passed    0.20 sec
      Start 34: Epetra_CrsMatrix_test_LL_MPI_4
34/61 Test #34: Epetra_CrsMatrix_test_LL_MPI_4 ..........................***Failed  Required regular expression not found.Regex=[tests PASSED
]  0.21 sec
      Start 35: Epetra_CrsMatrix_memorytest_LL_MPI_1
35/61 Test #35: Epetra_CrsMatrix_memorytest_LL_MPI_1 ....................   Passed    1.00 sec
      Start 36: Epetra_FusedImportExport_test_LL_MPI_4
36/61 Test #36: Epetra_FusedImportExport_test_LL_MPI_4 ..................***Failed  Required regular expression not found.Regex=[tests PASSED
]  0.18 sec
      Start 37: Epetra_RowMatrix_test_LL_MPI_1
37/61 Test #37: Epetra_RowMatrix_test_LL_MPI_1 ..........................   Passed    2.19 sec
      Start 38: Epetra_CrsRectMatrix_test_LL_MPI_1
38/61 Test #38: Epetra_CrsRectMatrix_test_LL_MPI_1 ......................   Passed    1.23 sec
      Start 39: Epetra_Directory_test_LL_MPI_1
39/61 Test #39: Epetra_Directory_test_LL_MPI_1 ..........................   Passed    0.54 sec
      Start 40: Epetra_FECrsGraph_test_LL_MPI_1
40/61 Test #40: Epetra_FECrsGraph_test_LL_MPI_1 .........................   Passed    1.05 sec
      Start 41: Epetra_FECrsMatrix_test_LL_MPI_1
41/61 Test #41: Epetra_FECrsMatrix_test_LL_MPI_1 ........................   Passed    0.94 sec
      Start 42: Epetra_FEmatrix2_LL_MPI_1
42/61 Test #42: Epetra_FEmatrix2_LL_MPI_1 ...............................   Passed    0.24 sec
      Start 43: Epetra_ImportExport_test_LL_MPI_4
43/61 Test #43: Epetra_ImportExport_test_LL_MPI_4 .......................   Passed    0.18 sec
      Start 44: Epetra_Map_test_LL_MPI_1
44/61 Test #44: Epetra_Map_test_LL_MPI_1 ................................   Passed    0.86 sec
      Start 45: Epetra_MapColoring_test_LL_MPI_1
45/61 Test #45: Epetra_MapColoring_test_LL_MPI_1 ........................   Passed    0.92 sec
      Start 46: Epetra_SerialDense_test_LL_MPI_1
46/61 Test #46: Epetra_SerialDense_test_LL_MPI_1 ........................   Passed    1.33 sec
      Start 47: Epetra_Bug_5791_StaticProifle_LL_MPI_4
47/61 Test #47: Epetra_Bug_5791_StaticProifle_LL_MPI_4 ..................   Passed    0.19 sec
      Start 48: Epetra_Bug_5794_IndexBase_LL_MPI_4
48/61 Test #48: Epetra_Bug_5794_IndexBase_LL_MPI_4 ......................   Passed    0.19 sec
      Start 49: Epetra_Bug_5797_Sort_MakeColMap_LL_MPI_4
49/61 Test #49: Epetra_Bug_5797_Sort_MakeColMap_LL_MPI_4 ................   Passed    0.19 sec
      Start 50: Epetra_Bug_5988_FECrs_buildNonlocalGraph_LL_MPI_4
50/61 Test #50: Epetra_Bug_5988_FECrs_buildNonlocalGraph_LL_MPI_4 .......   Passed    0.19 sec
      Start 51: Epetra_Bug_6079_DistObject_CombineMode_flags_LL_MPI_4
51/61 Test #51: Epetra_Bug_6079_DistObject_CombineMode_flags_LL_MPI_4 ...   Passed    0.18 sec
      Start 52: Epetra_Oski_test_MPI_1
52/61 Test #52: Epetra_Oski_test_MPI_1 ..................................   Passed    0.23 sec
      Start 53: Epetra_my_example_MPI_4
53/61 Test #53: Epetra_my_example_MPI_4 .................................   Passed    0.18 sec
      Start 54: Epetra_my_example_LL_MPI_4
54/61 Test #54: Epetra_my_example_LL_MPI_4 ..............................   Passed    0.20 sec
      Start 55: Epetra_lesson01_mpi_only_through_Epetra_MPI_4
55/61 Test #55: Epetra_lesson01_mpi_only_through_Epetra_MPI_4 ...........***Failed  Required regular expression not found.Regex=[End Result: TEST PASSED
]  0.18 sec
      Start 56: Epetra_lesson01_mpi_on_its_own_MPI_4
56/61 Test #56: Epetra_lesson01_mpi_on_its_own_MPI_4 ....................***Failed  Required regular expression not found.Regex=[End Result: TEST PASSED
]  0.18 sec
      Start 57: Epetra_lesson01_no_mpi_MPI_4
57/61 Test #57: Epetra_lesson01_no_mpi_MPI_4 ............................***Failed  Required regular expression not found.Regex=[End Result: TEST PASSED
]  0.19 sec
      Start 58: Epetra_lesson02_init_map_vec_MPI_4
58/61 Test #58: Epetra_lesson02_init_map_vec_MPI_4 ......................***Failed  Required regular expression not found.Regex=[End Result: TEST PASSED
]  0.18 sec
      Start 59: Epetra_lesson02_read_modify_vec_MPI_4
59/61 Test #59: Epetra_lesson02_read_modify_vec_MPI_4 ...................***Failed  Required regular expression not found.Regex=[End Result: TEST PASSED
]  0.18 sec
      Start 60: Epetra_lesson03_power_method_MPI_4
60/61 Test #60: Epetra_lesson03_power_method_MPI_4 ......................***Failed  Required regular expression not found.Regex=[End Result: TEST PASSED
]  0.22 sec
      Start 61: Epetra_lesson05_redistribution_MPI_4
61/61 Test #61: Epetra_lesson05_redistribution_MPI_4 ....................***Failed  Required regular expression not found.Regex=[End Result: TEST PASSED
]  0.18 sec

82% tests passed, 11 tests failed out of 61

Label Time Summary:
Epetra    =  53.41 sec (61 tests)

Total Test time (real) =  53.54 sec

The following tests FAILED:
      5 - Epetra_CrsMatrix_test_MPI_4 (Failed)
     16 - Epetra_FusedImportExport_test_MPI_4 (Failed)
     34 - Epetra_CrsMatrix_test_LL_MPI_4 (Failed)
     36 - Epetra_FusedImportExport_test_LL_MPI_4 (Failed)
     55 - Epetra_lesson01_mpi_only_through_Epetra_MPI_4 (Failed)
     56 - Epetra_lesson01_mpi_on_its_own_MPI_4 (Failed)
     57 - Epetra_lesson01_no_mpi_MPI_4 (Failed)
     58 - Epetra_lesson02_init_map_vec_MPI_4 (Failed)
     59 - Epetra_lesson02_read_modify_vec_MPI_4 (Failed)
     60 - Epetra_lesson03_power_method_MPI_4 (Failed)
     61 - Epetra_lesson05_redistribution_MPI_4 (Failed)
Errors while running CTest

[jjellio]

@mhoemmen is there a specific unit test that will test the ABI problem you had? Teuchos Numerics and Epetra passed all tests. I am building Belos now. I am not sure if I've built Belos before with this compiler/blas combo.

[jjellio]

Belos fails one test:

jjellio@ride14:~/build/trilinos-gcc-pw8/packages/belos$ ctest 
Test project /ascldap/users/jjellio/build/trilinos-gcc-pw8/packages/belos
      Start  1: Belos_ValidParameters_MPI_1
 1/61 Test  #1: Belos_ValidParameters_MPI_1 ..........................   Passed    0.56 sec
      Start  2: Belos_Factory_MPI_4
 2/61 Test  #2: Belos_Factory_MPI_4 ..................................   Passed    0.80 sec
      Start  3: Belos_Aztec_to_Belos_param_MPI_4
 3/61 Test  #3: Belos_Aztec_to_Belos_param_MPI_4 .....................   Passed    0.34 sec
      Start  4: Belos_BiCGStab_prec_file_MPI_4
 4/61 Test  #4: Belos_BiCGStab_prec_file_MPI_4 .......................   Passed    1.14 sec
      Start  5: Belos_bl_gmres_diag_MPI_4
 5/61 Test  #5: Belos_bl_gmres_diag_MPI_4 ............................   Passed    1.18 sec
      Start  6: Belos_bl_gmres_hb_0_MPI_4
 6/61 Test  #6: Belos_bl_gmres_hb_0_MPI_4 ............................   Passed    2.01 sec
      Start  7: Belos_bl_gmres_hb_1_MPI_4
 7/61 Test  #7: Belos_bl_gmres_hb_1_MPI_4 ............................   Passed    1.42 sec
      Start  8: Belos_resolve_gmres_hb_0_MPI_4
 8/61 Test  #8: Belos_resolve_gmres_hb_0_MPI_4 .......................   Passed    2.31 sec
      Start  9: Belos_resolve_gmres_hb_1_MPI_4
 9/61 Test  #9: Belos_resolve_gmres_hb_1_MPI_4 .......................   Passed    2.21 sec
      Start 10: Belos_pseudo_gmres_hb_MPI_4
10/61 Test #10: Belos_pseudo_gmres_hb_MPI_4 ..........................   Passed    3.35 sec
      Start 11: Belos_bl_pgmres_hb_0_MPI_4
11/61 Test #11: Belos_bl_pgmres_hb_0_MPI_4 ...........................   Passed    2.18 sec
      Start 12: Belos_bl_pgmres_hb_1_MPI_4
12/61 Test #12: Belos_bl_pgmres_hb_1_MPI_4 ...........................   Passed    1.36 sec
      Start 13: Belos_bl_pgmres_hb_2_MPI_4
13/61 Test #13: Belos_bl_pgmres_hb_2_MPI_4 ...........................   Passed    2.60 sec
      Start 14: Belos_pseudo_pgmres_hb_MPI_4
14/61 Test #14: Belos_pseudo_pgmres_hb_MPI_4 .........................   Passed    2.02 sec
      Start 15: Belos_bl_fgmres_hb_MPI_4
15/61 Test #15: Belos_bl_fgmres_hb_MPI_4 .............................   Passed    2.56 sec
      Start 16: Belos_pseudo_cg_indefinite_MPI_4
16/61 Test #16: Belos_pseudo_cg_indefinite_MPI_4 .....................   Passed    1.26 sec
      Start 17: Belos_bl_cg_hb_MPI_4
17/61 Test #17: Belos_bl_cg_hb_MPI_4 .................................   Passed    1.59 sec
      Start 18: Belos_resolve_cg_hb_MPI_4
18/61 Test #18: Belos_resolve_cg_hb_MPI_4 ............................   Passed    2.26 sec
      Start 19: Belos_bl_cg_set_res_MPI_4
19/61 Test #19: Belos_bl_cg_set_res_MPI_4 ............................   Passed    1.38 sec
      Start 20: Belos_bl_pcg_hb_0_MPI_4
20/61 Test #20: Belos_bl_pcg_hb_0_MPI_4 ..............................   Passed    1.41 sec
      Start 21: Belos_bl_pcg_hb_1_MPI_4
21/61 Test #21: Belos_bl_pcg_hb_1_MPI_4 ..............................   Passed    1.46 sec
      Start 22: Belos_bl_pcg_set_res_0_MPI_4
22/61 Test #22: Belos_bl_pcg_set_res_0_MPI_4 .........................   Passed    1.52 sec
      Start 23: Belos_bl_pcg_set_res_1_MPI_4
23/61 Test #23: Belos_bl_pcg_set_res_1_MPI_4 .........................   Passed    1.43 sec
      Start 24: Belos_pseudo_pcg_hb_0_MPI_4
24/61 Test #24: Belos_pseudo_pcg_hb_0_MPI_4 ..........................   Passed    1.41 sec
      Start 25: Belos_pseudo_pcg_hb_1_MPI_4
25/61 Test #25: Belos_pseudo_pcg_hb_1_MPI_4 ..........................   Passed    1.34 sec
      Start 26: Belos_pseudo_stochastic_pcg_hb_0_MPI_4
26/61 Test #26: Belos_pseudo_stochastic_pcg_hb_0_MPI_4 ...............   Passed    1.31 sec
      Start 27: Belos_pseudo_stochastic_pcg_hb_1_MPI_4
27/61 Test #27: Belos_pseudo_stochastic_pcg_hb_1_MPI_4 ...............   Passed    1.48 sec
      Start 28: Belos_gcrodr_hb_MPI_4
28/61 Test #28: Belos_gcrodr_hb_MPI_4 ................................   Passed   15.96 sec
      Start 29: Belos_prec_gcrodr_hb_0_MPI_4
29/61 Test #29: Belos_prec_gcrodr_hb_0_MPI_4 .........................   Passed    1.56 sec
      Start 30: Belos_prec_gcrodr_hb_1_MPI_4
30/61 Test #30: Belos_prec_gcrodr_hb_1_MPI_4 .........................   Passed    1.48 sec
      Start 31: Belos_rcg_hb_MPI_4
31/61 Test #31: Belos_rcg_hb_MPI_4 ...................................   Passed   19.30 sec
      Start 32: Belos_prec_rcg_hb_0_MPI_4
32/61 Test #32: Belos_prec_rcg_hb_0_MPI_4 ............................   Passed    1.51 sec
      Start 33: Belos_prec_rcg_hb_1_MPI_4
33/61 Test #33: Belos_prec_rcg_hb_1_MPI_4 ............................   Passed    1.57 sec
      Start 34: Belos_pseudo_tfqmr_hb_MPI_4
34/61 Test #34: Belos_pseudo_tfqmr_hb_MPI_4 ..........................   Passed    1.29 sec
      Start 35: Belos_tfqmr_hb_0_MPI_4
35/61 Test #35: Belos_tfqmr_hb_0_MPI_4 ...............................   Passed    1.14 sec
      Start 36: Belos_tfqmr_hb_1_MPI_4
36/61 Test #36: Belos_tfqmr_hb_1_MPI_4 ...............................   Passed    1.12 sec
      Start 37: Belos_tfqmr_hb_2_MPI_4
37/61 Test #37: Belos_tfqmr_hb_2_MPI_4 ...............................   Passed    1.19 sec
      Start 38: Belos_ptfqmr_hb_0_MPI_4
38/61 Test #38: Belos_ptfqmr_hb_0_MPI_4 ..............................   Passed    1.23 sec
      Start 39: Belos_ptfqmr_hb_1_MPI_4
39/61 Test #39: Belos_ptfqmr_hb_1_MPI_4 ..............................   Passed    1.52 sec
      Start 40: Belos_ptfqmr_hb_2_MPI_4
40/61 Test #40: Belos_ptfqmr_hb_2_MPI_4 ..............................   Passed    1.32 sec
      Start 41: Belos_ptfqmr_hb_3_MPI_4
41/61 Test #41: Belos_ptfqmr_hb_3_MPI_4 ..............................   Passed    1.48 sec
      Start 42: Belos_pseudo_ptfqmr_hb_0_MPI_4
42/61 Test #42: Belos_pseudo_ptfqmr_hb_0_MPI_4 .......................   Passed    1.27 sec
      Start 43: Belos_pseudo_ptfqmr_hb_1_MPI_4
43/61 Test #43: Belos_pseudo_ptfqmr_hb_1_MPI_4 .......................   Passed    1.47 sec
      Start 44: Belos_pseudo_ptfqmr_hb_2_MPI_4
44/61 Test #44: Belos_pseudo_ptfqmr_hb_2_MPI_4 .......................   Passed    1.20 sec
      Start 45: Belos_pseudo_ptfqmr_hb_3_MPI_4
45/61 Test #45: Belos_pseudo_ptfqmr_hb_3_MPI_4 .......................   Passed    1.51 sec
      Start 46: Belos_minres_indefinite_MPI_4
46/61 Test #46: Belos_minres_indefinite_MPI_4 ........................   Passed    1.27 sec
      Start 47: Belos_minres_hb_MPI_4
47/61 Test #47: Belos_minres_hb_MPI_4 ................................   Passed    3.97 sec
      Start 48: Belos_fp_hb_MPI_4
48/61 Test #48: Belos_fp_hb_MPI_4 ....................................   Passed    1.25 sec
      Start 49: Belos_MVOPTester_MPI_4
49/61 Test #49: Belos_MVOPTester_MPI_4 ...............................   Passed    1.14 sec
      Start 50: Belos_solver_factory_test_MPI_4
50/61 Test #50: Belos_solver_factory_test_MPI_4 ......................   Passed    1.14 sec
      Start 51: Belos_Epetra64_Laplacian_Driver_0_MPI_4
51/61 Test #51: Belos_Epetra64_Laplacian_Driver_0_MPI_4 ..............   Passed    1.17 sec
      Start 52: Belos_Epetra64_Laplacian_Driver_1_MPI_4
52/61 Test #52: Belos_Epetra64_Laplacian_Driver_1_MPI_4 ..............   Passed    1.15 sec
      Start 53: Belos_BlockFlexPrecGmres_Epetra_File_Ex_0_MPI_4
53/61 Test #53: Belos_BlockFlexPrecGmres_Epetra_File_Ex_0_MPI_4 ......   Passed    1.15 sec
      Start 54: Belos_BlockFlexPrecGmres_Epetra_File_Ex_1_MPI_4
54/61 Test #54: Belos_BlockFlexPrecGmres_Epetra_File_Ex_1_MPI_4 ......   Passed    1.28 sec
      Start 55: Belos_Tpetra_BlockCG_hb_test_MPI_4
55/61 Test #55: Belos_Tpetra_BlockCG_hb_test_MPI_4 ...................   Passed    1.66 sec
      Start 56: Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4
56/61 Test #56: Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4 .............***Failed    1.68 sec
      Start 57: Belos_Tpetra_PseudoBlockStochasticCG_hb_test_MPI_4
57/61 Test #57: Belos_Tpetra_PseudoBlockStochasticCG_hb_test_MPI_4 ...   Passed    1.69 sec
      Start 58: Belos_Tpetra_FixedPoint_hb_test_MPI_4
58/61 Test #58: Belos_Tpetra_FixedPoint_hb_test_MPI_4 ................   Passed    1.53 sec
      Start 59: Belos_Tpetra_LinearSolverFactory_MPI_4
59/61 Test #59: Belos_Tpetra_LinearSolverFactory_MPI_4 ...............   Passed    0.82 sec
      Start 60: Belos_Tpetra_MultipleSolves_MPI_4
60/61 Test #60: Belos_Tpetra_MultipleSolves_MPI_4 ....................   Passed    1.26 sec
      Start 61: Belos_Tpetra_MVOPTester_complex_test_MPI_4
61/61 Test #61: Belos_Tpetra_MVOPTester_complex_test_MPI_4 ...........   Passed    1.27 sec

98% tests passed, 1 tests failed out of 61

Label Time Summary:
Belos    = 124.38 sec (61 tests)

Total Test time (real) = 124.50 sec

The following tests FAILED:
     56 - Belos_Tpetra_PseudoBlockCG_hb_test_MPI_4 (Failed)
Errors while running CTest

I suppose this is the error. I was hoping Teuchos numerics would have caught it.

56:  ========================   JOB MAP   ========================
56: 
56:  Data for node: ride14  Num slots: 4    Max slots: 0    Num procs: 4
56:     Process OMPI jobid: [59307,1] App: 0 Process rank: 0
56:     Process OMPI jobid: [59307,1] App: 0 Process rank: 1
56:     Process OMPI jobid: [59307,1] App: 0 Process rank: 2
56:     Process OMPI jobid: [59307,1] App: 0 Process rank: 3
56: 
56:  =============================================================
56: Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ride14 and rank 0!
56: Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ride14 and rank 1!
56: Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ride14 and rank 2!
56: Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ride14 and rank 3!
56:  ** On entry to DSTEQR parameter number  1 had an illegal value
56:  ** On entry to DSTEQR parameter number  1 had an illegal value
56:  ** On entry to DSTEQR parameter number  1 had an illegal value
56:  ** On entry to DSTEQR parameter number  1 had an illegal value

[krcb]

@jjellio this is very helpful; thanks. Could you add the output from the configure step, specifically the find of LAPACK/BLAS?

[jjellio]

ESSL does not provide dsteqr, that is atleast one reason you have to use OpenBLAS.

This was on Ride, but the same thing should work on White.

This doesn't fix the ABI issue, but it does show how to build with ESSLSMP/OpenMP/GCC

grab 1 node, with 4 procs per node

bsub -x -Is -W 24:00 -n 4 -R "span[hosts=1] affinity[thread(8, same=core, exclusive=(core,injob|alljobs))*4: cpubind=core]"  -q rhel7F /usr/bin/bash

modules

module load devpack/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
module load ibm/essl

Processing enabled TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF)
-- Searching for libs in BLAS_LIBRARY_DIRS='/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlC/13.1.4/lib;/home/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib;/ascldap/users/projects/pwr8-rhel72/ibm/xl/lib;/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib'
-- Searching for a lib in the set "esslsmp":
--   Searching for lib 'esslsmp' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64/libesslsmp.so'
-- Searching for a lib in the set "xlf90_r":
--   Searching for lib 'xlf90_r' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlf90_r.so'
-- Searching for a lib in the set "xlfmath":
--   Searching for lib 'xlfmath' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlfmath.so'
-- Searching for a lib in the set "xlopt":
--   Searching for lib 'xlopt' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlopt.a'
-- Searching for a lib in the set "xl":
--   Searching for lib 'xl' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxl.a'
-- Searching for a lib in the set "xlsmp":
--   Searching for lib 'xlsmp' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlsmp.so'
-- Searching for a lib in the set "xlomp_ser":
--   Searching for lib 'xlomp_ser' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlomp_ser.so'
-- Searching for a lib in the set "openblas":
--   Searching for lib 'openblas' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/libopenblas.so'

Full output:

jjellio@ride15:~/build/dummy$ ../do-configure-trilinos-gcc
CXX details: /home/projects/pwr8-rhel73-lsf/gcc/5.4.0/bin/g++
C details: /home/projects/pwr8-rhel73-lsf/gcc/5.4.0/bin/gcc
OMPI_CXX:
OMPI_CC:
BUILD=gcc-5.4.0_openmpi-1.10.4
TRILINOS_INSTALL=/ascldap/users/jjellio/install/Trilinos/gcc-5.4.0_openmpi-1.10.4_cuda-8.0.44_dynamic_opt-omp_muelu-experimental
TRILINOS_HOME=/ascldap/users/jjellio/src/Trilinos
COMPILER_ROOT=
MPI_ROOT=/home/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
LAPACK_DIR=/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlC/13.1.4/lib;/home/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib;/ascldap/users/projects/pwr8-rhel72/ibm/xl/lib;/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib
Using Lapack/BLAS lib: esslsmp;xlf90_r;xlfmath;xlopt;xl;xlsmp;xlomp_ser;openblas
BLAS_ROOT=/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0
LAPACK_ROOT=/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0
HDF5_ROOT=/home/projects/pwr8-rhel73-lsf/hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
NETCDF_ROOT=/home/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
ZLIB_ROOT=/home/projects/pwr8-rhel73-lsf/zlib/1.2.8
BOOST_ROOT=/home/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
METIS_ROOT=/home/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0
PARMETIS_ROOT=/home/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44
SUPERLUDIST_ROOT=

Configuring Trilinos build directory

-- PROJECT_SOURCE_DIR='/ascldap/users/jjellio/src/Trilinos'
-- PROJECT_BINARY_DIR='/ascldap/users/jjellio/build/dummy'
-- Trilinos_TRIBITS_DIR='/ascldap/users/jjellio/src/Trilinos/cmake/tribits'
-- TriBITS_VERSION_STRING='0.9 (Dev)'
-- CMAKE_VERSION='3.6.2'
-- Reading in configuration options from /ascldap/users/jjellio/src/Trilinos/cmake/StdDevEnvs.cmake ...
-- Trilinos_USE_BUILD_ENV=''
-- CMAKE_HOST_SYSTEM_NAME='Linux'
-- Trilinos_HOSTNAME='ride15'
-- Found PythonInterp: /usr/bin/python (found suitable version "2.7.5", minimum required is "2.6") 
-- PYTHON_EXECUTABLE='/usr/bin/python'

Setting up major user options ...

-- Setting Trilinos_ENABLE_TriKota=OFF because '/ascldap/users/jjellio/src/Trilinos/packages/TriKota/Dakota' does not exist!

Trilinos repos versions:
--------------------------------------------------------------------------------
*** Base Git Repo: Trilinos
ce1372a [Wed Dec 7 14:35:53 2016 -0700] <jjellio@sandia.gov>
Merge branch 'jjellio/fine-grain-tpetra' of https://github.com/jjellio/Trilinos 
 --------------------------------------------------------------------------------

Reading list of native packages from /ascldap/users/jjellio/src/Trilinos/PackagesList.cmake

-- Trilinos_NUM_PACKAGES='56'

Reading list of native TPLs from /ascldap/users/jjellio/src/Trilinos/TPLsList.cmake

-- Trilinos_NUM_TPLS='102'

Processing Project, Repository, and Package dependency files and building internal dependencies graph ...

-- Trilinos_NUM_SE_PACKAGES='148'

Explicitly enabled packages on input (by user):  Kokkos Teuchos Epetra Tpetra EpetraExt Xpetra Isorropia Galeri Zoltan2 Belos Amesos2 Anasazi Ifpack2 MueLu 14

Explicitly enabled SE packages on input (by user):  Kokkos Teuchos Epetra Tpetra EpetraExt Xpetra Isorropia Galeri Zoltan2 Belos Amesos2 Anasazi Ifpack2 MueLu 14

Explicitly disabled packages on input (by user or by default):  Claps Trios Teko TriKota Stokhos Panzer NewPackage MeshingGenie 8

Explicitly disabled SE packages on input (by user or by default):  KokkosExample Claps ShyLUHTS ShyLUTacho ShyLUBDDC ShyLUBasker ShyLUFastILU Trioscommsplitter Triossupport Triosnnti Triosnssi Triosprograms Triosexamples Triostests Triosnetcdf-service Trios Teko TriKota STKClassic STKExp Stokhos Panzer NewPackage MeshingGenie 24

Explicitly enabled TPLs on input (by user):  MPI BLAS LAPACK Boost METIS ParMETIS Zlib Netcdf BoostLib DLlib 10

Explicitly disabled TPLs on input (by user or by default):  yaml-cpp CUDA Pthread 3

Disabling all packages that have a required dependency on disabled TPLs and optional package TPL support based on TPL_ENABLE_<TPL>=OFF ...

Disabling subpackages for hard disables of parent packages due to Trilinos_ENABLE_<PARENT_PACKAGE>=OFF ...

-- Setting subpackage enable Trilinos_ENABLE_PanzerCore=OFF because parent package Trilinos_ENABLE_Panzer=OFF
-- Setting subpackage enable Trilinos_ENABLE_PanzerDofMgr=OFF because parent package Trilinos_ENABLE_Panzer=OFF
-- Setting subpackage enable Trilinos_ENABLE_PanzerDiscFE=OFF because parent package Trilinos_ENABLE_Panzer=OFF
-- Setting subpackage enable Trilinos_ENABLE_PanzerAdaptersSTK=OFF because parent package Trilinos_ENABLE_Panzer=OFF

Disabling forward required SE packages and optional intra-package support that have a dependancy on disabled SE packages Trilinos_ENABLE_<TRIBITS_PACKAGE>=OFF ...

-- Setting Kokkos_ENABLE_KokkosExample=OFF because Kokkos has an optional library dependence on disabled package KokkosExample
-- Setting Ifpack2_ENABLE_ShyLUHTS=OFF because Ifpack2 has an optional library dependence on disabled package ShyLUHTS
-- Setting Amesos2_ENABLE_ShyLUBasker=OFF because Amesos2 has an optional library dependence on disabled package ShyLUBasker
-- Setting Ifpack2_ENABLE_ShyLUBasker=OFF because Ifpack2 has an optional library dependence on disabled package ShyLUBasker
-- Setting MueLu_ENABLE_Teko=OFF because MueLu has an optional library dependence on disabled package Teko

Enabling subpackages for hard enables of parent packages due to Trilinos_ENABLE_<PARENT_PACKAGE>=ON ...

-- Setting subpackage enable Trilinos_ENABLE_KokkosCore=ON because parent package Trilinos_ENABLE_Kokkos=ON
-- Setting subpackage enable Trilinos_ENABLE_KokkosContainers=ON because parent package Trilinos_ENABLE_Kokkos=ON
-- Setting subpackage enable Trilinos_ENABLE_KokkosAlgorithms=ON because parent package Trilinos_ENABLE_Kokkos=ON
-- Setting subpackage enable Trilinos_ENABLE_TeuchosCore=ON because parent package Trilinos_ENABLE_Teuchos=ON
-- Setting subpackage enable Trilinos_ENABLE_TeuchosParameterList=ON because parent package Trilinos_ENABLE_Teuchos=ON
-- Setting subpackage enable Trilinos_ENABLE_TeuchosComm=ON because parent package Trilinos_ENABLE_Teuchos=ON
-- Setting subpackage enable Trilinos_ENABLE_TeuchosNumerics=ON because parent package Trilinos_ENABLE_Teuchos=ON
-- Setting subpackage enable Trilinos_ENABLE_TeuchosRemainder=ON because parent package Trilinos_ENABLE_Teuchos=ON
-- Setting subpackage enable Trilinos_ENABLE_TeuchosKokkosCompat=ON because parent package Trilinos_ENABLE_Teuchos=ON
-- Setting subpackage enable Trilinos_ENABLE_TeuchosKokkosComm=ON because parent package Trilinos_ENABLE_Teuchos=ON
-- Setting subpackage enable Trilinos_ENABLE_TpetraClassic=ON because parent package Trilinos_ENABLE_Tpetra=ON
-- Setting subpackage enable Trilinos_ENABLE_TpetraKernels=ON because parent package Trilinos_ENABLE_Tpetra=ON
-- Setting subpackage enable Trilinos_ENABLE_TpetraTSQR=ON because parent package Trilinos_ENABLE_Tpetra=ON
-- Setting subpackage enable Trilinos_ENABLE_TpetraCore=ON because parent package Trilinos_ENABLE_Tpetra=ON

Enabling all tests and/or examples that have not been explicitly disabled because Trilinos_ENABLE_[TESTS,EXAMPLES]=ON ...

-- Setting KokkosCore_ENABLE_TESTS=ON
-- Setting KokkosCore_ENABLE_EXAMPLES=ON
-- Setting KokkosContainers_ENABLE_TESTS=ON
-- Setting KokkosContainers_ENABLE_EXAMPLES=ON
-- Setting KokkosAlgorithms_ENABLE_TESTS=ON
-- Setting KokkosAlgorithms_ENABLE_EXAMPLES=ON
-- Setting Kokkos_ENABLE_EXAMPLES=ON
-- Setting TeuchosCore_ENABLE_TESTS=ON
-- Setting TeuchosCore_ENABLE_EXAMPLES=ON
-- Setting TeuchosParameterList_ENABLE_TESTS=ON
-- Setting TeuchosParameterList_ENABLE_EXAMPLES=ON
-- Setting TeuchosComm_ENABLE_TESTS=ON
-- Setting TeuchosComm_ENABLE_EXAMPLES=ON
-- Setting TeuchosNumerics_ENABLE_TESTS=ON
-- Setting TeuchosNumerics_ENABLE_EXAMPLES=ON
-- Setting TeuchosRemainder_ENABLE_TESTS=ON
-- Setting TeuchosRemainder_ENABLE_EXAMPLES=ON
-- Setting TeuchosKokkosCompat_ENABLE_TESTS=ON
-- Setting TeuchosKokkosCompat_ENABLE_EXAMPLES=ON
-- Setting TeuchosKokkosComm_ENABLE_TESTS=ON
-- Setting TeuchosKokkosComm_ENABLE_EXAMPLES=ON
-- Setting Teuchos_ENABLE_TESTS=ON
-- Setting Teuchos_ENABLE_EXAMPLES=ON
-- Setting Epetra_ENABLE_TESTS=ON
-- Setting Epetra_ENABLE_EXAMPLES=ON
-- Setting TpetraClassic_ENABLE_TESTS=ON
-- Setting TpetraClassic_ENABLE_EXAMPLES=ON
-- Setting TpetraKernels_ENABLE_TESTS=ON
-- Setting TpetraKernels_ENABLE_EXAMPLES=ON
-- Setting TpetraTSQR_ENABLE_TESTS=ON
-- Setting TpetraTSQR_ENABLE_EXAMPLES=ON
-- Setting TpetraCore_ENABLE_TESTS=ON
-- Setting TpetraCore_ENABLE_EXAMPLES=ON
-- Setting EpetraExt_ENABLE_TESTS=ON
-- Setting EpetraExt_ENABLE_EXAMPLES=ON
-- Setting Xpetra_ENABLE_TESTS=ON
-- Setting Xpetra_ENABLE_EXAMPLES=ON
-- Setting Isorropia_ENABLE_TESTS=ON
-- Setting Isorropia_ENABLE_EXAMPLES=ON
-- Setting Galeri_ENABLE_TESTS=ON
-- Setting Galeri_ENABLE_EXAMPLES=ON
-- Setting Zoltan2_ENABLE_TESTS=ON
-- Setting Zoltan2_ENABLE_EXAMPLES=ON
-- Setting Belos_ENABLE_TESTS=ON
-- Setting Belos_ENABLE_EXAMPLES=ON
-- Setting Amesos2_ENABLE_TESTS=ON
-- Setting Amesos2_ENABLE_EXAMPLES=ON
-- Setting Anasazi_ENABLE_TESTS=ON
-- Setting Anasazi_ENABLE_EXAMPLES=ON
-- Setting Ifpack2_ENABLE_TESTS=ON
-- Setting Ifpack2_ENABLE_EXAMPLES=ON
-- Setting MueLu_ENABLE_TESTS=ON
-- Setting MueLu_ENABLE_EXAMPLES=ON

Enabling all required (and optional since Trilinos_ENABLE_ALL_OPTIONAL_PACKAGES=ON) upstream SE packages for current set of enabled packages ...

-- Setting Trilinos_ENABLE_Amesos=ON because MueLu has an optional dependence on Amesos
-- Setting Trilinos_ENABLE_Ifpack=ON because MueLu has an optional dependence on Ifpack
-- Setting Trilinos_ENABLE_Intrepid2=ON because MueLu has an optional dependence on Intrepid2
-- Setting Trilinos_ENABLE_ML=ON because MueLu has an optional dependence on ML
-- Setting Trilinos_ENABLE_Zoltan=ON because MueLu has an optional dependence on Zoltan
-- Setting Trilinos_ENABLE_Stratimikos=ON because MueLu has an optional dependence on Stratimikos
-- Setting Trilinos_ENABLE_Thyra=ON because MueLu has an optional dependence on Thyra
-- Setting Trilinos_ENABLE_ThyraTpetraAdapters=ON because MueLu has an optional dependence on ThyraTpetraAdapters
-- Setting Trilinos_ENABLE_AztecOO=ON because MueLu has an optional dependence on AztecOO
-- Setting Trilinos_ENABLE_Pamgen=ON because MueLu has an optional dependence on Pamgen
-- Setting Trilinos_ENABLE_Shards=ON because Intrepid2 has a required dependence on Shards
-- Setting Trilinos_ENABLE_Sacado=ON because Intrepid2 has a required dependence on Sacado
-- Setting Trilinos_ENABLE_Gtest=ON because Intrepid2 has an optional dependence on Gtest
-- Setting Trilinos_ENABLE_ThyraEpetraAdapters=ON because Stratimikos has a required dependence on ThyraEpetraAdapters
-- Setting Trilinos_ENABLE_Triutils=ON because Stratimikos has an optional dependence on Triutils
-- Setting Trilinos_ENABLE_ThyraCore=ON because Anasazi has an optional dependence on ThyraCore
-- Setting Trilinos_ENABLE_ThyraEpetraExtAdapters=ON because Thyra has an optional dependence on ThyraEpetraExtAdapters
-- Setting Trilinos_ENABLE_RTOp=ON because ThyraCore has a required dependence on RTOp

Enabling all optional intra-package enables <TRIBITS_PACKAGE>_ENABLE_<DEPPACKAGE> that are not currently disabled if both sets of packages are enabled ...

-- Setting Kokkos_ENABLE_KokkosContainers=ON since Trilinos_ENABLE_Kokkos=ON AND Trilinos_ENABLE_KokkosContainers=ON
-- Setting Kokkos_ENABLE_KokkosAlgorithms=ON since Trilinos_ENABLE_Kokkos=ON AND Trilinos_ENABLE_KokkosAlgorithms=ON
-- Setting TeuchosCore_ENABLE_KokkosCore=ON since Trilinos_ENABLE_TeuchosCore=ON AND Trilinos_ENABLE_KokkosCore=ON
-- Setting Teuchos_ENABLE_TeuchosKokkosCompat=ON since Trilinos_ENABLE_Teuchos=ON AND Trilinos_ENABLE_TeuchosKokkosCompat=ON
-- Setting Teuchos_ENABLE_TeuchosKokkosComm=ON since Trilinos_ENABLE_Teuchos=ON AND Trilinos_ENABLE_TeuchosKokkosComm=ON
-- Setting Sacado_ENABLE_Teuchos=ON since Trilinos_ENABLE_Sacado=ON AND Trilinos_ENABLE_Teuchos=ON
-- Setting Sacado_ENABLE_KokkosCore=ON since Trilinos_ENABLE_Sacado=ON AND Trilinos_ENABLE_KokkosCore=ON
-- Setting Sacado_ENABLE_TeuchosKokkosComm=ON since Trilinos_ENABLE_Sacado=ON AND Trilinos_ENABLE_TeuchosKokkosComm=ON
-- Setting Sacado_ENABLE_TeuchosCore=ON since Trilinos_ENABLE_Sacado=ON AND Trilinos_ENABLE_TeuchosCore=ON
-- Setting Sacado_ENABLE_KokkosContainers=ON since Trilinos_ENABLE_Sacado=ON AND Trilinos_ENABLE_KokkosContainers=ON
-- Setting Epetra_ENABLE_Teuchos=ON since Trilinos_ENABLE_Epetra=ON AND Trilinos_ENABLE_Teuchos=ON
-- Setting Shards_ENABLE_Teuchos=ON since Trilinos_ENABLE_Shards=ON AND Trilinos_ENABLE_Teuchos=ON
-- Setting TpetraCore_ENABLE_Epetra=ON since Trilinos_ENABLE_TpetraCore=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting TpetraCore_ENABLE_TpetraTSQR=ON since Trilinos_ENABLE_TpetraCore=ON AND Trilinos_ENABLE_TpetraTSQR=ON
-- Setting Tpetra_ENABLE_TpetraKernels=ON since Trilinos_ENABLE_Tpetra=ON AND Trilinos_ENABLE_TpetraKernels=ON
-- Setting Tpetra_ENABLE_TpetraTSQR=ON since Trilinos_ENABLE_Tpetra=ON AND Trilinos_ENABLE_TpetraTSQR=ON
-- Setting EpetraExt_ENABLE_Triutils=ON since Trilinos_ENABLE_EpetraExt=ON AND Trilinos_ENABLE_Triutils=ON
-- Setting ThyraTpetraAdapters_ENABLE_ThyraEpetraAdapters=ON since Trilinos_ENABLE_ThyraTpetraAdapters=ON AND Trilinos_ENABLE_ThyraEpetraAdapters=ON
-- Setting Thyra_ENABLE_ThyraEpetraAdapters=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_ThyraEpetraAdapters=ON
-- Setting Thyra_ENABLE_ThyraEpetraExtAdapters=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_ThyraEpetraExtAdapters=ON
-- Setting Thyra_ENABLE_ThyraTpetraAdapters=ON since Trilinos_ENABLE_Thyra=ON AND Trilinos_ENABLE_ThyraTpetraAdapters=ON
-- Setting Xpetra_ENABLE_Epetra=ON since Trilinos_ENABLE_Xpetra=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting Xpetra_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Xpetra=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Xpetra_ENABLE_Tpetra=ON since Trilinos_ENABLE_Xpetra=ON AND Trilinos_ENABLE_Tpetra=ON
-- Setting Xpetra_ENABLE_KokkosCore=ON since Trilinos_ENABLE_Xpetra=ON AND Trilinos_ENABLE_KokkosCore=ON
-- Setting Xpetra_ENABLE_KokkosContainers=ON since Trilinos_ENABLE_Xpetra=ON AND Trilinos_ENABLE_KokkosContainers=ON
-- Setting Xpetra_ENABLE_Thyra=ON since Trilinos_ENABLE_Xpetra=ON AND Trilinos_ENABLE_Thyra=ON
-- Setting Isorropia_ENABLE_Tpetra=ON since Trilinos_ENABLE_Isorropia=ON AND Trilinos_ENABLE_Tpetra=ON
-- Setting AztecOO_ENABLE_Teuchos=ON since Trilinos_ENABLE_AztecOO=ON AND Trilinos_ENABLE_Teuchos=ON
-- Setting Galeri_ENABLE_Epetra=ON since Trilinos_ENABLE_Galeri=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting Galeri_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Galeri=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Galeri_ENABLE_Xpetra=ON since Trilinos_ENABLE_Galeri=ON AND Trilinos_ENABLE_Xpetra=ON
-- Setting Galeri_ENABLE_Tpetra=ON since Trilinos_ENABLE_Galeri=ON AND Trilinos_ENABLE_Tpetra=ON
-- Setting Amesos_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Amesos=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Amesos_ENABLE_Triutils=ON since Trilinos_ENABLE_Amesos=ON AND Trilinos_ENABLE_Triutils=ON
-- Setting Amesos_ENABLE_Galeri=ON since Trilinos_ENABLE_Amesos=ON AND Trilinos_ENABLE_Galeri=ON
-- Setting Zoltan2_ENABLE_Epetra=ON since Trilinos_ENABLE_Zoltan2=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting Zoltan2_ENABLE_Galeri=ON since Trilinos_ENABLE_Zoltan2=ON AND Trilinos_ENABLE_Galeri=ON
-- Setting Zoltan2_ENABLE_Pamgen=ON since Trilinos_ENABLE_Zoltan2=ON AND Trilinos_ENABLE_Pamgen=ON
-- Setting Ifpack_ENABLE_Amesos=ON since Trilinos_ENABLE_Ifpack=ON AND Trilinos_ENABLE_Amesos=ON
-- Setting Ifpack_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Ifpack=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Ifpack_ENABLE_AztecOO=ON since Trilinos_ENABLE_Ifpack=ON AND Trilinos_ENABLE_AztecOO=ON
-- Setting Ifpack_ENABLE_Galeri=ON since Trilinos_ENABLE_Ifpack=ON AND Trilinos_ENABLE_Galeri=ON
-- Setting ML_ENABLE_Teuchos=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_Teuchos=ON
-- Setting ML_ENABLE_Epetra=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting ML_ENABLE_Zoltan=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_Zoltan=ON
-- Setting ML_ENABLE_Galeri=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_Galeri=ON
-- Setting ML_ENABLE_Amesos=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_Amesos=ON
-- Setting ML_ENABLE_Ifpack=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_Ifpack=ON
-- Setting ML_ENABLE_AztecOO=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_AztecOO=ON
-- Setting ML_ENABLE_EpetraExt=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting ML_ENABLE_Isorropia=ON since Trilinos_ENABLE_ML=ON AND Trilinos_ENABLE_Isorropia=ON
-- Setting Belos_ENABLE_Epetra=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting Belos_ENABLE_Tpetra=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_Tpetra=ON
-- Setting Belos_ENABLE_Thyra=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_Thyra=ON
-- Setting Belos_ENABLE_AztecOO=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_AztecOO=ON
-- Setting Belos_ENABLE_Galeri=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_Galeri=ON
-- Setting Belos_ENABLE_Triutils=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_Triutils=ON
-- Setting Belos_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Belos_ENABLE_Ifpack=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_Ifpack=ON
-- Setting Belos_ENABLE_ML=ON since Trilinos_ENABLE_Belos=ON AND Trilinos_ENABLE_ML=ON
-- Setting Amesos2_ENABLE_Epetra=ON since Trilinos_ENABLE_Amesos2=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting Amesos2_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Amesos2=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Amesos2_ENABLE_Kokkos=ON since Trilinos_ENABLE_Amesos2=ON AND Trilinos_ENABLE_Kokkos=ON
-- Setting Anasazi_ENABLE_Tpetra=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_Tpetra=ON
-- Setting Anasazi_ENABLE_Epetra=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting Anasazi_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Anasazi_ENABLE_ThyraCore=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_ThyraCore=ON
-- Setting Anasazi_ENABLE_ThyraEpetraAdapters=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_ThyraEpetraAdapters=ON
-- Setting Anasazi_ENABLE_Belos=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_Belos=ON
-- Setting Anasazi_ENABLE_Amesos=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_Amesos=ON
-- Setting Anasazi_ENABLE_AztecOO=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_AztecOO=ON
-- Setting Anasazi_ENABLE_Galeri=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_Galeri=ON
-- Setting Anasazi_ENABLE_Ifpack=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_Ifpack=ON
-- Setting Anasazi_ENABLE_Triutils=ON since Trilinos_ENABLE_Anasazi=ON AND Trilinos_ENABLE_Triutils=ON
-- Setting Ifpack2_ENABLE_Xpetra=ON since Trilinos_ENABLE_Ifpack2=ON AND Trilinos_ENABLE_Xpetra=ON
-- Setting Ifpack2_ENABLE_Zoltan2=ON since Trilinos_ENABLE_Ifpack2=ON AND Trilinos_ENABLE_Zoltan2=ON
-- Setting Ifpack2_ENABLE_ThyraTpetraAdapters=ON since Trilinos_ENABLE_Ifpack2=ON AND Trilinos_ENABLE_ThyraTpetraAdapters=ON
-- Setting Ifpack2_ENABLE_Amesos2=ON since Trilinos_ENABLE_Ifpack2=ON AND Trilinos_ENABLE_Amesos2=ON
-- Setting Ifpack2_ENABLE_ML=ON since Trilinos_ENABLE_Ifpack2=ON AND Trilinos_ENABLE_ML=ON
-- Setting Stratimikos_ENABLE_Amesos=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_Amesos=ON
-- Setting Stratimikos_ENABLE_AztecOO=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_AztecOO=ON
-- Setting Stratimikos_ENABLE_Belos=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_Belos=ON
-- Setting Stratimikos_ENABLE_Ifpack=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_Ifpack=ON
-- Setting Stratimikos_ENABLE_ML=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_ML=ON
-- Setting Stratimikos_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Stratimikos_ENABLE_Triutils=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_Triutils=ON
-- Setting Stratimikos_ENABLE_Ifpack2=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_Ifpack2=ON
-- Setting Stratimikos_ENABLE_ThyraTpetraAdapters=ON since Trilinos_ENABLE_Stratimikos=ON AND Trilinos_ENABLE_ThyraTpetraAdapters=ON
-- Setting Intrepid2_ENABLE_Epetra=ON since Trilinos_ENABLE_Intrepid2=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting Intrepid2_ENABLE_EpetraExt=ON since Trilinos_ENABLE_Intrepid2=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting Intrepid2_ENABLE_Amesos=ON since Trilinos_ENABLE_Intrepid2=ON AND Trilinos_ENABLE_Amesos=ON
-- Setting Intrepid2_ENABLE_Pamgen=ON since Trilinos_ENABLE_Intrepid2=ON AND Trilinos_ENABLE_Pamgen=ON
-- Setting Intrepid2_ENABLE_Gtest=ON since Trilinos_ENABLE_Intrepid2=ON AND Trilinos_ENABLE_Gtest=ON
-- Setting MueLu_ENABLE_Amesos=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Amesos=ON
-- Setting MueLu_ENABLE_Amesos2=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Amesos2=ON
-- Setting MueLu_ENABLE_Epetra=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Epetra=ON
-- Setting MueLu_ENABLE_EpetraExt=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_EpetraExt=ON
-- Setting MueLu_ENABLE_Ifpack=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Ifpack=ON
-- Setting MueLu_ENABLE_Ifpack2=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Ifpack2=ON
-- Setting MueLu_ENABLE_Intrepid2=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Intrepid2=ON
-- Setting MueLu_ENABLE_ML=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_ML=ON
-- Setting MueLu_ENABLE_Tpetra=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Tpetra=ON
-- Setting MueLu_ENABLE_Zoltan=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Zoltan=ON
-- Setting MueLu_ENABLE_Zoltan2=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Zoltan2=ON
-- Setting MueLu_ENABLE_Stratimikos=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Stratimikos=ON
-- Setting MueLu_ENABLE_Thyra=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Thyra=ON
-- Setting MueLu_ENABLE_ThyraTpetraAdapters=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_ThyraTpetraAdapters=ON
-- Setting MueLu_ENABLE_Isorropia=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Isorropia=ON
-- Setting MueLu_ENABLE_KokkosCore=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_KokkosCore=ON
-- Setting MueLu_ENABLE_KokkosContainers=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_KokkosContainers=ON
-- Setting MueLu_ENABLE_AztecOO=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_AztecOO=ON
-- Setting MueLu_ENABLE_Galeri=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Galeri=ON
-- Setting MueLu_ENABLE_Belos=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Belos=ON
-- Setting MueLu_ENABLE_Pamgen=ON since Trilinos_ENABLE_MueLu=ON AND Trilinos_ENABLE_Pamgen=ON

Enabling all remaining required TPLs for current set of enabled packages ...

Enabling all optional package TPL support for currently enabled TPLs ...

-- Setting KokkosCore_ENABLE_DLlib=ON since TPL_ENABLE_DLlib=ON
-- Setting TeuchosCore_ENABLE_Boost=ON since TPL_ENABLE_Boost=ON
-- Setting TeuchosCore_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON
-- Setting TeuchosKokkosComm_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON
-- Setting Epetra_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON
-- Setting Zoltan_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON
-- Setting Zoltan_ENABLE_METIS=ON since TPL_ENABLE_METIS=ON
-- Setting Zoltan_ENABLE_ParMETIS=ON since TPL_ENABLE_ParMETIS=ON
-- Setting Zoltan_ENABLE_Zlib=ON since TPL_ENABLE_Zlib=ON
-- Setting TpetraCore_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON
-- Setting Amesos_ENABLE_ParMETIS=ON since TPL_ENABLE_ParMETIS=ON
-- Setting Pamgen_ENABLE_Boost=ON since TPL_ENABLE_Boost=ON
-- Setting Zoltan2_ENABLE_METIS=ON since TPL_ENABLE_METIS=ON
-- Setting Zoltan2_ENABLE_ParMETIS=ON since TPL_ENABLE_ParMETIS=ON
-- Setting Ifpack_ENABLE_Boost=ON since TPL_ENABLE_Boost=ON
-- Setting ML_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON
-- Setting ML_ENABLE_METIS=ON since TPL_ENABLE_METIS=ON
-- Setting ML_ENABLE_ParMETIS=ON since TPL_ENABLE_ParMETIS=ON
-- Setting Amesos2_ENABLE_MPI=ON since TPL_ENABLE_MPI=ON
-- Setting Amesos2_ENABLE_ParMETIS=ON since TPL_ENABLE_ParMETIS=ON
-- Setting Amesos2_ENABLE_METIS=ON since TPL_ENABLE_METIS=ON
-- Setting Intrepid2_ENABLE_Boost=ON since TPL_ENABLE_Boost=ON
-- Setting MueLu_ENABLE_Boost=ON since TPL_ENABLE_Boost=ON

Enabling TPLs based on <TRIBITS_PACKAGE>_ENABLE_<TPL>=ON if TPL is not explicitly disabled ...

Set cache entries for optional packages/TPLs and tests/examples for packages actually enabled ...

Enabling all parent packages that have at least one subpackage enabled ...

Final set of enabled packages:  Gtest Kokkos Teuchos RTOp Sacado Epetra Zoltan Shards Triutils Tpetra EpetraExt Thyra Xpetra Isorropia AztecOO Galeri Amesos Pamgen Zoltan2 Ifpack ML Belos Amesos2 Anasazi Ifpack2 Stratimikos Intrepid2 MueLu 28

Final set of enabled SE packages:  Gtest KokkosCore KokkosContainers KokkosAlgorithms Kokkos TeuchosCore TeuchosParameterList TeuchosComm TeuchosNumerics TeuchosRemainder TeuchosKokkosCompat TeuchosKokkosComm Teuchos RTOp Sacado Epetra Zoltan Shards Triutils TpetraClassic TpetraKernels TpetraTSQR TpetraCore Tpetra EpetraExt ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Thyra Xpetra Isorropia AztecOO Galeri Amesos Pamgen Zoltan2 Ifpack ML Belos Amesos2 Anasazi Ifpack2 Stratimikos Intrepid2 MueLu 46

Final set of non-enabled packages:  ThreadPool GlobiPack Domi OptiPack Pliris Claps ShyLU SEACAS Trios Komplex FEI Teko TriKota Intrepid STK Phalanx NOX Moertel Rythmos Stokhos ROL Piro Panzer PyTrilinos NewPackage MeshingGenie TrilinosCouplings Pike 28

Final set of non-enabled SE packages:  ThreadPool KokkosExample GlobiPack Domi OptiPack Pliris Claps ShyLUHTS ShyLUTacho ShyLUBDDC ShyLUBasker ShyLUFastILU ShyLUCore ShyLU SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASNemesis SEACASIoss SEACASChaco SEACASAprepro_lib SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASPLT SEACASAlgebra SEACASAprepro SEACASBlot SEACASConjoin SEACASEjoin SEACASEpu SEACASExo2mat SEACASExodiff SEACASExomatlab SEACASExotxt SEACASExo_format SEACASEx1ex2v2 SEACASFastq SEACASGjoin SEACASGen3D SEACASGenshell SEACASGrepos SEACASGrope SEACASMapvarlib SEACASMapvar SEACASMapvar-kd SEACASMat2exo SEACASNemslice SEACASNemspread SEACASNumbers SEACASTxtexo SEACASEx2ex1v2 SEACAS Trioscommsplitter Triossupport Triosnnti Triosnssi Triosprograms Triosexamples Triostests Triosnetcdf-service Trios Komplex FEI Teko TriKota Intrepid STKClassic STKUtil STKTopology STKMesh STKIO STKUnit_test_utils STKSearch STKSearchUtil STKTransfer STKUnit_tests STKDoc_tests STKExp STKExprEval STK Phalanx NOX Moertel Rythmos Stokhos ROL Piro PanzerCore PanzerDofMgr PanzerDiscFE PanzerAdaptersSTK Panzer PyTrilinos NewPackage MeshingGenie TrilinosCouplings PikeBlackBox PikeImplicit Pike 102

Final set of enabled TPLs:  MPI BLAS LAPACK Boost METIS ParMETIS Zlib Netcdf BoostLib DLlib 10

Final set of non-enabled TPLs:  MKL yaml-cpp Peano CUDA CUSPARSE Thrust Cusp TBB Pthread HWLOC QTHREAD BinUtils ARPREC QD Scotch OVIS gpcd CGNS PuLP TopoManager LibTopoMap PaToH CppUnit ADOLC ADIC TVMET MF ExodusII Nemesis XDMF HDF5 y12m SuperLUDist SuperLUMT SuperLU Cholmod UMFPACK MA28 AMD CSparse HYPRE PETSC BLACS SCALAPACK MUMPS PARDISO_MKL PARDISO Oski TAUCS ForUQTK Dakota HIPS MATLAB CASK SPARSKIT QT gtest BoostAlbLib OpenNURBS Portals CrayPortals Gemini InfiniBand BGPDCMF BGQPAMI Pablo HPCToolkit Pnetcdf Clp GLPK qpOASES Matio PAPI MATLABLib Eigen X11 Lemon GLM quadmath CAMAL RTlib AmgX CGAL CGALCore VTune TASMANIAN ArrayFireCPU SimMesh SimModel SimParasolid SimAcis SimField 92

Setting up export dependencies for all enabled SE packages ...

Probing the environment ...

-- USE_XSDK_DEFAULTS='FALSE'
-- BUILD_SHARED_LIBS='ON'
-- CMAKE_BUILD_TYPE='RELEASE'
-- MPI_USE_COMPILER_WRAPPERS='ON'
-- Leaving current CMAKE_C_COMPILER=mpicc since it is already set!
-- Leaving current CMAKE_CXX_COMPILER=mpicxx since it is already set!
-- Leaving current CMAKE_Fortran_COMPILER=mpif90 since it is already set!
-- MPI_EXEC='mpirun'
-- MPI_EXEC='/ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpirun'
-- The C compiler identification is GNU 5.4.0
-- Check for working C compiler: /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpicc
-- Check for working C compiler: /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpicc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- CMAKE_C_COMPILER_ID='GNU'
-- CMAKE_C_COMPILER_VERSION='5.4.0'
-- The CXX compiler identification is GNU 5.4.0
-- Check for working CXX compiler: /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpicxx
-- Check for working CXX compiler: /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpicxx -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- CMAKE_CXX_COMPILER_ID='GNU'
-- CMAKE_CXX_COMPILER_VERSION='5.4.0'
-- The Fortran compiler identification is GNU 5.4.0
-- Check for working Fortran compiler: /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpif90
-- Check for working Fortran compiler: /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpif90  -- works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpif90 supports Fortran 90
-- Checking whether /ascldap/users/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpif90 supports Fortran 90 -- yes
-- Trilinos_SET_INSTALL_RPATH='TRUE'
-- CMAKE_INSTALL_RPATH_USE_LINK_PATH='TRUE'
-- Setting default for CMAKE_INSTALL_RPATH pointing to Trilinos_INSTALL_LIB_DIR
-- CMAKE_INSTALL_RPATH='/ascldap/users/jjellio/install/Trilinos/gcc-5.4.0_openmpi-1.10.4_cuda-8.0.44_dynamic_opt-omp_muelu-experimental/lib'
-- Looking for C++ include sys/time.h
-- Looking for C++ include sys/time.h - found
-- Looking for C++ include time.h
-- Looking for C++ include time.h - found
-- Looking for C++ include stdint.h
-- Looking for C++ include stdint.h - found
-- Looking for C++ include inttypes.h
-- Looking for C++ include inttypes.h - found
-- Found Perl: /ascldap/users/projects/pwr8-rhel73-lsf/perl/5.22.1/bin/perl (found version "5.22.1") 
-- Fortran name mangling: LOWER UNDER
-- Detecting Fortran/C Interface
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling
-- Verifying Fortran/CXX Compiler Compatibility
-- Verifying Fortran/CXX Compiler Compatibility - Success
-- C++11 Flags already set: '-std=c++11'
-- Performing Test CXX11_CONSECUTIVE_RIGHT_ANGLE_BRACKETS
-- Performing Test CXX11_CONSECUTIVE_RIGHT_ANGLE_BRACKETS - Success
-- Performing Test CXX11_AUTOTYPEDVARIABLES
-- Performing Test CXX11_AUTOTYPEDVARIABLES - Success
-- Performing Test CXX11_LAMBDAS
-- Performing Test CXX11_LAMBDAS - Success
-- Trilinos_ENABLE_CXX11=ON
-- Try OpenMP C flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP CXX flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Try OpenMP Fortran flag = [-fopenmp]
-- Performing Test OpenMP_FLAG_DETECTED
-- Performing Test OpenMP_FLAG_DETECTED - Success
-- Found OpenMP: -fopenmp  
-- Performing Test MATH_LIBRARY_IS_SUPPLIED
-- Performing Test MATH_LIBRARY_IS_SUPPLIED - Success
-- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN
-- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISNAN - Success
-- Performing Test FINITE_VALUE_HAVE_STD_ISNAN
-- Performing Test FINITE_VALUE_HAVE_STD_ISNAN - Success
-- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF
-- Performing Test FINITE_VALUE_HAVE_GLOBAL_ISINF - Success
-- Performing Test FINITE_VALUE_HAVE_STD_ISINF
-- Performing Test FINITE_VALUE_HAVE_STD_ISINF - Success
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE) 

Getting information for all enabled TPLs ...

Processing enabled TPL: MPI (enabled explicitly, disable with -DTPL_ENABLE_MPI=OFF)
Processing enabled TPL: BLAS (enabled explicitly, disable with -DTPL_ENABLE_BLAS=OFF)
-- Searching for libs in BLAS_LIBRARY_DIRS='/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlC/13.1.4/lib;/home/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib;/ascldap/users/projects/pwr8-rhel72/ibm/xl/lib;/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib'
-- Searching for a lib in the set "esslsmp":
--   Searching for lib 'esslsmp' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64/libesslsmp.so'
-- Searching for a lib in the set "xlf90_r":
--   Searching for lib 'xlf90_r' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlf90_r.so'
-- Searching for a lib in the set "xlfmath":
--   Searching for lib 'xlfmath' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlfmath.so'
-- Searching for a lib in the set "xlopt":
--   Searching for lib 'xlopt' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlopt.a'
-- Searching for a lib in the set "xl":
--   Searching for lib 'xl' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxl.a'
-- Searching for a lib in the set "xlsmp":
--   Searching for lib 'xlsmp' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlsmp.so'
-- Searching for a lib in the set "xlomp_ser":
--   Searching for lib 'xlomp_ser' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlomp_ser.so'
-- Searching for a lib in the set "openblas":
--   Searching for lib 'openblas' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/libopenblas.so'
-- TPL_BLAS_LIBRARIES='/ascldap/users/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64/libesslsmp.so;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlf90_r.so;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlfmath.so;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlopt.a;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxl.a;/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlsmp.so;/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlomp_ser.so;/ascldap/users/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/libopenblas.so'
Processing enabled TPL: LAPACK (enabled explicitly, disable with -DTPL_ENABLE_LAPACK=OFF)
-- Searching for libs in LAPACK_LIBRARY_DIRS='/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib;/home/projects/pwr8-rhel73-lsf/ibm/xl/xlC/13.1.4/lib;/home/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib;/ascldap/users/projects/pwr8-rhel72/ibm/xl/lib;/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib'
-- Searching for a lib in the set "esslsmp":
--   Searching for lib 'esslsmp' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64/libesslsmp.so'
-- Searching for a lib in the set "xlf90_r":
--   Searching for lib 'xlf90_r' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlf90_r.so'
-- Searching for a lib in the set "xlfmath":
--   Searching for lib 'xlfmath' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlfmath.so'
-- Searching for a lib in the set "xlopt":
--   Searching for lib 'xlopt' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlopt.a'
-- Searching for a lib in the set "xl":
--   Searching for lib 'xl' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxl.a'
-- Searching for a lib in the set "xlsmp":
--   Searching for lib 'xlsmp' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlsmp.so'
-- Searching for a lib in the set "xlomp_ser":
--   Searching for lib 'xlomp_ser' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlomp_ser.so'
-- Searching for a lib in the set "openblas":
--   Searching for lib 'openblas' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/libopenblas.so'
-- TPL_LAPACK_LIBRARIES='/ascldap/users/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib64/libesslsmp.so;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlf90_r.so;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlfmath.so;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxlopt.a;/ascldap/users/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.4/lib/libxl.a;/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlsmp.so;/ascldap/users/projects/pwr8-rhel72/ibm/xl/xlsmp/4.1.4/lib/libxlomp_ser.so;/ascldap/users/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/libopenblas.so'
Processing enabled TPL: Boost (enabled explicitly, disable with -DTPL_ENABLE_Boost=OFF)
-- Searching for headers in Boost_INCLUDE_DIRS='/home/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- Searching for a header file in the set "boost/version.hpp":
--   Searching for header 'boost/version.hpp' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include/boost/version.hpp'
-- Searching for a header file in the set "boost/mpl/at.hpp":
--   Searching for header 'boost/mpl/at.hpp' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include/boost/mpl/at.hpp'
-- Found TPL 'Boost' include dirs '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- TPL_Boost_INCLUDE_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
Processing enabled TPL: METIS (enabled explicitly, disable with -DTPL_ENABLE_METIS=OFF)
-- Searching for libs in METIS_LIBRARY_DIRS='/home/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/lib'
-- Searching for a lib in the set "metis":
--   Searching for lib 'metis' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/lib/libmetis.a'
-- TPL_METIS_LIBRARIES='/ascldap/users/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/lib/libmetis.a'
-- Searching for headers in METIS_INCLUDE_DIRS='/home/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/include'
-- Searching for a header file in the set "metis.h":
--   Searching for header 'metis.h' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/include/metis.h'
-- Found TPL 'METIS' include dirs '/ascldap/users/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/include'
-- TPL_METIS_INCLUDE_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/include'
Processing enabled TPL: ParMETIS (enabled explicitly, disable with -DTPL_ENABLE_ParMETIS=OFF)
-- Searching for libs in ParMETIS_LIBRARY_DIRS='/home/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib;/home/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/lib'
-- Searching for a lib in the set "parmetis":
--   Searching for lib 'parmetis' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libparmetis.a'
-- Searching for a lib in the set "metis":
--   Searching for lib 'metis' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/lib/libmetis.a'
-- TPL_ParMETIS_LIBRARIES='/ascldap/users/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libparmetis.a;/ascldap/users/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/lib/libmetis.a'
-- Searching for headers in ParMETIS_INCLUDE_DIRS='/home/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include;/home/projects/pwr8-rhel73-lsf/metis/5.0.1/gcc/5.4.0/include'
-- Searching for a header file in the set "parmetis.h":
--   Searching for header 'parmetis.h' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include/parmetis.h'
-- Found TPL 'ParMETIS' include dirs '/ascldap/users/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- TPL_ParMETIS_INCLUDE_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/parmetis/4.0.3/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- Performing Test HAVE_PARMETIS_VERSION_4_0_3
-- Performing Test HAVE_PARMETIS_VERSION_4_0_3 - Success
Processing enabled TPL: Zlib (enabled explicitly, disable with -DTPL_ENABLE_Zlib=OFF)
-- Searching for libs in Zlib_LIBRARY_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/lib'
-- Searching for a lib in the set "z":
--   Searching for lib 'z' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/lib/libz.a'
-- TPL_Zlib_LIBRARIES='/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/lib/libz.a'
-- Searching for headers in Zlib_INCLUDE_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/lib'
-- Searching for a header file in the set "zlib.h":
--   Searching for header 'zlib.h' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/include/zlib.h'
-- Found TPL 'Zlib' include dirs '/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/include'
-- TPL_Zlib_INCLUDE_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/include'
Processing enabled TPL: Netcdf (enabled explicitly, disable with -DTPL_ENABLE_Netcdf=OFF)
-- Searching for libs in Netcdf_LIBRARY_DIRS='/home/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib;/home/projects/pwr8-rhel73-lsf/pnetcdf/1.6.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib;/home/projects/pwr8-rhel73-lsf/hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib;/home/projects/pwr8-rhel73-lsf/zlib/1.2.8/lib'
-- Searching for a lib in the set "netcdf":
--   Searching for lib 'netcdf' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libnetcdf.a'
-- Searching for a lib in the set "pnetcdf":
--   Searching for lib 'pnetcdf' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/pnetcdf/1.6.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libpnetcdf.a'
-- Searching for a lib in the set "hdf5_hl":
--   Searching for lib 'hdf5_hl' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libhdf5_hl.a'
-- Searching for a lib in the set "hdf5":
--   Searching for lib 'hdf5' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libhdf5.a'
-- Searching for a lib in the set "z":
--   Searching for lib 'z' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/lib/libz.a'
-- TPL_Netcdf_LIBRARIES='/ascldap/users/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libnetcdf.a;/ascldap/users/projects/pwr8-rhel73-lsf/pnetcdf/1.6.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libpnetcdf.a;/ascldap/users/projects/pwr8-rhel73-lsf/hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libhdf5_hl.a;/ascldap/users/projects/pwr8-rhel73-lsf/hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libhdf5.a;/ascldap/users/projects/pwr8-rhel73-lsf/zlib/1.2.8/lib/libz.a'
-- Searching for headers in Netcdf_INCLUDE_DIRS='/home/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include;/home/projects/pwr8-rhel73-lsf/hdf5/1.8.17/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- Searching for a header file in the set "netcdf.h":
--   Searching for header 'netcdf.h' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include/netcdf.h'
-- Found TPL 'Netcdf' include dirs '/ascldap/users/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- TPL_Netcdf_INCLUDE_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/netcdf/4.4.1/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
Processing enabled TPL: BoostLib (enabled explicitly, disable with -DTPL_ENABLE_BoostLib=OFF)
-- Searching for libs in BoostLib_LIBRARY_DIRS='/home/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib'
-- Searching for a lib in the set "boost_program_options":
--   Searching for lib 'boost_program_options' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libboost_program_options.so'
-- Searching for a lib in the set "boost_system":
--   Searching for lib 'boost_system' ...
--     Found lib '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libboost_system.so'
-- TPL_BoostLib_LIBRARIES='/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libboost_program_options.so;/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/lib/libboost_system.so'
-- Searching for headers in BoostLib_INCLUDE_DIRS='/home/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- Searching for a header file in the set "boost/version.hpp":
--   Searching for header 'boost/version.hpp' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include/boost/version.hpp'
-- Searching for a header file in the set "boost/mpl/at.hpp":
--   Searching for header 'boost/mpl/at.hpp' ...
--     Found header '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include/boost/mpl/at.hpp'
-- Found TPL 'BoostLib' include dirs '/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
-- TPL_BoostLib_INCLUDE_DIRS='/ascldap/users/projects/pwr8-rhel73-lsf/boost/1.60.0/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/include'
Processing enabled TPL: DLlib (enabled explicitly, disable with -DTPL_ENABLE_DLlib=OFF)
-- Searching for libs in DLlib_LIBRARY_DIRS=''
-- Searching for a lib in the set "dl":
--   Searching for lib 'dl' ...
--     Found lib '/usr/lib64/libdl.so'
-- TPL_DLlib_LIBRARIES='/usr/lib64/libdl.so'

Setting up testing support ...

-- CTEST_DROP_METHOD='http'
-- CTEST_DROP_SITE='testing.sandia.gov'
-- CTEST_PROJECT_NAME='Trilinos'
-- CTEST_DROP_LOCATION='/cdash/submit.php?project=Trilinos'
-- CTEST_TRIGGER_SITE=''
-- CTEST_DROP_SITE_CDASH='TRUE'

Configuring individual enabled Trilinos packages ...

Processing enabled package: Gtest (Libs)
Processing enabled package: Kokkos (Core, Containers, Algorithms, Tests, Examples)
Processing enabled package: Teuchos (Core, ParameterList, Comm, Numerics, Remainder, KokkosCompat, KokkosComm, Tests, Examples)
-- Performing Test HAVE_GCC_ABI_DEMANGLE
-- Performing Test HAVE_GCC_ABI_DEMANGLE - Success
-- Performing Test HAVE_TEUCHOSCORE_BOOST_IS_POLYMORPHIC
-- Performing Test HAVE_TEUCHOSCORE_BOOST_IS_POLYMORPHIC - Success
-- Performing Test HAVE_TEUCHOS_BLASFLOAT
-- Performing Test HAVE_TEUCHOS_BLASFLOAT - Success
-- Performing Test LAPACK_SLAPY2_WORKS
-- Performing Test LAPACK_SLAPY2_WORKS - Success
-- Performing Test HAVE_TEUCHOS_LAPACKLARND
-- Performing Test HAVE_TEUCHOS_LAPACKLARND - Failed
-- Performing Test HAVE_CXX_ATTRIBUTE_CONSTRUCTOR
-- Performing Test HAVE_CXX_ATTRIBUTE_CONSTRUCTOR - Success
-- C++ compiler supports __attribute__((constructor)) syntax
-- Performing Test HAVE_CXX_ATTRIBUTE_WEAK
-- Performing Test HAVE_CXX_ATTRIBUTE_WEAK - Success
-- C++ compiler supports __attribute__((weak)) syntax and testing weak functions
-- Performing Test HAVE_CXX_PRAGMA_WEAK
-- Performing Test HAVE_CXX_PRAGMA_WEAK - Failed
-- C++ compiler does NOT support #pragma weak syntax and testing weak functions
Processing enabled package: RTOp (Libs)
Processing enabled package: Sacado (Libs)
-- Performing Test HAS_C99_TR1_CMATH
-- Performing Test HAS_C99_TR1_CMATH - Success
Processing enabled package: Epetra (Libs, Tests, Examples)
CMake Warning at cmake/tribits/core/package_arch/TribitsAddExecutable.cmake:487 (MESSAGE):
  WARNING: Passing extra defines through 'DEFINES' -DEPETRA_LIB_EXPORTS_MODE
  is deprecated.  Instead, pass them through 'TARGET_DEFINES'.  The 'DEFINES'
  argument was incorrectly implemented by calling ADD_DEFINITIONS() which has
  directory scope and not function scope as was documented.  This resulted in
  confusing behavior.  If one wishes to set defines at the directly level,
  just call ADD_DEFINITIONS() directly.
Call Stack (most recent call first):
  cmake/tribits/core/package_arch/TribitsAddExecutableAndTest.cmake:68 (TRIBITS_ADD_EXECUTABLE)
  cmake/tribits/core/package_arch/TribitsAddExecutableAndTest.cmake:209 (TRIBITS_ADD_EXECUTABLE_WRAPPER)
  packages/epetra/test/FEVector/CMakeLists.txt:4 (TRIBITS_ADD_EXECUTABLE_AND_TEST)

CMake Warning at cmake/tribits/core/package_arch/TribitsAddExecutable.cmake:487 (MESSAGE):
  WARNING: Passing extra defines through 'DEFINES' -DEPETRA_LIB_EXPORTS_MODE
  is deprecated.  Instead, pass them through 'TARGET_DEFINES'.  The 'DEFINES'
  argument was incorrectly implemented by calling ADD_DEFINITIONS() which has
  directory scope and not function scope as was documented.  This resulted in
  confusing behavior.  If one wishes to set defines at the directly level,
  just call ADD_DEFINITIONS() directly.
Call Stack (most recent call first):
  cmake/tribits/core/package_arch/TribitsAddExecutableAndTest.cmake:68 (TRIBITS_ADD_EXECUTABLE)
  cmake/tribits/core/package_arch/TribitsAddExecutableAndTest.cmake:209 (TRIBITS_ADD_EXECUTABLE_WRAPPER)
  packages/epetra/test/FEVector/CMakeLists.txt:14 (TRIBITS_ADD_EXECUTABLE_AND_TEST)

-- Epetra_petra_power_method: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- Epetra_verySimple: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- Epetra_UG_Ex1: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- Epetra_UG_ExMaps: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- Epetra_petra_power_method_LL: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- Epetra_verySimple_LL: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
Processing enabled package: Zoltan (Libs)
Processing enabled package: Shards (Libs)
Processing enabled package: Triutils (Libs)
Processing enabled package: Tpetra (Classic, Kernels, TSQR, Core, Tests, Examples)
-- Tpetra execution space availability (ON means available): 
--   - Serial:  OFF
--   - Threads: OFF
--   - OpenMP:  ON
--   - Cuda:    OFF
-- Setting default Node to Kokkos::Compat::KokkosOpenMPWrapperNode.
Adding Experimental KokkosGraph features
-- Not enabling Tpetra/RTI
-- TpetraCore_TpetraUtils_UnitTests: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- TpetraCore_TpetraUtils_merge: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
Processing enabled package: EpetraExt (Libs, Tests, Examples)
-- EpetraExt_EpetraExt_MapColoring_test: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
-- EpetraExt_EpetraExt_MapColoring_test_LL: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
Processing enabled package: Thyra (Core, EpetraAdapters, EpetraExtAdapters, TpetraAdapters)
Processing enabled package: Xpetra (Libs, Tests, Examples)
-- Xpetra support for 32 bit Epetra enabled.
--    Xpetra_Epetra_NO_32BIT_GLOBAL_INDICES=OFF
-- Xpetra support for 64 bit Epetra enabled.
--    Xpetra_Epetra_NO_64BIT_GLOBAL_INDICES=OFF
Processing enabled package: Isorropia (Libs, Tests, Examples)
Processing enabled package: AztecOO (Libs)
Processing enabled package: Galeri (Libs, Tests, Examples)
-- Galeri_VerySimple: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
CMake Warning at cmake/tribits/core/package_arch/TribitsAddExecutable.cmake:487 (MESSAGE):
  WARNING: Passing extra defines through 'DEFINES'
  -DXPETRA_TEST_USE_LONGLONG_GO is deprecated.  Instead, pass them through
  'TARGET_DEFINES'.  The 'DEFINES' argument was incorrectly implemented by
  calling ADD_DEFINITIONS() which has directory scope and not function scope
  as was documented.  This resulted in confusing behavior.  If one wishes to
  set defines at the directly level, just call ADD_DEFINITIONS() directly.
Call Stack (most recent call first):
  cmake/tribits/core/package_arch/TribitsAddExecutableAndTest.cmake:68 (TRIBITS_ADD_EXECUTABLE)
  cmake/tribits/core/package_arch/TribitsAddExecutableAndTest.cmake:209 (TRIBITS_ADD_EXECUTABLE_WRAPPER)
  packages/galeri/example-xpetra/CMakeLists.txt:15 (TRIBITS_ADD_EXECUTABLE_AND_TEST)

Processing enabled package: Amesos (Libs)
Processing enabled package: Pamgen (Libs)
Processing enabled package: Zoltan2 (Libs, Tests, Examples)
Processing enabled package: Ifpack (Libs)
Processing enabled package: ML (Libs)
Processing enabled package: Belos (Libs, Tests, Examples)
Processing enabled package: Amesos2 (Libs, Tests, Examples)
-- Amesos2_klu2_simple: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
Processing enabled package: Anasazi (Libs, Tests, Examples)
-- Anasazi_Epetra_BKS_QEVP_test: NOT added test because TPL_ENABLE_MPI='ON' and COMM='serial'!
Processing enabled package: Ifpack2 (Libs, Tests, Examples)
Processing enabled package: Stratimikos (Libs)
Processing enabled package: Intrepid2 (Libs)
Processing enabled package: MueLu (Libs, Tests, Examples)
-- Enabling MueLu experimental code
-- MueLu ETI support enabled
-- <float,   int, int>       : OFF
-- <double,  int, int>       : ON
-- <double,  int, long>      : OFF
-- <double,  int, long long> : ON
-- <complex, int, int>       : OFF
-- HAVE_MUELU_SERIAL       : ON
-- HAVE_MUELU_PTHREAD      : OFF
-- HAVE_MUELU_OPENMP       : ON
-- HAVE_MUELU_CUDA         : OFF
-- Could NOT find Doxygen (missing:  DOXYGEN_EXECUTABLE) 

Processing explicit instantiation support for enabled packages ...

Processing ETI support: Ifpack2
-- Ifpack2: Processing ETI / test support
-- Enabled Scalar types:        double
-- Enabled LocalOrdinal types:  int
-- Enabled GlobalOrdinal types: int|long long
-- Enabled Node types:          Kokkos::Compat::KokkosOpenMPWrapperNode
-- User/Downstream ETI set: 
-- Excluded type combinations: 
-- Set of enabled types, before exclusions: S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long}
Processing ETI support: TpetraCore
-- TpetraCore: Processing ETI / test support
-- Enabled Scalar types:        int|long long|double
-- Enabled LocalOrdinal types:  int
-- Enabled GlobalOrdinal types: int|long long
-- Enabled Node types:          Kokkos::Compat::KokkosOpenMPWrapperNode
-- Excluded type combinations: :
-- Set of enabled types, before exclusions: S={int} LO={int} GO={int} N={Kokkos::Compat::KokkosOpenMPWrapperNode};S={int} LO={int} GO={long long} N={Kokkos::Compat::KokkosOpenMPWrapperNode};S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long};S={int} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={int} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long};S={long long} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={long long} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long};S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={int};S={double} N={Kokkos::Compat::KokkosOpenMPWrapperNode} LO={int} GO={long long}
Processing ETI support: TpetraKernels
-- TpetraKernels: Processing ETI / test support
-- Enabled Scalar types:       int|long long|double
-- Enabled LocalOrdinal types: int
-- Enabled Device types:       Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>
-- Set of enabled types, before exclusions: S={int} LO={int} D={Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>};S={long long} LO={int} D={Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>};S={double} LO={int} D={Kokkos::Device<Kokkos::OpenMP, Kokkos::HostSpace>}

Set up for creating a distribution ...

Finished configuring Trilinos!

-- Configuring done

[hkthorn]

It seems extremely unlucky that Belos is the only package that has a test that is using an BLAS/LAPACK routine that passes in a character. James can you insert the error from the failed PseudoBlockCG test, just to be sure that error is coming from LAPACK. Thanks!

[krcb]

@jjellio is it possible to not link in ESSL, that is only use OpenBLAS? It would be useful to know if this problem originates from the combination of the two libraries, i.e. some kind of namespace collision. We have seen something like this behavior before when we have a C++/Fortran interface and two different versions of the same library.

[mhoemmen]

@hkthorn Lots of BLAS routines take a character. I would really like to use GEMV in classical Gram-Schimidt, for example, and it takes a character (whether to apply the transpose or conjugate transpose of the matrix).

[jjellio]

I think I fixed this. The function constant 'one' is being optimized oddly.

Ctest declared passed, even though the processes were failing. Please ignore =(

[jjellio]

Ctest declared passed, even though the processes were failing. Please ignore =(

[mhoemmen]

@jjellio Wait, wait? Does this work everywhere or just for POWER + OpenBLAS + GCC?

[jjellio]

@mhoemmen LDZ should be an int. That is how it is defined: https://github.com/trilinos/Trilinos/blob/f0350316239aaf41e8e6b82378612aaedf9cc72c/packages/teuchos/numerics/src/Teuchos_LAPACK.cpp#L735

I used N since it was around. It needs to be any integer >=1 per the Lapack reference. 'Z' is not referenced, the value should be a dummy. The literal one is probably getting optimized in some odd way.

[mhoemmen]

@jjellio I'm confused. Is this a fix that only works for OpenBLAS+POWER+GCC, or a fix that works everywhere?

[jjellio]

Hang on - ctest is lying to me.

[jjellio]

@mhoemmen I shouldn't believe ctest apparently. The MPI process was not launching because I had <4 processes (slots) available. Somehow, that merits a 'passed' status.

No, this does not fix anything.

jjellio@ride16:~/build/trilinos-gcc-pw8/packages/belos/tpetra/test/BlockCG$ /ascldap/users/jjellio/build/trilinos-gcc-pw8/packages/belos/tpetra/test/BlockCG/Belos_Tpetra_PseudoBlockCG_hb_test.exe --verbose
Teuchos::GlobalMPISession::GlobalMPISession(): started processor with name ride16 and rank 0!
Input parameter list: 
Block Size = 1   [unused]
Maximum Iterations = 1805
Convergence Tolerance = 1e-05
Verbosity = 57
Estimate Condition Number = 1

Belos parameter list: 
Maximum Iterations = 1805   [unused]
Verbosity = 57   [unused]
Output Frequency = -1   [unused]
Convergence Tolerance = 1e-05   [unused]

Dimension of matrix: 1806
Number of right-hand sides: 1
Block size used by solver: 1
Max number of CG iterations: 1805
Relative residual tolerance: 1e-05

Belos::StatusTestGeneralOutput: Passed
  (Num calls,Mod test,State test): (909, 1, Passed)
   Passed.......OR Combination -> 
     OK...........Number of Iterations = 908 < 1805
     Converged....(2-Norm Imp Res Vec) / (2-Norm Res0)
                  residual [ 0 ] = 8.99515e-06 < 1e-05

Passed.......OR Combination -> 
  OK...........Number of Iterations = 908 < 1805
  Converged....(2-Norm Imp Res Vec) / (2-Norm Res0)
               residual [ 0 ] = 8.99515e-06 < 1e-05

================================================================================

                      TimeMonitor results over 1 processor

Timer Name                                     Global time (num calls)    
--------------------------------------------------------------------------------
Belos: Operation Op*x                          0.02266 (909)              
Belos: Operation Prec*x                        0 (0)                      
Belos: PseudoBlockCGSolMgr total solve time    0.1734 (1)                 
================================================================================
 ** On entry to DSTEQR parameter number  1 had an illegal value

p=0: *** Caught standard std::exception of type 'std::logic_error' :

 /ascldap/users/jjellio/src/Trilinos/packages/belos/src/BelosPseudoBlockCGSolMgr.hpp:1051:

 Throw number = 1

 Throw test that evaluated to true: info < 0

 Belos::PseudoBlockCGSolMgr::compute_condnum_tridiag_sym: LAPACK's _STEQR failed with info = -1 < 0.  This suggests there might be a bug in the way Belos is calling LAPACK.  Please report this to the Belos developers.

[hkthorn]

@mhoemmen @jjellio Since only the eigenvalues are being requested and not the eigenvectors, the argument of "1" for ldz is fine since "z" is being ignored. The error (-1) indicates that the issue is with the first argument, that is the character.

As I said before, it seems so unusual that only the Belos pseudo-block CG is calling an BLAS/LAPACK routine with a character in the test suite. In fact, it isn't the only one calling a BLAS/LAPACK routine with a character. These kinds of calls are riddled throughout the SerialDenseMatrix/Solver classes and are well tested in the test/DenseMatrix subdirectory of the Teuchos tests. If those tests are passing, which seems to be the case, then this issue might be something else. I have, on very rare occasion seen a badly named variable or function name have a symbol conflict and wreak mysterious havoc. On the off chance this may be the case here. If you have a second @jjellio can you directly replace the char_N with a 'N' in the call to lapack.STEQR and make sure the failure is still occurring.

[jjellio]

@hkthorn @mhoemmen let me dig a bit more. I also think it is very odd Teuchos Lapack/BLAS unit tests + most of trilinos is working. If the ABI was brazenly wrong, it seems we should have compile time or valgrind errors, but I am not seeing any.

I got this working last night, and the problem seems to be memory corruption, yet valgrind comes back clean. It seems odd Valgrind would be happy if the function is being called incorrectly.

Also, I do not know if the eigenvalues returned make sense. I am adding steqr to the Lapack unit tests, and plan to use a diagonal matrix so the eigenvalues are obvious, and see if that simple test is correct.

To get it working, I assigned values to the dummy scalar arrays and allocated the workspace on the heap with size > 1. I also placed a cout statement in Teuchos::Lapack, before the wrapped LAPACK call. The fact things work when cout's are used seems inline with stack/memory corruption. It is also possible Valgrind is not well tested on power8.

@krcb I think this could be done with OpenBLAS alone, it does provide a full Lapack/BLAS implementation. The function in question is not provided by ESSL, but it is possible there is some some symbol problems with ESSL+OpenBLAS, e.g., the lapack_utils functions may expect different inputs, and so the checks at the start of the routine may be failing even through everything is correct.

[jwillenbring]

@crtrott was interested in this issue it sounded like.

[jjellio]

I've written a short unit test in Teuchos' Lapack test file. I construct a diagonal, as d[i] = DIAG_SZ - i; where DIAG_SZ > 1; For the off diagonal, I simply put epsilon.

If STEQR works, then on output diag will be sorted in reverse order of how I construct it. e.g., d[0] = DIAG_SZ, and d[DIAG_SZ-1] = 1; The largest eigenvalue should be DIAG_SZ, and it should be in d[0]. I test this, and this works.... but only if I do some of the shenanigans I discuss above. E.g., cout in the wrapper, and allocate the workspace on the heap.

Also, the variable for the number of failed tests goes completely haywire, which would seem reasonable if the stack is getting wrecked. I'll work a bit more later, got a meeting now.

Teuchos in Trilinos 12.11 (Dev)

GESV test ... passed!
LAPY2 test ... passed!
ILAENV test ... passed!
STEQR test ... N= 1031
Char_N = N
Passed! ( Lambda min: expected 1, computed 1; Lambda max: expected 1031, computed 1031)
Number of failed tests: 70367862787936
End Result: TEST FAILED

Simple test is:

  {
    if (verbose) std::cout << "STEQR test ... ";

    typedef double ScalarType;
    typedef Teuchos::ScalarTraits<ScalarType> STS;
    typedef typename Teuchos::ScalarTraits<ScalarType>::magnitudeType MagnitudeType;

    const int DIAG_SZ = 1031;

    ScalarType * diagonal    = new ScalarType[DIAG_SZ];
    ScalarType * subdiagonal = new ScalarType[DIAG_SZ-1];

    for (int i=0; i < DIAG_SZ; ++i) {
      diagonal[i] = DIAG_SZ - i;
      subdiagonal[i] = STS::eps () * i;
    }

   ScalarType expected_lambda_min = STS::one ();
   ScalarType expected_lambda_max = DIAG_SZ;

   int dont_call_me_info = 0;
   const int dummy_ldz = 1;
   ScalarType scalar_dummy = -1.0;
   char char_N = 'N';
   Teuchos::LAPACK<int,ScalarType> lapack;
   const int N = DIAG_SZ;
   MagnitudeType * mag_dummy = (MagnitudeType *) malloc(sizeof(MagnitudeType) * N);

   ScalarType lambda_min = STS::one ();
   ScalarType lambda_max = STS::one ();

   if( N > 2 ) {
     lapack.STEQR (char_N, N, diagonal, subdiagonal,
                   &scalar_dummy, dummy_ldz, mag_dummy, &dont_call_me_info);

     if (dont_call_me_info < 0) {
       if (verbose)
         std::cout << "STEQR: compute symmetric tridiagonal eigenvalues: "
                   << "LAPACK's _STEQR failed with info = "
                   << dont_call_me_info << " < 0.";

     numberFailedTests++;
     }
     lambda_min = diagonal[0];
     lambda_max = diagonal[N-1];
   }

    using std::fabs;
    bool good_lambda_min = (fabs (lambda_min - expected_lambda_min) <= 1.e-8);
    bool good_lambda_max = (fabs (lambda_max - expected_lambda_max) <= 1.e-8);

   if (good_lambda_min && good_lambda_max) {
      if (verbose) std::cout << "Passed! ( Lambda min: expected "
                             << expected_lambda_min << ", computed " << lambda_min
                             << "; Lambda max: expected " << expected_lambda_max << ", computed " << lambda_max <<  ")"
                             << std::endl;

    } else {
      if (verbose) std::cout << "FAILED ( Lambda min: expected "
                             << expected_lambda_min << ", computed " << lambda_min
                             << "; Lambda max: expected " << expected_lambda_max << ", computed " << lambda_max <<  ")"
                             << std::endl;
      numberFailedTests++;
    }
  }

[krcb]

@jjellio Was this test compiled against ESSL+OpenBLAS, or only OpenBLAS? If the former, does the corruption go away if the latter is chosen?

[jjellio]

@krcb, I do not believe ESSL provides STEQR. The code is linked against ESSL (then) OpenBLAS. I suppose, you could link Trilinos against OpenBLAS skipping ESSL all together. I haven't tried that.

I've had to work on some other things, but one thing I did do was build OpenBLAS myself and run the LAPACK test suite against the openblas lib. The Lapack test suite (provided in the openblas source distribution, under lapack-netlib) completes all tests with 100% numerical accuracy.

This looks like Teuchos using OpenBLAS incorrectly. The stack is getting trashed, hence it 'kinda' works, but messes up the variables on the stack at the caller. I didn't try @mhoemmen's suggested ABI fix.

Also worth pointing out: The Teuchos LAPACK unit test is probably not adequate, as it does not catch this. Even the simple test I show, can potentially pass. In November, I started work on being able to link Teuchos Lapack/BLAS + vendor BLAS against the Netlib LAPACK test suite. This would be the ultimate and best unit test for BLAS/LAPACK sanity. It is also a bit tricky to setup, because different LAPACK/BLAS vendors provide slightly different functions. E.g., the error handler/checker functions, which are not technically part of the spec. I believe Intel provides there own, and OpenBLAS does not provide one (Or I may have that reversed). With a little work, we could provide the necessary CMake glue to make this possible. This would also unveil the loss in accuracy from enabling something like -O3 vs -O3 -fp-model precise. The latter was my motivation when I originally did this last year.

[krcb]

As a sanity check, I just built OpenBLAS using gcc 4.9.3 and gfortran on a x86_64 machine. The OpenBLAS test suite passes. I then built Trilinos Teuchos Numerics and added @jjellio's test above. The tests associated with this also pass. I think we can therefore rule out a fundamental misuse of OpenBLAS by Teuchos. Colleagues here have, in the past, experienced (very) similar problems with ESSL/OpenBLAS on IBM BlueGene systems. This turned out to be issues involved in linking. My next step is going to be to get onto Ride/White and repeat this same experiment.

[krcb]

I have turned @jjellio's test of STEQR above into a standalone program and built on ride. After a bug fix to make sure that out of range memory wasn't being accessed, I built with several different combinations of ESSL/OpenBLAS. The following compile lines succeeded (i.e. executing the executable resulted in the test reported passed with sensible output):

/home/projects/pwr8-rhel73-lsf/gcc/5.4.0/bin/gcc main.cpp -L/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/ -lopenblas -lstdc++

/home/projects/pwr8-rhel73-lsf/gcc/5.4.0/bin/gcc main.cpp -L/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib -lessl -L/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.5/lib/ -lxlf90_r -lxl -L/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/ -lopenblas -lstdc++

/home/projects/pwr8-rhel73-lsf/openmpi/1.10.4/gcc/5.4.0/cuda/8.0.44/bin/mpicxx main.cpp -L/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib -lessl -L/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.5/lib/ -lxlf90_r -lxl -L/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.3.0/lib/ -lopenblas -lstdc++

/home/projects/pwr8-rhel73-lsf/ibm/xl/xlC/13.1.5/bin/xlc main.cpp -L/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib -lessl -L/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.5/lib/ -lxlf90_r -lxl -L/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/ibm/xl/13.1.3/lib -lopenblas -libmc++ -lstdc++

The only case where I've been able to get the test to compile and then fail with something like the error above is where I compile the test using xlc with gcc compiled OpenBLAS:

/home/projects/pwr8-rhel73-lsf/ibm/xl/xlC/13.1.5/bin/xlc main.cpp -L/home/projects/pwr8-rhel73-lsf/ibm/essl/5.4.0/lib -lessl -L/home/projects/pwr8-rhel73-lsf/ibm/xl/xlf/15.1.5/lib/ -lxlf90_r -lxl -L/home/projects/pwr8-rhel73-lsf/openblas/0.2.19/gcc/5.4.0/lib -lopenblas -libmc++ -lstdc++

Next, I'm going to try this same set of experiments with Teuchos.