search

CASL / Futility

VERA Fortran Utilities
Other
44 stars 20 forks source link

Enable Trilinos/TeuchosWrappersExt as a TPL #66

Closed bscollin closed 3 years ago

bscollin commented 6 years ago

In order to reduce build times, specifically in the CI, it is desirable to use a prebuilt version of Trilinos and TeuchosWrappersExt instead of building the packages. @bartlettroscoe has introduced a small patch to TriBITS which allows projects to switch between building the package and using the prebuilt version. This issue is to track the implementation of this feature.

Definition of Done Prototype of using Trillinos and TeuchosWrappersExt as prebuilt libraries is completed

bscollin commented 6 years ago

@bartlettroscoe, while we were talking this afternoon, I realized that some of the Dependencies listed in Futility aren't direct dependencies but indirect. So I went in and removed the dependency on Epetra and Amesos2. This change can be found in #67.

bartlettroscoe commented 6 years ago

@bscollin, I tried building Futility as is using the Trilinos version in 'develop' in github.com:trilinos/Trilinos.git at commit:

a203c5f "Merge remote-tracking branch 'rol-trilinos/develop' into develop"
Author: Denis Ridzal <dridzal@sandia.gov>
Date:   Fri Nov 3 15:14:49 2017 -0600 (49 minutes ago)

and if failed with:

In file included from /ascldap/users/rabartl/Futility.base/Futility/src/trilinos_interfaces/trilinos_mat_vec.cpp:10:0:
/ascldap/users/rabartl/Futility.base/Futility/src/trilinos_interfaces/trilinos_mat_vec.hpp:44:53: error: ‘OpenMP’ is not a member of ‘Kokkos’
     typedef Kokkos::Compat::KokkosDeviceWrapperNode<Kokkos::OpenMP> NO;
                                                     ^

...

What version of Trilinos is currently represented in the 'master' branch of the repo ners-arc-05.engin.umich.edu:/git-root/MPACT/Trilinos? If you provide the full git log --pretty=fuller -1 for the tip of that branch, I can match up the comment even with the different filtered branches.

bscollin commented 6 years ago

@bartlettroscoe our CI normally pulls the github version of Trilinos (git clone https://github.com/Trilinos/Trilinos). There is a build script in build_scripts/build_in_docker.sh that should be pretty easy to adapt.

bscollin commented 6 years ago

@bartlettroscoe, I was able to reproduce this issue but doing -DFutility_ENABLE_OPENMP=ON fixed it.

bartlettroscoe commented 6 years ago

Okay, configuring with -DFutility_ENABLE_OpenMP=ON resulted in a successful configure, build and test using the do-configure script:

#!/bin/bash
cmake \
-GNinja \
-DTPL_ENABLE_MPI=ON \
-DCMAKE_BUILD_TYPE=DEBUG \
-DFutility_ENABLE_OpenMP=ON \
-DFutility_ENABLE_Futility=ON \
../../../Futility

I did:

$ cd <builddir>/
$ ./do-configure
$ make
$ ctest -j16

and it returned:


100% tests passed, 0 tests failed out of 55

Label Time Summary:
CTeuchos      =   0.63 sec (4 tests)
ForTeuchos    =   0.76 sec (4 tests)
Futility      =  54.83 sec (47 tests)

Total Test time (real) =  15.44 sec

It should go quickly from here.

bscollin commented 6 years ago

Could you give me some basic instructions on how to use this and test it?

I"m guessing:

bscollin commented 6 years ago

Playing with this tonight. First note, CASL version of Trilinos has an issue with Tpetra and this newest version of Trilinos. Other than that, I didn't get too far. I built and installed Futility with Trilinos and TeuchosWrappers enabled but didn't have success getting Futility built by pointing to that. Must be missing same flag or something.

bartlettroscoe commented 6 years ago

Below are instructions on how to install Triilnos+TeuchosWrappersExt packages used by Futility and then build Futility stand-alone pointing to these installed packages using the updated Futility in the branch in PR #68.

This may need some updating but someone who knows Futility well should be able to see what is yet to be done.

Turning this Issue over to you @bscollin.

Let me know if you have questions.

DETAILED NOTES:

Setting up a stand-alone TriBITS build for Futility ...

Using the do-configure script:

#!/bin/bash
cmake \
-GNinja \
-DTPL_ENABLE_MPI=ON \
-DCMAKE_BUILD_TYPE=DEBUG \
-DFutility_ENABLE_OpenMP=ON \
-DFutility_ENABLE_Futility=ON \
-DFutility_ENABLE_TESTS=ON \
-DCTeuchos_ENABLE_TESTS=OFF \
-DCTeuchos_ENABLE_EXAMPLES=OFF \
-DForTeuchos_ENABLE_TESTS=OFF \
-DForTeuchos_ENABLE_EXAMPLES=OFF \
-DFutility_ENABLE_INSTALL_CMAKE_CONFIG_FILES=ON \
-DCMAKE_INSTALL_PREFIX=$PWD/install \
../../../Futility

I configure, build, and test the Futility packages with:

$ ~/Futility.base/BUILD/GCC-4.8.4/MPI_DEBUG/

$ rm -r CMake*

$ time ./do-configure &> configure.out

real    0m10.806s
user    0m8.465s
sys     0m2.601s

$ make &> make.out

$ time ctest -j16 &> ctest.out

real    0m14.232s
user    2m47.130s
sys     0m1.023s

That gives the test output:

100% tests passed, 0 tests failed out of 47

Label Time Summary:
Futility    =  63.87 sec (47 tests)

Total Test time (real) =  14.23 sec

Then I install the packages and headers with:

$ time make -j16 install &> make.install.out

real    0m1.987s
user    0m0.259s
sys     0m1.724s

Then I set up Futility to build stand-alone and just point to these installed packages using the updated Futility in the branch in PR #68.

With the stand-alone do-configure script:

#!/bin/bash
cmake \
-GNinja \
-DTPL_ENABLE_MPI=ON \
-DCMAKE_BUILD_TYPE=DEBUG \
-DFutility_ENABLE_OpenMP=ON \
-DFutility_ENABLE_Futility=ON \
-DFutility_ENABLE_TESTS=ON \
-DFutility_ENABLE_STANDALONE=ON \
-DTPL_ENABLE_Tpetra=ON \
-DTPL_ENABLE_MueLu=ON \
-DTPL_ENABLE_Ifpack2=ON \
-DTPL_ENABLE_Anasazi=ON \
-DTPL_ENABLE_Belos=ON \
-DTPL_ENABLE_CTeuchos=ON \
-DTPL_ENABLE_ForTeuchos=ON \
-DCMAKE_PREFIX_PATH=$HOME/Futility.base/BUILD/GCC-4.8.4/MPI_DEBUG/install \
../../../Futility

I thne configure and build stand-alone Futility with:

$ cd ~/Futility.base/BUILD/GCC-4.8.4/MPI_DEBUG_STAND_ALONE/

$ rm -r CMake*

$ time ./do-configure &> configure.out 

real    0m5.018s
user    0m3.967s
sys     0m1.950s

$ make &> make.out

$ time ctest -j16 &> ctest.out

real    0m12.661s
user    0m30.436s
sys     0m0.375s

and that returned the test result:

100% tests passed, 0 tests failed out of 47

Label Time Summary:
Futility    =  46.35 sec (47 tests)

Total Test time (real) =  12.66 sec

Note that the faster runtime may be a sign that some unit tests are not getting run. But someone who knows Futility will need to determine that.

bscollin commented 6 years ago

@bartlettroscoe, it doesn't look like the Trilinos interface is getting enabled. There is a check in cmake/Futility_Defines.cmake that looks like this:

IF(${PACKAGE_NAME}_ENABLE_MPI AND ${PROJECT_NAME}_ENABLE_Tpetra AND
        ${PROJECT_NAME}_ENABLE_Ifpack2 AND ${PROJECT_NAME}_ENABLE_MueLu AND
        ${PROJECT_NAME}_ENABLE_Belos AND ${PROJECT_NAME}_ENABLE_Anasazi AND
        ${PROJECT_NAME}_ENABLE_ForTeuchos)
    SET(${PACKAGE_NAME}_DEFINES ${${PACKAGE_NAME}_DEFINES} FUTILITY_HAVE_Trilinos)
    IF(${PACKAGE_NAME} STREQUAL "Futility")
        MESSAGE(STATUS "Enabling Trilinos Solvers")
    ENDIF()
ENDIF()

Is there a more portable way to do this? I"m guessing the <Project_Name>_ENABLE_<Package_Name>=ON won't work anymore?

bartlettroscoe commented 6 years ago

@bscollin, since these are all optional TPLs, TriBITS should be defining vars Futility_ENABLE_Tpetra, Futility_ENABLE_Ifpack2, etc. I will take a look.

bartlettroscoe commented 6 years ago

I see the problem. I forgot to update the file Futility/cmake/Dependencies.cmake. Stay tuned ...

bartlettroscoe commented 6 years ago

I added a new commit to Futility pushed to the branch in PR #68 that allows it to (I think) correctly build stand-alone against pre-installed Triinos + TeuchosWrappersExt packages. I am pretty sure this is working this time because I got build errors for stand-alone Futiltiy at first because CTeuchos_config.h was not getting installed correctly and it could not find the ForTeuchos .mod files in the install directory. I had to manually copy the CTeuchos_config.h file and the generated ForTeuchos .mod files as described below.

I am pushing the fix for the install of CTeuchos_config.h file but I am not sure of the right way to fix the install of the ForTeuchos .mod module files. We need to update TriBITS to automatically install these .mod files generated as part of creating the package library files. I am not sure how to do that but I am sure (I hope) CMake has a clean and portable way to handle this.

DETAILED NOTES:

Building Futility stand-alone ...

A) Install Trilinos + TeuchosWrappersExt packages used by Futility:

Using the do-configure script:

#!/bin/bash
cmake \
-GNinja \
-DTPL_ENABLE_MPI=ON \
-DCMAKE_BUILD_TYPE=DEBUG \
-DFutility_ENABLE_OpenMP=ON \
-DFutility_ENABLE_Futility=ON \
-DFutility_ENABLE_TESTS=ON \
-DCTeuchos_ENABLE_TESTS=OFF \
-DCTeuchos_ENABLE_EXAMPLES=OFF \
-DForTeuchos_ENABLE_TESTS=OFF \
-DForTeuchos_ENABLE_EXAMPLES=OFF \
-DFutility_ENABLE_INSTALL_CMAKE_CONFIG_FILES=ON \
-DCMAKE_INSTALL_PREFIX=$PWD/install \
../../../Futility

I configure, build, and test the Futility packages with:

$ ~/Futility.base/BUILD/GCC-4.8.4/MPI_DEBUG/

$ rm -r CMake*

$ time ./do-configure &> configure.out

real    0m10.806s
user    0m8.465s
sys     0m2.601s

$ make &> make.out

$ time ctest -j16 &> ctest.out

real    0m14.232s
user    2m47.130s
sys     0m1.023s

That gives the test output:

100% tests passed, 0 tests failed out of 47

Label Time Summary:
Futility    =  63.87 sec (47 tests)

Total Test time (real) =  14.23 sec

Then I install the packages and headers and modules with:

$ time make -j16 install &> make.install.out

real    0m1.987s
user    0m0.259s
sys     0m1.724s

$ cp TeuchosWrappersExt/cteuchos/src/CTeuchos_config.h install/include/

$ cp TeuchosWrappersExt/forteuchos/src/*.mod install/include/

(NOTE: I have the install of CTeuchos_config.h fixed on a branch that I will merge with 'casl-dev/master' but I am not sure how to correctly install the ForTeuchos *.mod files.)

B) Building stand-alone Futility against pre-installed Trilinos + TeuchosWrappersExt packages:

With the stand-alone do-configure script:

#!/bin/bash
cmake \
-GNinja \
-DTPL_ENABLE_MPI=ON \
-DCMAKE_BUILD_TYPE=DEBUG \
-DFutility_ENABLE_OpenMP=ON \
-DFutility_ENABLE_Futility=ON \
-DFutility_ENABLE_TESTS=ON \
-DFutility_ENABLE_STANDALONE=ON \
-DTPL_ENABLE_Tpetra=ON \
-DTPL_ENABLE_MueLu=ON \
-DTPL_ENABLE_Ifpack2=ON \
-DTPL_ENABLE_Anasazi=ON \
-DTPL_ENABLE_Belos=ON \
-DTPL_ENABLE_CTeuchos=ON \
-DTPL_ENABLE_ForTeuchos=ON \
-DCMAKE_PREFIX_PATH=$HOME/Futility.base/BUILD/GCC-4.8.4/MPI_DEBUG/install \
../../../Futility

I then configure and build stand-alone Futility with:

$ cd ~/Futility.base/BUILD/GCC-4.8.4/MPI_DEBUG_STAND_ALONE/

$ rm -r CMake*

$ time ./do-configure &> configure.out 

real    0m5.018s
user    0m3.967s
sys     0m1.950s

$ make &> make.out

$ time ctest -j16 &> ctest.out

real    0m14.845s
user    2m52.801s
sys     0m0.722s

and that returned the test result:


100% tests passed, 0 tests failed out of 47

Label Time Summary:
Futility    =  65.61 sec (47 tests)

Total Test time (real) =  14.84 sec
bscollin commented 6 years ago

Was able the duplicate the issue with CTeuchos. One option for now is that I just build TeuchosWrappers. It adds very little compile time.

bscollin commented 6 years ago

I was able to reproduce the results once I copied those files over. I do have a few questions that discussion over bluejeans would probably be better. Do you have some time tomorrow @bartlettroscoe ?

bscollin commented 6 years ago

Also, just some notes on trying to hid some of the complexity of this. I'm thinking that we do the following:

  1. Always enable ${PROJECT_NAME}_ENABLE_INSTALL_CMAKE_CONFIG_FILES=ON
  2. Change -DFutility_ENABLE_STANDALONE=ON -DTPL_ENABLE_Tpetra=ON -DTPL_ENABLE_MueLu=ON -DTPL_ENABLE_Ifpack2=ON -DTPL_ENABLE_Anasazi=ON -DTPL_ENABLE_Belos=ON -DTPL_ENABLE_CTeuchos=ON -DTPL_ENABLE_ForTeuchos=ON -DCMAKE_PREFIX_PATH=$HOME/Futility.base/BUILD/GCC-4.8.4/MPI_DEBUG/install to something like -D{PROJECT_NAME}_TRILINOS_AS_TPL=<path to install> and handle setting CMAKE_PREFIX_PATH and all the TPL_ENABLE's inside the cmake.
bartlettroscoe commented 6 years ago

@bscollin,

Also, just some notes on trying to hid some of the complexity of this.

Yup, you can put in whatever logic you want to simplify this for your use cases. We should also look into getting CMake to install the *.mod files automatically. I will pin our Kitware contacts about this (and CC you).

Do you have some time tomorrow @bartlettroscoe ?

Yes, after 3 would be good. Please send me a calendar appointment.