search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.21k stars 563 forks source link

Initial adjustment of Primary Tested set of Trilinos Packages and TPLs #410

Closed bartlettroscoe closed 7 years ago

bartlettroscoe commented 8 years ago

Next Action Status:

Sent email to SNL customers of Trilinos, asking for list of Trilinos packages and TPLs. Next: Wait for responses ...

CC: @jwillenbring, @bmpersc, @maherou, @trilinos/framework

Blocked By: #158

Blocking: #370

Description:

This story is to do an initial adjustment (mostly an extension) the set of Primary Tested (PT) TPLs to include HDF5, Netcdf, Boost, SWIG and any others needed to fully build and test packages like SEACAS, STK, etc. This initial set of packages and TPLs will be limited to what is provided by the SEMS Dev Env (see #158). Otherwise, some packages that could be included based on that set of TPLs should otherwise be excluded from the set of PT packages as guided by the needs of important Trilinos customers and critical research efforts (which are need to be well protected). Purely exploratory software or packages without important current users uses should generally be excluded from the PT set (to speed up the builds and tests). Also, non-critical configurations (e.g. complex, float) should be avoided to speed up builds and testing.

This initial story will be to support the usage of the checkin-test.py script to update from the 'develop' to the 'master' branch (see #370).

Tasks:

  1. Get initial feedback from Trilinos developers and stakeholders (captured in emails to Ross) [Done]
  2. Document, analyze and combine the lists from SNL customers (see TRIL-59) ...
  3. Come up with a new list of PT Packages and TPLs (that can be supported by the SEMS Dev Env) ...
  4. Fill out the "Testing Category" column in the Trilinos Package Classification Database ...
  5. Update the file Trilinos/PackagesList.cmake and TPLsList.cmake
bartlettroscoe commented 8 years ago

I just sent an email to internal SNL customers of Trilinos in order to try to gather up the list of important Trilinos packages. Unfortunately, I can't include those emails here because they name specific SNL codes. But I will gather up the final list in this Issue ticket.

bartlettroscoe commented 8 years ago

NOTE: I just created the internal SON ticket:

to process the list of "important" packages and TPLs for internal SNL customers. The outcome from that story will be an updated candidate list of PT packages and TPLs.

bartlettroscoe commented 7 years ago

I completed the analysis of the Trilinos packages, SE package, and TPLs used by a bunch of Trilinos customers and created a combined list from all of these. You can see what customers were considered and where the data came from by looking at:

The combined list sorted alphabetically is given below. The challenge that we face is that large set of TPLs that all of our Trilinos customers are using:

The problem is that the SEMS env can only provide the TPLs:

What is missing from SEMS is:

Not much we can do about that if we are going to build our CI system on the SEMS env. It has what it has.

I am going to put this story into review now and move on with #482.

@trilinos/framework, any comments on this?

Full set of packages:

Amesos Amesos2 Anasazi AztecOO Belos Claps Domi Drekar DrekarMHD Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 Isorropia Kokkos Mesquite ML MOOCHO MueLu NOX OptiPack Pamgen Panzer Phalanx Pike Piro ROL RTOp Rythmos Sacado SEACAS Shards ShyLU STK Stokhos Stratimikos Teko Teuchos ThreadPool Thyra Tpetra TriKota TrilinosCouplings Triutils Xpetra Zoltan Zoltan2

Full set of SE packages:

Amesos Amesos2 Anasazi AztecOO Belos Claps Domi Drekar DrekarMHD Epetra EpetraExt FEI Galeri GlobiPack Gtest Ifpack Ifpack2 Intrepid Intrepid2 IntrepidCore Isorropia Kokkos KokkosAlgorithms KokkosContainers KokkosCore KokkosTPL Mesquite ML MOOCHO MueLu NOX OptiPack Pamgen Panzer PanzerAdaptersSTK PanzerCore PanzerDiscFE PanzerDofMgr Phalanx Pike PikeBlackBox PikeImplicit Piro ROL RTOp Rythmos Sacado SEACAS SEACASAlgebra SEACASAprepro SEACASAprepro_lib SEACASBlot SEACASChaco SEACASConjoin SEACASEjoin SEACASEpu SEACASEx1ex2v2 SEACASEx2ex1v2 SEACASExo_format SEACASExodiff SEACASExodus SEACASExodus_for SEACASExoIIv2for32 SEACASExomatlab SEACASExotxt SEACASFastq SEACASGen3D SEACASGenshell SEACASGjoin SEACASGrepos SEACASGrope SEACASIoss SEACASMapvar SEACASMapvar-kd SEACASMapvarlib SEACASNemesis SEACASNemslice SEACASNemspread SEACASNumbers SEACASPLT SEACASSupes SEACASSuplib SEACASSuplibC SEACASSuplibCpp SEACASSVDI SEACASTxtexo Shards ShyLU ShyLUCore STK STKClassic STKDoc_tests STKExprEval STKIO STKMesh STKSearch STKSearchUtil STKTopology STKTransfer STKUnit_test_utils STKUnit_tests STKUtil Stokhos Stratimikos Teko Teuchos TeuchosComm TeuchosCore TeuchosKokkosComm TeuchosKokkosCompat TeuchosNumerics TeuchosParameterList TeuchosRemainder ThreadPool Thyra ThyraCore ThyraEpetraAdapters ThyraEpetraExtAdapters ThyraTpetraAdapters Tpetra TpetraClassic TpetraCore TpetraKernels TpetraTSQR TriKota TrilinosCouplings Triutils Xpetra Zoltan Zoltan2

Full set of TPLs:

AMD BLACS BLAS Boost BoostLib DLlib HDF5 LAPACK METIS MPI MUMPS Netcdf PARDISO_MKL ParMETIS PETSC Pthread QT SCALAPACK SuperLU SuperLUDist UMFPACK X11 y12m Zlib

srajama1 commented 7 years ago

I see Scotch is missing.

bartlettroscoe commented 7 years ago

I see Scotch is missing.

That is interesting. Looks like none of the internal Trilinos customers that responded to my request are using Scotch with Trilinos. You can see the full set of customers that were considered in:

But also note that the SEMS env only provides a 32-bit build for Scotch which is in-compatible with the 64-bit version they provide for ParMETIS. You can read about that in #475.

srajama1 commented 7 years ago

Scotch is a recent addition. SIERRA officially started testing it late September, if my memory is right.

bartlettroscoe commented 7 years ago

Scotch is a recent addition. SIERRA officially started testing it late September, if my memory is right.

Good to know, thanks. I did a search of the SEMS JIRA project and I found:

that suggests that SEMS might now provide a compatible 64-bit version of Scotch. Can anyone confirm that? @kddevin, have you been able to test out the SEMS env with 64-bit ParMETIS and Scotch both enabled with Zoltan and Zoltan2?

srajama1 commented 7 years ago

Is there a 32-bit version. Tacho works with 32-bit. We can add a 64-bit version interface as well. @kyungjoo-kim

bartlettroscoe commented 7 years ago

Is there a 32-bit version. Tacho works with 32-bit. We can add a 64-bit version interface as well.

The problem is that you can't enabled both the 64-bit version of ParMETIS and the 32-bit version of Scotch at the same time. If you do, a bunch of Zoltan and Zoltan2 tests fail. That is no good.

kyungjoo-kim commented 7 years ago

Tacho does not require for 32 bit version specifically. Does SIERRA use a 32 bit version ?

kddevin commented 7 years ago
  1. Zoltan2 works fine with mixed 32-bit/64-bit TPL index types. Only Zoltan requires consistent ID sizes among the TPLs; it is unlikely we will ever have resources to remove that constraint.
  2. All Zoltan and Zoltan2 tests pass with SEMS' 64-bit ParMETIS and Scotch, and with SEMS' 32-bit ParMETIS and Scotch.
bartlettroscoe commented 7 years ago

Zoltan2 works fine with mixed 32-bit/64-bit TPL index types. Only Zoltan requires consistent ID sizes among the TPLs; it is unlikely we will ever have resources to remove that constraint.

All Zoltan and Zoltan2 tests pass with SEMS' 64-bit ParMETIS and Scotch, and with SEMS' 32-bit ParMETIS and Scotch.

@kddevin, has anyone tried this with the ParMETIS and Scotch TPLs provided by the SEMS Env? If not, we can try this later once we get #482 done.

Adding more TPLs to our CI and Nightly testing is something that we will need to do gradually as we can. (We just need infrastructure manpower to help with this). If you look at the list of TPLs that are actually used by our important Sandia Trilinos customers listed above, you will see that we are falling far short with what Trilinos is currently testing in Nightly builds and what is provided by the SEMS env.

kddevin commented 7 years ago

@bartlettroscoe I don't understand. What is "this" in your question above?

bartlettroscoe commented 7 years ago

What is "this" in your question above?

Sorry, I should have been more explicit. Has any Zoltan or Zoltan2 developer tried the updated SEMS-provided 64-bit Scotch and ParMETIS TPLs? It looks like these are now present under:

If not, that is fine. I was just asking.

kddevin commented 7 years ago

Yes; see my comment 2 above. All tested. All good. Thanks.

bartlettroscoe commented 7 years ago

Realized that we need to make a few changes to the list of packages and SE packages listed above.

First, Claps is an EX. It is not actively developed in Trilinos and there is no reason to test it.

Second, I accidentally listed Drekar and DrekarMHD. Those are not even Trilinos packages.

Third, TriKota depends on Dakota. I don't think we should be forcing Trilinos developers to have to build Dakota when the test Trilinos. I think that is asking too much. Also, there is not a solid integration model for what version of Dakota is compatible with TriKota. Therefore, I think it makes sense to just let Albany test TriKota.

Forth, Mesquite and MOOCHO are in a separate git repo from the main Trilinos repo. Therefore, it makes no sense to include it in the set of PT packages.

So taking out Claps, Drekar, DrekarMHD, Mesquite, and TriKota, the new list of primary tested packages and TPLs becomes:

bartlettroscoe commented 7 years ago

Another adjustment needs to be made. The subpackage IntrepidCore no longer exists in Trilinos. It was taken out in the commit:

commit 889d5e8e493cafe3ef151d6b886e815053e704a5
Author: avdelor <avdelor@sandia.gov>
Date:   Tue Jan 5 16:50:14 2016 -0700

    Intrepid:

    fixed phalanx dependencies
    took intrepid out of core subpackage

diff --git a/packages/intrepid/cmake/Dependencies.cmake b/packages/intrepid/cmake/Dependencies.cmake
index f1b64a3..c2eba4a 100644
--- a/packages/intrepid/cmake/Dependencies.cmake
+++ b/packages/intrepid/cmake/Dependencies.cmake
@@ -1,8 +1,3 @@
-SET(SUBPACKAGES_DIRS_CLASSIFICATIONS_OPTREQS
-  #SubPackageName       Directory         Class    Req/Opt
-  #
-  Core                  core               PT      OPTIONAL
-  )
 SET(LIB_REQUIRED_DEP_PACKAGES Teuchos Shards Sacado)
 SET(LIB_OPTIONAL_DEP_PACKAGES KokkosCore KokkosAlgorithms)
 SET(TEST_REQUIRED_DEP_PACKAGES)

CASL VERA is using very old version of Intrepid before this commit was made.

Removing IntrepidCore the new set of of PT package are:

bartlettroscoe commented 7 years ago

Another change to the list of SE packages. Seems that Kokkos no longer has a subpackage called "KokkosTPL". It was removed in this commit:

commit e1e44072a7e8c1c76037f4c7766f321062b2dd6c
Author: Carter Edwards <hcedwar@sandia.gov>
Date:   Tue Apr 7 10:45:15 2015 -0600

    Kokkos: Remove unused TPL subdirectory & subpackage.

D       packages/kokkos/TPL/CMakeLists.txt
M       packages/kokkos/config/configure_tpetra_kokkos_cuda_nvcc_wrapper.sh
M       packages/kokkos/core/cmake/Dependencies.cmake

Only one Trilinos customer listed that (you can see which one by examining the combined_list_of_trlinos_customer_enables.sh script in TRIL-59). That customer must have been using a very old version of Trilinos when they reported this.

The update list of enables removing KokkosTPL is:

bartlettroscoe commented 7 years ago

I went back and put the script combined_list_of_trlinos_customer_enables.sh under version control and carefully make all of the tweaks to it that are described above and few others (see TRIL-59). Now I can get a local configure of Trilinos to exactly match the updated list and it looks right.

Here is the updated list of PT enables:

bartlettroscoe commented 7 years ago

This is complete and is being used in new CI build. It is unlikely that anyone will seriously review this so closing as complete.

Also, we are no longer going to purse extending the checkin-test-sems.sh (or checkin-test.py) script for Trilinos (see https://github.com/trilinos/Trilinos/issues/482#issuecomment-266124179). No idea what selection of packages and TPLs will be used for new automatic PR system but that can be determined later by someone else.