Open bartlettroscoe opened 1 year ago
With the merge of PR #12044, the new test TrilinosInstallTests_reduced_tarball
is defined but it does not run due to the PR testing process not passing the GenConfig <settings>.cmake
file using -D Trilinos_CONFIGURE_OPTIONS_FILE=<base-path>/<settings>.cmake
. Instead, it uses -C <base-path>/<settings>.cmake
. You can see that this test gets disabled automatically in PR testing as shown, for example, in the clang configuration showing:
Processing enabled top-level package: TrilinosInstallTests (Libs, Tests, Examples)
...
-- TrilinosInstallTests_reduced_tarball: NOT added test because EXCLUDE_IF_NOT_TRUE Trilinos_CONFIGURE_OPTIONS_FILE=''!
...
The experimental builds that show the problems are given in the following comment blocks:
TrilinosInstallTests_reduced_tarball
using -D Trilinos_CONFIGURE_OPTIONS_FILE=<base-path>/<settings>.cmake
That submitted results to CDash at:
showing:
In the new test output TrilinosInstallTests_reduced_tarball
you see:
================================================================================
TEST_2
Configure the reduced tarball
Creating new working directory '/home/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/packages/TrilinosInstallTests/TrilinosInstallTests_reduced_tarball/trilinos-14.3-Source_build'
Running: "/home/projects/sems/install/rhel7-x86_64/sems/utility/cmake/3.24.3/gcc/8.3.0/base/pwyrvxm/bin/cmake" "-DTrilinos_CONFIGURE_OPTIONS_FILE=/ascldap/users/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/GenConfigSettings.cmake" "-DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpicc" "-DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpicxx" "-DCMAKE_Fortan_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpif90" "-DTrilinos_ENABLE_ALL_PACKAGES=ON" "-DTrilinos_ASSERT_DEFINED_DEPENDENCIES=OFF" "../trilinos-14.3-Source"
Running in working directory "/home/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/packages/TrilinosInstallTests/TrilinosInstallTests_reduced_tarball/trilinos-14.3-Source_build"
--------------------------------------------------------------------------------
-- The Current CXX Standard is : 17
Configuring Trilinos build directory
...
-- Reading in configuration options from /ascldap/users/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/GenConfigSettings.cmake ...
...
...
...
Processing enabled external package/TPL: HDF5 (enabled explicitly, disable with -DTPL_ENABLE_HDF5=OFF)
-- HDF5_LIBRARY_NAMES='hdf5;z;hdf5_hl'
-- TPL_HDF5_LIBRARIES='/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/5.3.0/openmpi/1.10.1/parallel/lib/libhdf5_hl.so;/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/5.3.0/openmpi/1.10.1/parallel/lib/libhdf5.so;/projects/sems/install/rhel7-x86_64/sems/tpl/zlib/1.2.8/gcc/5.3.0/base/lib/libz.so;-ldl'
-- Searching for headers in HDF5_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/5.3.0/openmpi/1.10.1/parallel/include'
-- Searching for a header file in the set "hdf5.h":
-- Searching for header 'hdf5.h' ...
-- Found header '/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/5.3.0/openmpi/1.10.1/parallel/include//hdf5.h'
-- Found TPL 'HDF5' include dirs '/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/5.3.0/openmpi/1.10.1/parallel/include'
-- TPL_HDF5_INCLUDE_DIRS='/projects/sems/install/rhel7-x86_64/sems/tpl/hdf5/1.10.6/gcc/5.3.0/openmpi/1.10.1/parallel/include'
...
...
-- Configuring done
-- Generating done
...
-- Build files have been written to: /home/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/packages/TrilinosInstallTests/TrilinosInstallTests_reduced_tarball/trilinos-14.3-Source_build
That shows that it is enabling and finding the TPLs and the configure passes.
TrilinosInstallTests_reduced_tarball
using -C<base-path>/<settings>.cmake -D Trilinos_CONFIGURE_OPTIONS_FILE=<dummy>.cmake
That submitted results to CDash at:
showing:
showing the failing test TrilinosInstallTests_reduced_tarball showing:
================================================================================
TEST_2
Configure the reduced tarball
Creating new working directory '/home/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/packages/TrilinosInstallTests/TrilinosInstallTests_reduced_tarball/trilinos-14.3-Source_build'
Running: "/home/projects/sems/install/rhel7-x86_64/sems/utility/cmake/3.24.3/gcc/8.3.0/base/pwyrvxm/bin/cmake" "-DTrilinos_CONFIGURE_OPTIONS_FILE=/ascldap/users/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/dummy-cache-file.cmake" "-DCMAKE_C_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpicc" "-DCMAKE_CXX_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpicxx" "-DCMAKE_Fortan_COMPILER=/projects/sems/install/rhel7-x86_64/sems/compiler/gcc/5.3.0/openmpi/1.10.1/bin/mpif90" "-DTrilinos_ENABLE_ALL_PACKAGES=ON" "-DTrilinos_ASSERT_DEFINED_DEPENDENCIES=OFF" "../trilinos-14.3-Source"
Running in working directory "/home/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/packages/TrilinosInstallTests/TrilinosInstallTests_reduced_tarball/trilinos-14.3-Source_build"
--------------------------------------------------------------------------------
...
-- Reading in configuration options from /ascldap/users/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/dummy-cache-file.cmake ...
...
...
Processing enabled external package/TPL: Matio (enabled by SEACASMat2exo, disable with -DTPL_ENABLE_Matio=OFF)
-- Matio_LIBRARY_NAMES='matio'
-- Searching for libs in Matio_LIBRARY_DIRS=''
-- Searching for a lib in the set "matio":
-- Searching for lib 'matio' ...
-- NOTE: Did not find a lib in the lib set "matio" for the TPL 'Matio'!
-- ERROR: Could not find the libraries for the TPL 'Matio'!
-- TIP: If the TPL 'Matio' is on your system then you can set:
-DMatio_LIBRARY_DIRS='<dir0>;<dir1>;...'
to point to the directories where these libraries may be found.
Or, just set:
-DTPL_Matio_LIBRARIES='<path-to-libs0>;<path-to-libs1>;...'
to point to the full paths for the libraries which will
bypass any search for libraries and these libraries will be used without
question in the build. (But this will result in a build-time error
if not all of the necessary symbols are found.)
-- ERROR: Failed finding all of the parts of TPL 'Matio' (see above), Aborting!
-- NOTE: The find module file for this failed TPL 'Matio' is:
/home/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/packages/TrilinosInstallTests/TrilinosInstallTests_reduced_tarball/trilinos-14.3-Source/cmake/TPLs/FindTPLMatio.cmake
which is pointed to in the file:
/home/rabartl/Trilinos.base/BUILDS/PR/pr_builds/clang-11.0.1/packages/TrilinosInstallTests/TrilinosInstallTests_reduced_tarball/trilinos-14.3-Source/TPLsList.cmake
TIP: One way to get past the configure failure for the
TPL 'Matio' is to simply disable it with:
-DTPL_ENABLE_Matio=OFF
which will disable it and will recursively disable all of the
downstream packages that have required dependencies on it, including
the package 'SEACASMat2exo' which triggered its enable.
When you reconfigure, just grep the cmake stdout for 'Matio'
and then follow the disables that occur as a result to see what impact
this TPL disable has on the configuration of Trilinos.
CMake Error at cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:278 (message):
ERROR: TPL_Matio_NOT_FOUND=TRUE, aborting!
Call Stack (most recent call first):
cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:171 (tribits_address_failed_tpl_find)
cmake/tribits/core/package_arch/TribitsProcessEnabledTpls.cmake:105 (tribits_process_enabled_standard_tpl)
cmake/tribits/core/package_arch/TribitsProjectImpl.cmake:197 (tribits_process_enabled_tpls)
cmake/tribits/core/package_arch/TribitsProject.cmake:92 (tribits_project_impl)
CMakeLists.txt:101 (TRIBITS_PROJECT)
-- Configuring incomplete, errors occurred!
The problem is that it can't find the TPLs and other info specified in the file GenConfigSettings.cmake
because the location of that file is unknown.
Below is the mail I sent to Kitware CMake experts.
From: Bartlett, Roscoe A
Sent: Saturday, July 8, 2023 11:53 AM
To: Brad King <...>
; Kyle Edwards <...>
; Zack Galbreath <...>
Subject: Info on cache files passed thorugh -C <cache-file>.cmake
arguments?
Hello Brad, Kyle, and Zack,
Do you know if there is a way to determine what files were passed to CMake with the -C <cache-file>.cmake
option? I need to know this from inside of the CMake project in order to create a reduced tarball test that untars the reduced tarball and then configures the reduced source tree. Here is the test in Trilinos that does that:
https://github.com/bartlettroscoe/Trilinos/commit/8c5b9188f4c16c59c3b9714175927d71062426e9
Would this need to be as CMake extension to provide the paths for these -C files?
-Ross
Update 7/14/2023:
Below is the response from @bradking.
From: Brad King <...>
Sent: Friday, July 14, 2023 6:29 AM
To: Bartlett, Roscoe A <...>
Cc: Kyle Edwards <...>
; Zack Galbreath <...>
Subject: [EXTERNAL] Re: Info on cache files passed thorugh -C <cache-file>.cmake
arguments?
On Sat, Jul 8, 2023 at 1:53 PM Bartlett, Roscoe A wrote:
Do you know if there is a way to determine what files were passed to CMake with the -C
.cmake option? ... Would this need to be as CMake extension to provide the paths for these -C files?
I don't think we currently offer any way to get that information.
We also don't currently record what -C
files were used in previous runs of CMake in a given build tree. One can accumulate the files across multiple runs in a single build tree. For example:
$ cmake -S src -B build -C init1.cmake -C init2.cmake
$ cmake -S src -B build -C update1.cmake
$ cmake -S src -B build -C update2.cmake
If we were to add a way to get the list of files, we'd have to define semantics w.r.t. multiple runs.
-Brad
@sebrowne, note that the new test TrilinosInstallTests_reduced_tarball
due to the issue described above and above. Unless Kitware says otherwise, I think the only solution is to change the PR testing drivers to pass in the generated fragment file using -D Trilinos_CONFIGURE_OPTIONS_FILE=<settings>.cmake
instead of using -C <base-path>/<settings>.cmake
.
@sebrowne, I would like to propose that a subset of the PR builds switch from using -C generatedFragment.cmake
to instead use -D Trilinos_CONFIGURE_OPTIONS_FILE=generatedFragment.cmake
. I think it would be sufficient to have one of the non-CUDA builds and one of the CUDA builds switch to using -D Trilinos_CONFIGURE_OPTIONS_FILE=generatedFragment.cmake
. I think that would be enough for reduced tarball testing.
And just the ability to have some tests for Trilinos that configure (and optionally build) subsets of Trilinos would be very attractive. That avoids needing to add new significant infrastructure in the top level testing systems and makes it trivial for Trlinos developers to run these types of tests locally.
CC: @ccober6, @jwillenbring
@sebrowne, as I noted two weeks ago above, the reduced tarball tests are not running in any automated testing because of the way that the GenConfig settings are passed. To see the issue, see:
Without making this change, we can't support reduced tarball creation or testing.
FYI: There was an internal request for supporting reduced Trilinos tarballs for the Aleph project:
I believe that use case is addressed in Trilinos 'develop' but we still don't have any PR or Nightly testing running for this.
We can try making that change. The way that things are currently structured prevents us from testing it easily. I will add a story to move the SimpleTestingRefactor (or snapshot it for now) ctest driver code that is used by the PR system into the Trilinos repository. After that, it should be an easy (and testable) PR to change from the cachefile option to the TriBITS one. Although moving away from the standard cachefile import feels less than ideal to me. Better if Kitware can modify cmake to keep track of which cachefiles were loaded as described above.
FYI: @sebrowne and I discussed this last week. We are going to wait until the SimpleTestingRefactor CTest -S driver code used by the Trilinos PR testing process is added to the Trilinos Git repo (so it can be tested along with PR testing) and then we will update to pass in the GenConfig .cmake file using -D Trilinos_CONFIGURE_OPTIONS_FILE
so we can enable reduced tarball testing. (Also having a few PR builds that use -C
to pass in the .cmake file may be maintained as well.)
@sebrowne, with the merge of PR:
and the completion of TRILFRAME-592, can we now pull the trigger on changing from -C <options>.cmake
to -D Trilinos_CONFIGURE_OPTIONS_FILE=<options>.cmake
for the PR and Nightly build run with the Framework teams SimpleTestingRefactor
script?
Second story on the backlog now, will do it as soon as we release 15.0.
This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity.
If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE
label.
If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE
.
If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.
CC: @sebrowne, @ccober6, @dridzal
Description
As described in #11976, there was no testing for the creation of reduced tarballs for Trilinos as described in Creating a tarball of the source tree. But there is a desire to maintain that feature, so we need tests. Some beefier tests for reduced tarball creation was added to TriBITS proper in TriBITS PR:
However, Trilinos needs its own automated tests for reduced tarball creation (in order to avoid Trilinos-specific problems like reported in #11976).
Tasks
-C <path>/<settings>.cmake
to-D Trilinos_CONFIGURE_OPTIONS_FILE=<path>/<settings>.cmake
... See below