bartlettroscoe
commented
2 years ago I posted the internal issue:
to try to get this on someone's radar.
Closed bartlettroscoe closed 2 years ago
bartlettroscoe
commented
2 years ago I posted the internal issue:
to try to get this on someone's radar.
csiefer2
commented
2 years ago I filed this exact problem as an internal issue on Wednesday
bartlettroscoe
commented
2 years ago I filed this exact problem as an internal issue on Wednesday
Unfortunately, we can't see each other's TRILINOSHD issues. I would like to propose that we always create a Trilinos GitHub issue first and then create an internal issue and put in links both ways (which is what I did here which is why we are having this interaction). Then Trilinos developers can just search the GitHub issues and not duplicate effort.
bartlettroscoe
commented
2 years ago CC: @rppawlo, @ccober6,
See comment from @csiefer2 above.
csiefer2
commented
2 years ago One of my colleagues has been reduced to adding printfs to the PR branch and waiting for the AT to rerun (!)
Time to go gradedig the old env scripts and hope they're similar enough to the genconfig to reproduce his error....
csiefer2
commented
2 years ago Got it! Can reproduce via pre-genconfig scripts.
jhux2
commented
2 years ago Unfortunately, we can't see each other's TRILINOSHD issues. I would like to propose that we always create a Trilinos GitHub issue first and then create an internal issue and put in links both ways (which is what I did here which is why we are having this interaction). Then Trilinos developers can just search the GitHub issues and not duplicate effort.
Or just stop using jira/confluence.
fryeguy52
commented
2 years ago I am able to reproduce on sems-srn-rhel7-login-01 and the probelm for me is that get_dependencies.sh --srn fails to clone the srn-ini-files repo. @bartlettroscoe is the same thing happening for you? does packages/framework/srn-ini-files exists? if I clone that repo separately then I can get it to work
TRILINOS_SRC=${HOME}/Trilinos
source ${TRILINOS_SRC}/packages/framework/GenConfig/gen-config.sh \
rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables \
--force \
${TRILINOS_SRC}
...
(rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial) $ module list
Currently Loaded Modules:
1) sems-env 3) sems-clang/10.0.0 5) sems-netcdf/4.7.3/parallel 7) sems-scotch/6.0.3/nopthread_64bit_parallel 9) sems-git/2.10.1 11) sems-zlib/1.2.8/base 13) sems-superlu/4.3/base 15) sems-boost/1.69.0/base
2) sems-gcc/5.3.0 4) sems-openmpi/1.10.1 6) sems-hdf5/1.10.6/parallel 8) sems-parmetis/4.0.3/parallel 10) sems-ninja_fortran/1.10.0 12) sems-boost/1.63.0/base 14) sems-python/2.7.9 16) sems-cmake/3.17.1note that I added --force to the options. My understanding is that this will force it to use rhel7 and not try and match a machine name
bartlettroscoe
commented
2 years ago I am able to reproduce on sems-srn-rhel7-login-01 and the probelm for me is that get_dependencies.sh --srn fails to clone the srn-ini-files repo. @bartlettroscoe is the same thing happening for you?
@fryeguy52, now that you mention it, looking at the expand Details on attempt to reproduce clang-10.0.0 PR build above it seems that might be the case looking more carefully at:
$ cd Trilinos/
$ gitdist-status --dist-repos=.,packages/framework/GenConfig,packages/framework/srn-ini-files
------------------------------------------------------------------------------------------------------------------------------------------------------------
| ID | Repo Dir | Branch | Tracking Branch | C | M | ? |
|----|----------------------------------|--------------------------------------------|-------------------------------------------------------|-----|---|---|
| 0 | Trilinos (Base) | tribits-299-modern-cmake-targets-1-again-2 | rab-github/tribits-299-modern-cmake-targets-1-again-2 | 301 | | |
| 1 | packages/framework/GenConfig | HEAD | | | 2 | 1 |
| 2 | packages/framework/srn-ini-files | tribits-299-modern-cmake-targets-1-again-2 | rab-github/tribits-299-modern-cmake-targets-1-again-2 | 301 | | |
------------------------------------------------------------------------------------------------------------------------------------------------------------
$ git log-short --name-status -1 $(git merge-base HEAD github/develop)
6c842cb13d9 "Merge Pull Request #10706 from brian-kelley/Trilinos/SpeedupAdditiveSchwarz"
Author: trilinos-autotester <trilinos@sandia.gov>
Date: Thu Jul 14 17:45:09 2022 -0600 (17 hours ago)
$ gitdist-repo-versions --dist-repos=packages/framework/GenConfig,packages/framework/srn-ini-files
*** Git Repo: packages/framework/GenConfig
924a08af66f0a0573b5dd1128179731489339aec [Wed Jun 15 15:55:48 2022 +0000] <eharvey@sandia.gov>
Merge branch 'del-gen-symlinks' into 'master'
*** Git Repo: packages/framework/srn-ini-files
dfc3edd437bfde61f90a41feb5a16530cb7bff6b [Fri Jul 15 09:54:24 2022 -0600] <rabartl@sandia.gov>
Ignore locally generated *.iniGiven that the same branch name tribits-299-modern-cmake-targets-1-again-2 is reported for both the base Trilinos and the srn-init-files directories. That means that the srn-init-files does not have a .git directory.
bartlettroscoe
commented
2 years ago The issue was that I had run get_dependencies.sh with the prior version of Trilinos that used Git Submodules which created symlinks into a regular subdirectory packages/framework/srn-ini-files/. When switching over to the new implementation, those symlink files still existed and Git kept the subdir packages/framework/srn-ini-files/. So the logic in the updated get_dependencies.sh script just assumed the srn-ini-files repo was already cloned and tried t update it. Well, running git commands a non-tracked subdir in a Git repo just runs the command in the base git repo (which is Trilinos). This is terrible behavior.
Once I deleted that subdir and ran:
$ cd Trilinos
$ rm -rf packages/framework/srn-ini-files
$ ./packages/framework/get_dependencies.sh --srnit cloned the srn-ini-files repo correctly.
Now I can get it to run with:
$ ssh crf450
$ cd ~/Trilinos.base/BUILDS/PR/clang-10.0.0/
$ cat load-env.sh
TRILINOS_SRC=${HOME}/Trilinos.base/Trilinos
source ${TRILINOS_SRC}/packages/framework/GenConfig/gen-config.sh \
rhel7_sems-clang-10.0.0-openmpi-1.10.1-serial_release-debug_shared_no-kokkos-arch_no-asan_no-complex_no-fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables \
--force \
${TRILINOS_SRC}
$ . load-env.shI get a crazy amount of warnings like:
!! ================================================================================
!! EXCEPTION SKIPPED
!! Event Type : WARNING
!! Exception : ValueError
!! Message : bash generator - `Trilinos_PARALLEL_LINK_JOBS_LIMIT=2` skipped because CACHE var `Trilinos_PARALLEL_LINK_JOBS_LIMIT` is already set and CMake requires FORCE to be set to change the value.
!!
!! Call Stack:
!! File "/ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/GenConfig/gen_config.py", line 834, in <module>
!! main(sys.argv[1:]) # pragma: no cover
!! File "/ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/GenConfig/gen_config.py", line 829, in main
!! F.write(gc.generated_config_flags_str)
!! File "/ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/GenConfig/gen_config.py", line 162, in generated_config_flags_str
!! self.complete_config, "bash"
!! File "/home/rabartl/Trilinos.base/Trilinos/packages/framework/GenConfig/setprogramoptions/SetProgramOptions.py", line 493, in gen_option_list
!! line = self._gen_option_entry(option_entry, generator=generator)
!! File "/home/rabartl/Trilinos.base/Trilinos/packages/framework/GenConfig/setprogramoptions/SetProgramOptions.py", line 571, in _gen_option_entry
!! output = method_ref(params, value)
!! File "/home/rabartl/Trilinos.base/Trilinos/packages/framework/GenConfig/setprogramoptions/SetProgramOptionsCMake.py", line 294, in _program_option_handler_opt_set_cmake_var_bash
!! self.exception_control_event("WARNING", ValueError, message=msg)
!!
!! Increase `exception_control_level` to 5 to raise this exception.
!! ================================================================================shown below and this command actually tries to configure Trilinos (with no packages enabled)!
This command needs to not actually configure Trilinos, only write the CMake fragment file to pick up with:
$ cmake -C <fragmentFile>.cmake [other options]FYI: I just posted PR #10778 that should help catch this problem for other developers and tell when to do.
bartlettroscoe
commented
2 years ago @csiefer2, had you tried to run GenConfig weeks ago? That might explain how you were able to produce this error.
csiefer2
commented
2 years ago @bartlettroscoe Nope! This was the first time I tried it and it didn't work.
bartlettroscoe
commented
2 years ago Okay, I seem to be able to create a bash source script like load-env-and-cmake-frag-file.sh for an SNL RHEL7 machine with the SEMS modules:
$ cat load-env-and-cmake-frag-file.sh
if [[ -e GenConfigDevEnv.cmake ]] ; then
echo "Remvoing existing file GenConfigDevEnv.cmake ..."
rm GenConfigDevEnv.cmake
fi
source /fgs/rabartl/Trilinos.base/Trilinos/packages/framework/GenConfig/gen-config.sh \
--cmake-fragment GenConfigDevEnv.cmake \
rhel7_sems-intel-17.0.1-mpich-3.2-serial_release-debug_static_no-kokkos-arch_no-asan_no-complex_fpic_mpi_no-pt_no-rdc_no-uvm_deprecated-on_no-package-enables \
--force \
"$@"which when I source it with:
$ . load-env-and-cmake-frag-file.shputs me in a new shell and then generates the file GenConfigDevEnv.cmake.
With that, I seem to be able to create the configure script {{do-configure}}:
$ cat do-configure
if [[ -e CMakeCache.txt ]] ; then
echo "Removing CMakeCache.txt ..."
rm CMakeCache.txt
fi
if [[ -d CMakeFiles ]] ; then
echo "Removing CMakeFiles ..."
rm -r CMakeFiles
fi
cmake \
-G Ninja \
-C GenConfigDevEnv.cmake \
-D Trilinos_ENABLE_ALL_FORWARD_DEP_PACKAGES=OFF \
-D Trilinos_ENABLE_TESTS=ON \
-D Trilinos_TRACE_ADD_TEST=ON \
"$@" \
/fgs/rabartl/Trilinos.base/TrilinosThis allows me to enable any package I want using this configuration like with:
$ ./do-configure -DTrilinos_ENABLE_Tpetra=ONand only get that package and its tests enabled.
The build is running now. So far, so good.
bartlettroscoe
commented
2 years ago Okay, I have worked out a reasonable scheme to reproduce one of the SEMS RHEL7 builds as documented in detail in: https://github.com/trilinos/Trilinos/pull/10813#issuecomment-1204372271. This is my first successful usage of the Trilinos GenConfig system to reproduce a failure.
Closing as complete.
NOTE: It would be good to update the wiki instructions to document how to use gen-config.sh with the argument --cmake-fragment <file-name>.cmake to be able to more carefully control how the configuration is performed and make this a more useful tool for developers.
@trilinos/framework
Related Issues:
Description
I tried to follow the instructions to reproduce a PR build with the new GenConfig system and I got a error from gen_config.py right away showing:
I am on an SNL SRN COE machine with the SEMS modules synced.
Full details are given below.
Details on attempt to reproduce clang-10.0.0 PR build (click to expand)
I merged in the 'develop' branch to my local branch and updated the GenConfig files with: ``` $ cd Trilinos/ $ git fetch github $ git merge --no-edit github/develop $ ./packages/framework/get_dependencies.sh --srn STATUS: GenConfig: Cloning from 'git@gitlab-ex.sandia.gov:trilinos-devops-consolidation/code/GenConfig.git' ... Note: switching to '924a08af66f0a0573b5dd1128179731489339aec'. ... STATUS: srn-ini-files: Fetching remote repo STATUS: srn-ini-files: Merging tip of remote tracking branch Already up to date. INFO: You selected the following LoadEnv ini files: lrwxrwxrwx 1 rabartl rabartl 119 Jul 15 10:00 GenConfig/deps/LoadEnv/ini_files/environment-specs.ini -> /ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/srn-ini-files/trilinos/framework/environment-specs.ini lrwxrwxrwx 1 rabartl rabartl 93 Jul 15 10:00 GenConfig/deps/LoadEnv/ini_files/supported-envs.ini -> /ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/ini-files/supported-envs.ini lrwxrwxrwx 1 rabartl rabartl 119 Jul 15 10:00 GenConfig/deps/LoadEnv/ini_files/supported-systems.ini -> /ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/srn-ini-files/trilinos/framework/supported-systems.ini INFO: You selected the following GenConfig ini files: lrwxrwxrwx 1 rabartl rabartl 91 Jul 15 10:00 GenConfig/ini_files/config-specs.ini -> /ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/ini-files/config-specs.ini lrwxrwxrwx 1 rabartl rabartl 101 Jul 15 10:00 GenConfig/ini_files/supported-config-flags.ini -> /ascldap/users/rabartl/Trilinos.base/Trilinos/packages/framework/ini-files/supported-config-flags.ini ... ``` and Trilinos and GenConf repo versions are: ``` $ cd Trilinos/ $ gitdist-status --dist-repos=.,packages/framework/GenConfig,packages/framework/srn-ini-files ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ID | Repo Dir | Branch | Tracking Branch | C | M | ? | |----|----------------------------------|--------------------------------------------|-------------------------------------------------------|-----|---|---| | 0 | Trilinos (Base) | tribits-299-modern-cmake-targets-1-again-2 | rab-github/tribits-299-modern-cmake-targets-1-again-2 | 301 | | | | 1 | packages/framework/GenConfig | HEAD | | | 2 | 1 | | 2 | packages/framework/srn-ini-files | tribits-299-modern-cmake-targets-1-again-2 | rab-github/tribits-299-modern-cmake-targets-1-again-2 | 301 | | | ------------------------------------------------------------------------------------------------------------------------------------------------------------ $ git log-short --name-status -1 $(git merge-base HEAD github/develop) 6c842cb13d9 "Merge Pull Request #10706 from brian-kelley/Trilinos/SpeedupAdditiveSchwarz" Author: trilinos-autotester