search

sandialabs / LCM

Laboratory for Computational Mechanics
Other
12 stars 7 forks source link

ACE MiniErosion test with denudation failing on Proxima/Algol post Fedora 35 update #52

Closed ikalash closed 2 years ago

ikalash commented 2 years ago

The ACE MiniErosion test with denudation began failing after Algol were upgraded to Fedora 35 when the code is built with a Clang compiler. FPEs are encountered in the test:

https://sems-cdash-son.sandia.gov/cdash/test/1879062

-- Nonlinear Solver Step 0 -- 
||F|| = 2.557e+03  step = 0.000e+00  dx = 0.000e+00
************************************************************************

[algol:2823662:0:2823662] Caught signal 8 (Floating point exception: floating-point invalid operation)
==== backtrace (tid:2823662) ====
 0  /usr/lib64/libucs.so.0(ucs_handle_error+0x2a4) [0x7fe851d05864]
 1  /usr/lib64/libucs.so.0(+0x2a41d) [0x7fe851d0941d]
 2  /usr/lib64/libucs.so.0(+0x2a6fa) [0x7fe851d096fa]
 3  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libifpack2.so.13(_ZN7Ifpack27Details9ChebyshevIdN6Tpetra11MultiVectorIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS4_6SerialENS4_9HostSpaceEEEEEE7computeEv+0x41e) [0x7fe85a3354ae]
 4  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libifpack2.so.13(_ZN7Ifpack29ChebyshevIN6Tpetra9RowMatrixIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS3_6SerialENS3_9HostSpaceEEEEEE7computeEv+0x173) [0x7fe85a231843]
 5  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu15Ifpack2SmootherIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE14SetupChebyshevERNS_5LevelE+0xa52) [0x7fe85bc98aa2]
 6  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu15Ifpack2SmootherIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5SetupERNS_5LevelE+0xcee) [0x7fe85bc92e8e]
 7  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu16TrilinosSmootherIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5SetupERNS_5LevelE+0xd5) [0x7fe85bf888d5]
 8  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZNK5MueLu15SmootherFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE13BuildSmootherERNS_5LevelENS_9PreOrPostE+0x6e3) [0x7fe85bf05693]
 9  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu-adapters.so.13(_ZNK5MueLu22SingleLevelFactoryBase9CallBuildERNS_5LevelE+0x38c) [0x7fe85c8f491c]
10  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu-adapters.so.13(_ZN5MueLu5Level3GetIN7Teuchos3RCPINS_12SmootherBaseIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS5_6SerialENS5_9HostSpaceEEEEEEEEERT_RKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEPKNS_11FactoryBaseE+0x13f) [0x7fe85c8eea5f]
11  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZNK5MueLu18TopSmootherFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5BuildERNS_5LevelE+0x185) [0x7fe85bf80c15]
12  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu.so.13(_ZN5MueLu9HierarchyIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE5SetupEiN7Teuchos3RCPIKNS_18FactoryManagerBaseEEESC_SC_+0x2877) [0x7fe85bc4ef87]
13  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libmuelu-interface.so.13(_ZNK5MueLu16HierarchyManagerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE14SetupHierarchyERNS_9HierarchyIdixS6_EE+0x16bc) [0x7fe85c4f76bc]
14  /home/lcm/LCM/lcm-build-serial-clang-release/src/libalbanyLib.so(_ZN5MueLu26CreateXpetraPreconditionerIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEEEN7Teuchos3RCPINS_9HierarchyIT_T0_T1_T2_EEEENS8_IN6Xpetra6MatrixISA_SB_SC_SD_EEEERKNS7_13ParameterListE+0xd46) [0x7fe860a11956]
15  /home/lcm/LCM/lcm-build-serial-clang-release/src/libalbanyLib.so(_ZNK5Thyra26MueLuPreconditionerFactoryIdixN6Kokkos6Compat23KokkosDeviceWrapperNodeINS1_6SerialENS1_9HostSpaceEEEE14initializePrecERKN7Teuchos3RCPIKNS_18LinearOpSourceBaseIdEEEEPNS_18PreconditionerBaseIdEENS_16ESupportSolveUseE+0x43c6) [0x7fe860a08bc6]
16  /home/lcm/LCM/trilinos-install-serial-clang-release/lib/libnox.so.13(_ZNK3NOX5Thyra5Group10updateLOWSEv+0x4e4) [0x7fe85b016b44]

Curiously, the problem does not show up in the gcc build.

@lxmota : would you be able to provide instructions for how I can recreate the nightly clang build on algol to try to debug the problem in the serial run?

lxmota commented 2 years ago

Yes, I recommend working on the nightly test directory to minimize the effort:

cd /home/lcm/LCM
source ./env-all.sh
cd "$LCM_DIR"
module purge
module load serial-clang-release
./clean-config-build.sh trilinos 72 -V && ./clean-config-build.sh lcm 72 -V

if you want a debug build just change this line:

module load serial-clang-debug

ikalash commented 2 years ago

Curiously the issue does not show up in a debug build... unclear how to proceed. Perhaps one can try switching to an Ifpack2 preconditioner, since the FPE is in MueLu. I can't try this unfortunately since I don't have permissions to the /home/lcm clang release build from the nightlies.

ikalash commented 2 years ago

Ok, so it looks like the initial nonlinear solves fail for awhile (maybe the time-step is too big initially), and for some reason, MueLu doesn't like this and barfs with clang. You can see the failed initial nonlinear solves in the gcc build, which runs: https://sems-cdash-son.sandia.gov/cdash/test/1878483 (||F|| stagnates). Switching to Ifpack2 circumvents the problem and leads to the tests passing. I'll check in the fix now and close this tomorrow if the case tests clean.

lxmota commented 2 years ago

@ikalash Loads of fun! Thanks for digging into this!

ikalash commented 2 years ago

No problem! This was way easier than issue #51 !