search

trilinos / Trilinos

Primary repository for the Trilinos Project
https://trilinos.org/
Other
1.19k stars 565 forks source link

Framework: Unit tests fail to link with Intel 19 and shared libraries #10982

Open PhilMiller opened 2 years ago

PhilMiller commented 2 years ago

Bug Report

@trilinos/framework @bartlettroscoe

When building Trilinos for EMPIRE with Intel 19 compilers and shared libraries, I encounter linking errors in various unit test binaries, with Panzer coming up first:

[18/654] Linking CXX executable packages/panzer/dof-mgr/test/fe_assembly/PanzerDofMgr_Test_FE_Assembly_HEX.exe
FAILED: packages/panzer/dof-mgr/test/fe_assembly/PanzerDofMgr_Test_FE_Assembly_HEX.exe 
: && /projects/sems/install/rhel7-x86_64/sems/compiler/intel/19.0.5/openmpi/1.10.1/bin/mpicxx -D_GLIBCXX_USE_CXX11_ABI=0     -O3 -DNDEBUG -mkl packages/panzer/dof-mgr/test/fe_assembly/CMakeFiles/PanzerDofMgr_Test_FE_Assembly_HEX.dir/test_fe_assembly_HEX.cpp.o packages/panzer/dof-mgr/test/fe_assembly/CMakeFiles/PanzerDofMgr_Test_FE_Assembly_HEX.dir/__/cartesian_topology/CartesianConnManager.cpp.o -o packages/panzer/dof-mgr/test/fe_assembly/PanzerDofMgr_Test_FE_Assembly_HEX.exe  -Wl,-rpath,/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/panzer/dof-mgr/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/intrepid2/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/shards/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/phalanx/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/sacado/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/panzer/core/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/ext:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/inout:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/core/compat:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/tpetra/tsqr/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos-kernels/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos/algorithms/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos/containers/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/epetra/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/numerics/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/remainder/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/kokkoscomm/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/comm/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/kokkoscompat/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/parameterlist/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/parser/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/teuchos/core/src:/scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED/packages/kokkos/core/src  packages/panzer/dof-mgr/src/libpanzer-dof-mgr.so.13.5  packages/intrepid2/src/libintrepid2.so.13.5  packages/shards/src/libshards.so.13.5  packages/phalanx/src/libphalanx.so.13.5  packages/sacado/src/libsacado.so.13.5  packages/panzer/core/src/libpanzer-core.so.13.5  packages/tpetra/core/ext/libtpetraext.so.13.5  packages/tpetra/core/inout/libtpetrainout.so.13.5  packages/tpetra/core/src/libtpetra.so.13.5  packages/tpetra/core/compat/libtpetraclassic.so.13.5  packages/tpetra/tsqr/src/libkokkostsqr.so.13.5  packages/kokkos-kernels/src/libkokkoskernels.so.13.5  packages/kokkos/algorithms/src/libkokkosalgorithms.so.13.5  packages/kokkos/containers/src/libkokkoscontainers.so.13.5  packages/epetra/src/libepetra.so.13.5  packages/teuchos/numerics/src/libteuchosnumerics.so.13.5  packages/teuchos/remainder/src/libteuchosremainder.so.13.5  packages/teuchos/kokkoscomm/src/libteuchoskokkoscomm.so.13.5  packages/teuchos/comm/src/libteuchoscomm.so.13.5  packages/teuchos/kokkoscompat/src/libteuchoskokkoscompat.so.13.5  packages/teuchos/parameterlist/src/libteuchosparameterlist.so.13.5  packages/teuchos/parser/src/libteuchosparser.so.13.5  packages/teuchos/core/src/libteuchoscore.so.13.5  packages/kokkos/core/src/libkokkoscore.so.13.5  -ldl && :
ld: packages/panzer/dof-mgr/test/fe_assembly/CMakeFiles/PanzerDofMgr_Test_FE_Assembly_HEX.dir/test_fe_assembly_HEX.cpp.o(.text+0xc38c): unresolvable R_X86_64_TPOFF32 relocation against symbol `_ZN6Kokkos4Impl22SharedAllocationRecordIvvE18t_tracking_enabledE'
ld: final link failed: Nonrepresentable section on output

This looks exactly like kokkos/kokkos#4722, and potentially has the same cause - setting CMAKE_POSITION_INDEPENDENT_CODE where it's not needed, since BUILD_SHARED_LIBRARIES is sufficient.

With this patch, the relevant executable builds successfully:

diff --git a/cmake/std/atdm/ATDMDevEnvSettings.cmake b/cmake/std/atdm/ATDMDevEnvSettings.cmake
index c5a6c1c..f35157d 100644
--- a/cmake/std/atdm/ATDMDevEnvSettings.cmake
+++ b/cmake/std/atdm/ATDMDevEnvSettings.cmake
@@ -217,10 +217,6 @@ IF ((ATDM_COMPILER MATCHES ".*CLANG.*") AND ATDM_ADDRESS_SANITIZER)
   SET(EXTRA_EXTRA_LINK_FLAGS
     "${EXTRA_EXTRA_LINK_FLAGS} -ldl -fsanitize=address")
 ENDIF()
-# FPIC
-IF (ATDM_FPIC)
-  SET(CMAKE_POSITION_INDEPENDENT_CODE ON CACHE BOOL "Build targets with position independent code")
-ENDIF()

 SET(CMAKE_C_FLAGS
   "$ENV{ATDM_CONFIG_C_FLAGS} ${CMAKE_C_FLAGS}")

I also see it getting set in these places, but these changes or commenting them out made no difference in my case:

diff --git a/packages/framework/ini-files/config-specs.ini b/packages/framework/ini-files/config-specs.ini
index 0fd0f53..eda3f53 100644
--- a/packages/framework/ini-files/config-specs.ini
+++ b/packages/framework/ini-files/config-specs.ini
@@ -670,7 +670,7 @@ opt-set-cmake-var Trilinos_ENABLE_COMPLEX BOOL : OFF
 #

 [USE-FPIC|YES]
-opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : ON
+opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : OFF

 [USE-FPIC|NO]
 # Nothing to do here.
diff --git a/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini b/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini
index 6174dd7..583e480 100644
--- a/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini
+++ b/packages/framework/pr_tools/trilinosprhelpers/unittests/config-specs.ini
@@ -606,7 +606,7 @@ opt-set-cmake-var Trilinos_ENABLE_COMPLEX BOOL : OFF
 #

 [USE-FPIC|YES]
-opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : ON
+opt-set-cmake-var CMAKE_POSITION_INDEPENDENT_CODE BOOL : OFF

 [USE-FPIC|NO]
 # Nothing to do here.

Steps to Reproduce

  1. SHA1: 29305794b0b1e9b03ea75d0af4597a2df4de50b4
+ Ran the following on ascic169:

    build_trilinos.py \
      --compiler intel-19 \
      --build-type opt \
      --node-type serial \
      --lib-type shared \
      --src-dir /ascldap/users/pbmille/repos \
      --ref develop \
      --save-replay-file /scratch/pbmille/trilinos/.build_trilinos_replay_2022-09-01_18.24.51.014436 \
      --build-system ninja \
      --stage configure

+ Trilinos configured in:

    /scratch/pbmille/trilinos/build/INTEL-19.0.5-RELEASE-SERIAL-SHARED

    with:

    cmake \
      -D Trilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake,cmake/std/atdm/apps/empire/EMPIRETrilinosEnables.cmake \
      -D Panzer_ENABLE_EXAMPLES:BOOL=ON \
      -D Panzer_ENABLE_TESTS:BOOL=ON \
      -D TPL_ENABLE_gtest=OFF \
      -D Trilinos_ENABLE_Tempus=OFF \
      -D CMAKE_CXX_COMPILER_LAUNCHER=$(which ccache) \
      -D CMAKE_C_COMPILER_LAUNCHER=$(which ccache) \
      -D CMAKE_INSTALL_PREFIX=/scratch/pbmille/trilinos/install/INTEL-19.0.5-RELEASE-SERIAL-SHARED \
      -D TPL_ENABLE_CGNS=ON \
        -D CGNS_LIBRARY_DIRS=${CGNS_ROOT}/lib \
        -D CGNS_INCLUDE_DIRS=${CGNS_ROOT}/include \
      -D TPL_ENABLE_CUSPARSE:BOOL=OFF \
      -G Ninja \
      /home/pbmille/repos/Trilinos
rppawlo commented 2 years ago

Thanks for the workaround @PhilMiller ! I've filed a ticket for this with the framework team so they are aware of it - TRILINOSHD-199.

bartlettroscoe commented 2 years ago

Hello @PhilMiller, if you post a PR for the updates to files under cmake/std/atdm/ I will approve.

PhilMiller commented 2 years ago

https://github.com/trilinos/Trilinos/pull/10984 @bartlettroscoe There's the PR

github-actions[bot] commented 1 year ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

PhilMiller commented 1 year ago

Not resolved. Don't autoclose

github-actions[bot] commented 6 days ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.