Panzer: Intel 2021.[34].0 compiler fails with "internal error: 101003_1112" on packages/panzer/disc-fe/test/evaluator_tests/normals.cpp

PhilMiller commented 2 years ago

Bug Report

@trilinos/panzer @trilinos/framework

Description

I'm attempting to compiler Trilinos as configured for EMPIRE using the Intel 2021.3.0 compilers. The build fails thusly:

$ ninja packages/panzer/disc-fe/test/evaluator_tests/CMakeFiles/PanzerDiscFE_NormalsEvaluator.dir/normals.cpp.o
[1/1] Building CXX object packages/panzer/disc-fe/test/evaluator_tests/CMakeFiles/PanzerDiscFE_NormalsEvaluator.dir/normals.cpp.o
FAILED: packages/panzer/disc-fe/test/evaluator_tests/CMakeFiles/PanzerDiscFE_NormalsEvaluator.dir/normals.cpp.o 
/opt/openmpi/4.0/intel/bin/mpicxx  -I. -I/home/pbmille/repos/Trilinos/packages/panzer/disc-fe/test/evaluator_tests -Ipackages/panzer/disc-fe/src -I/home/pbmille/repos/Trilinos/packages/panzer/disc-fe/src -I/home/pbmille/repos/Trilinos/packages/panzer/disc-fe/src/lof -I/home/pbmille/repos/Trilinos/packages/panzer/disc-fe/src/evaluators -I/home/pbmille/repos/Trilinos/packages/panzer/disc-fe/src/responses -Ipackages/panzer/dof-mgr/src -I/home/pbmille/repos/Trilinos/packages/panzer/dof-mgr/src -Ipackages/panzer/core/src -I/home/pbmille/repos/Trilinos/packages/panzer/core/src -I/home/pbmille/repos/Trilinos/packages/tpetra/core/ext -Ipackages/tpetra/core/ext -I/home/pbmille/repos/Trilinos/packages/tpetra/core/inout -Ipackages/tpetra/core/inout -I/home/pbmille/repos/Trilinos/packages/tpetra/core/src -I/home/pbmille/repos/Trilinos/packages/tpetra/core/src/kokkos_refactor -Ipackages/tpetra/core/src -I/home/pbmille/repos/Trilinos/packages/tpetra/tsqr/src -Ipackages/tpetra/tsqr/src -I/home/pbmille/repos/Trilinos/packages/tpetra/classic/LinAlg -I/home/pbmille/repos/Trilinos/packages/tpetra/classic/NodeAPI -Ipackages/tpetra/classic/NodeAPI -Ipackages/tpetra/classic/src -I/home/pbmille/repos/Trilinos/packages/tpetra/classic/src -Ipackages/teuchos/kokkoscomm/src -I/home/pbmille/repos/Trilinos/packages/teuchos/kokkoscomm/src -Ipackages/teuchos/kokkoscompat/src -I/home/pbmille/repos/Trilinos/packages/teuchos/kokkoscompat/src -I/home/pbmille/repos/Trilinos/packages/teuchos/parameterlist/src -I/home/pbmille/repos/Trilinos/packages/teuchos/parser/src -Ipackages/teuchos/core/src -I/home/pbmille/repos/Trilinos/packages/teuchos/core/src -Ipackages/kokkos/core/src -I/home/pbmille/repos/Trilinos/packages/kokkos/core/src -Ipackages/kokkos -I/projects/sparc/tpls/cts1-bdw/boost-1.72.0/00000000/cts1-bdw_intel-19.0.5/include -I/home/pbmille/repos/Trilinos/packages/teuchos/comm/src -I/home/pbmille/repos/Trilinos/packages/teuchos/remainder/src -Ipackages/teuchos/remainder/src -I/home/pbmille/repos/Trilinos/packages/teuchos/numerics/src -Ipackages/kokkos-kernels/src -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/impl -Ipackages/kokkos-kernels/src/impl -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/impl/tpls -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/blas -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/blas/impl -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/sparse -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/sparse/impl -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/graph -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/graph/impl -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/batched -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/batched/dense -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/batched/dense/impl -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/batched/sparse -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/batched/sparse/impl -I/home/pbmille/repos/Trilinos/packages/kokkos-kernels/src/common -Ipackages/kokkos/algorithms/src -I/home/pbmille/repos/Trilinos/packages/kokkos/algorithms/src -Ipackages/kokkos/containers/src -I/home/pbmille/repos/Trilinos/packages/kokkos/containers/src -Ipackages/epetra/src -I/home/pbmille/repos/Trilinos/packages/epetra/src -Ipackages/phalanx/src -I/home/pbmille/repos/Trilinos/packages/phalanx/src -Ipackages/sacado/src -I/home/pbmille/repos/Trilinos/packages/sacado/src -I/home/pbmille/repos/Trilinos/packages/sacado/src/new_design -I/home/pbmille/repos/Trilinos/packages/sacado/src/template -I/home/pbmille/repos/Trilinos/packages/sacado/src/parameter -I/home/pbmille/repos/Trilinos/packages/sacado/src/mpl -Ipackages/intrepid2/src -I/home/pbmille/repos/Trilinos/packages/intrepid2/src/Cell -I/home/pbmille/repos/Trilinos/packages/intrepid2/src/Discretization/Basis -I/home/pbmille/repos/Trilinos/packages/intrepid2/src/Discretization/FunctionSpaceTools -I/home/pbmille/repos/Trilinos/packages/intrepid2/src/Discretization/Integration -I/home/pbmille/repos/Trilinos/packages/intrepid2/src/Orientation -I/home/pbmille/repos/Trilinos/packages/intrepid2/src/Projection -I/home/pbmille/repos/Trilinos/packages/intrepid2/src/Shared -I/home/pbmille/repos/Trilinos/packages/intrepid2/src -Ipackages/shards/src -I/home/pbmille/repos/Trilinos/packages/shards/src -I/home/pbmille/repos/Trilinos/packages/thyra/adapters/tpetra/src -I/home/pbmille/repos/Trilinos/packages/thyra/adapters/epetra/src -I/home/pbmille/repos/Trilinos/packages/thyra/core/src -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/interfaces/operator_vector/fundamental -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/interfaces/operator_vector/extended -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/support/operator_vector/client_support -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/support/operator_vector/adapter_support -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/interfaces/operator_solve/fundamental -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/interfaces/operator_solve/extended -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/support/operator_solve/client_support -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/interfaces/nonlinear/model_evaluator/fundamental -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/support/nonlinear/model_evaluator/client_support -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/interfaces/nonlinear/solvers/fundamental -I/home/pbmille/repos/Trilinos/packages/thyra/core/src/support/nonlinear/solvers/client_support -Ipackages/thyra/core/src -I/home/pbmille/repos/Trilinos/packages/thyra/core/example/operator_vector -I/home/pbmille/repos/Trilinos/packages/rtop/src -I/home/pbmille/repos/Trilinos/packages/rtop/src/interfaces -I/home/pbmille/repos/Trilinos/packages/rtop/src/support -I/home/pbmille/repos/Trilinos/packages/rtop/src/ops_lib -I/home/pbmille/repos/Trilinos/packages/rtop/src/lapack -Ipackages/rtop/src -I/home/pbmille/repos/Trilinos/packages/thyra/adapters/epetraext/src/model_evaluator -I/home/pbmille/repos/Trilinos/packages/thyra/adapters/epetraext/src/transformer -Ipackages/epetraext/src -I/home/pbmille/repos/Trilinos/packages/epetraext/src -I/home/pbmille/repos/Trilinos/packages/epetraext/src/transform -I/home/pbmille/repos/Trilinos/packages/epetraext/src/inout -I/home/pbmille/repos/Trilinos/packages/epetraext/src/coloring -I/home/pbmille/repos/Trilinos/packages/epetraext/src/model_evaluator -I/home/pbmille/repos/Trilinos/packages/epetraext/src/block -I/home/pbmille/repos/Trilinos/packages/epetraext/src/restrict -Ipackages/triutils/src -I/home/pbmille/repos/Trilinos/packages/triutils/src -Ipackages/zoltan/src -I/home/pbmille/repos/Trilinos/packages/zoltan/src/include -I/home/pbmille/repos/Trilinos/packages/zoltan/src/all -I/home/pbmille/repos/Trilinos/packages/zoltan/src/coloring -I/home/pbmille/repos/Trilinos/packages/zoltan/src/graph -I/home/pbmille/repos/Trilinos/packages/zoltan/src/ha -I/home/pbmille/repos/Trilinos/packages/zoltan/src/hier -I/home/pbmille/repos/Trilinos/packages/zoltan/src/hsfc -I/home/pbmille/repos/Trilinos/packages/zoltan/src/lb -I/home/pbmille/repos/Trilinos/packages/zoltan/src/matrix -I/home/pbmille/repos/Trilinos/packages/zoltan/src/order -I/home/pbmille/repos/Trilinos/packages/zoltan/src/par -I/home/pbmille/repos/Trilinos/packages/zoltan/src/params -I/home/pbmille/repos/Trilinos/packages/zoltan/src/tpls -I/home/pbmille/repos/Trilinos/packages/zoltan/src/phg -I/home/pbmille/repos/Trilinos/packages/zoltan/src/rcb -I/home/pbmille/repos/Trilinos/packages/zoltan/src/reftree -I/home/pbmille/repos/Trilinos/packages/zoltan/src/simple -I/home/pbmille/repos/Trilinos/packages/zoltan/src/timer -I/home/pbmille/repos/Trilinos/packages/zoltan/src/Utilities/Communication -I/home/pbmille/repos/Trilinos/packages/zoltan/src/Utilities/DDirectory -I/home/pbmille/repos/Trilinos/packages/zoltan/src/Utilities/Timer -I/home/pbmille/repos/Trilinos/packages/zoltan/src/Utilities/shared -I/home/pbmille/repos/Trilinos/packages/zoltan/src/zz -xCORE-AVX2  -g -fPIE -std=c++14 -MD -MT packages/panzer/disc-fe/test/evaluator_tests/CMakeFiles/PanzerDiscFE_NormalsEvaluator.dir/normals.cpp.o -MF packages/panzer/disc-fe/test/evaluator_tests/CMakeFiles/PanzerDiscFE_NormalsEvaluator.dir/normals.cpp.o.d -o packages/panzer/disc-fe/test/evaluator_tests/CMakeFiles/PanzerDiscFE_NormalsEvaluator.dir/normals.cpp.o -c /home/pbmille/repos/Trilinos/packages/panzer/disc-fe/test/evaluator_tests/normals.cpp
": internal error: 101003_1112

compilation aborted for /home/pbmille/repos/Trilinos/packages/panzer/disc-fe/test/evaluator_tests/normals.cpp (code 4)
ninja: build stopped: subcommand failed.

Steps to Reproduce

SHA1: Trilinos 155e45e86c2bc576f2802bc84db69b44f43d6c2c

$ module list

Currently Loaded Modules:
  1) sems-archive-env   2) sems-archive-ninja_fortran/1.8.2   3) sems-archive-git/2.10.1   4) sems-archive-cmake/3.19.1   5) intel/21.3.0   6) mkl/21.3.0   7) openmpi-intel/4.0

$ which mpicxx
/opt/openmpi/4.0/intel/bin/mpicxx

$ mpicxx --version
icpc (ICC) 2021.3.0 20210609
Copyright (C) 1985-2021 Intel Corporation.  All rights reserved.

Build log:

-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠[ Script Execution Summary ]-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠

Ran the following on eclipse-⁠login7:

build_trilinos.py \ -⁠-⁠compiler intel \ -⁠-⁠build-⁠type debug \ -⁠-⁠lib-⁠type shared \ -⁠-⁠src-⁠dir /ascldap/users/pbmille/repos/ \ -⁠-⁠ref develop \ -⁠-⁠save-⁠replay-⁠file /projects/empire/users/pbmille/build/trilinos/.build_trilinos_replay_2022-⁠01-⁠17_19.53.01.603381 \ -⁠-⁠build-⁠system ninja \ -⁠-⁠stage configure
Trilinos configured in:

/projects/empire/users/pbmille/build/trilinos/build/INTEL-⁠18.0.2_OPENMPI-⁠4.0.1-⁠DEBUG-⁠SERIAL-⁠SHARED

with:

cmake \ -⁠D Trilinos_CONFIGURE_OPTIONS_FILE:STRING=cmake/std/atdm/ATDMDevEnv.cmake,cmake/std/atdm/apps/empire/EMPIRETrilinosEnables.cmake \ -⁠D Panzer_ENABLE_EXAMPLES:BOOL=ON \ -⁠D Panzer_ENABLE_TESTS:BOOL=ON \ -⁠D TPL_ENABLE_gtest=OFF \ -⁠D CMAKE_INSTALL_PREFIX=/projects/empire/users/pbmille/build/trilinos/install/INTEL-⁠18.0.2_OPENMPI-⁠4.0.1-⁠DEBUG-⁠SERIAL-⁠SHARED \ -⁠D PanzerAdaptersSTK_CurlLaplacianExample-⁠ConvTest-⁠Quad-⁠Order-⁠4_DISABLE:BOOL=ON \ -⁠D PanzerAdaptersSTK_MixedPoissonExample-⁠ConvTest-⁠Hex-⁠Order-⁠3_DISABLE:BOOL=ON \ -⁠D TPL_ENABLE_CUSPARSE:BOOL=OFF \ -⁠G Ninja \ /home/pbmille/repos/Trilinos
Configured Trilinos with the following commit:

| Repository | SHA1 | |-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠|-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠| | Trilinos | f7fdd76 |
Load the environment used by this script with:

source /projects/empire/users/pbmille/build/trilinos/build/INTEL-⁠18.0.2_OPENMPI-⁠4.0.1-⁠DEBUG-⁠SERIAL-⁠SHARED/load_matching_env.sh
Total time: 0h 2m 32.17s
For complete details, see:

/projects/empire/users/pbmille/build/trilinos/log/2022-⁠01-⁠17_19.53.01.603636_7f7os6zq/Build_Trilinos.html

-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠[ End Script Execution Summary ]-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠-⁠

PhilMiller commented 2 years ago

After preprocessing the file to a self-contained test case with -E, I can reproduce the failure with just icpc -xCORE-AVX2 -g -fPIE -std=c++14 -c normals.cpp. When I drop the detailed arguments, the compiler succeeds. so that should ease bug isolation a bit.

PhilMiller commented 2 years ago

Just icpc -xCORE-AVX2 -c normals.cpp fails, so there's something wrong in optimization or code generation

PhilMiller commented 2 years ago

icpc -O0 -xCORE-AVX2 -c normals.cpp succeeds

PhilMiller commented 2 years ago

The compiler succeeds with -O1 and fails with -O2

PhilMiller commented 2 years ago

I'll see if I can reproduce this on a faster, less shared machine, so I can start running CReduce on it.

PhilMiller commented 2 years ago

At the very end of the file, if I comment as follows compilation still fails:

//template class normals_test2d_UnitTest<ResidualType>; normals_test2d_UnitTest<ResidualType> instance_normals_ResidualType_test2d_UnitTest("ResidualType");
template class normals_test2d_UnitTest<JacobianType>; normals_test2d_UnitTest<JacobianType> instance_normals_JacobianType_test2d_UnitTest("JacobianType");

If I comment the other instantiation instead, compilation succeeds

PhilMiller commented 2 years ago

There's at least some target specificity to the failure, maybe in vectorization, because if I run -O2 without -xCORE-AVX2 then it passes. I suspect it's not an issue in code generation, because failures take ~18 s, while successes take ~48 s.

PhilMiller commented 2 years ago

I can remove the explicit template specialization, and just leave the subsequent instantiation, and it still fails

normals_test2d_UnitTest<JacobianType> instance_normals_JacobianType_test2d_UnitTest("JacobianType");

PhilMiller commented 2 years ago

Inlining the JacobianType typedef preserves the failure.

PhilMiller commented 2 years ago

Further reduction reports to come.

PhilMiller commented 2 years ago

There's some environment variable dependence, since just running the compiler directly reports syntax errors, while loading the module and then running it hits the internal error.

PhilMiller commented 2 years ago

In response to queries from @rppawlo about memory usage, I ran under /usr/bin/time, and found a maximum resident set size of 1.2 GB, which should pose no problem on the system with 128 GB total. I also didn't see the virtual memory size blowing up in top.

PhilMiller commented 2 years ago

I've cleared up the environment dependence by passing the relevant GCC compatibility flag, so that it doesn't pick up the ancient 4.8.5 from the bare PATH.

/projects/global/toss3/compilers/intel/intel_2021/oneapi/compiler/2021.3.0/linux/bin/intel64/icpc -V -O2 -xCORE-AVX2 -gxx-name=/usr/tce/packages/gcc/gcc-6.1.0/bin/g++ -c cleaned.cpp

jhux2 commented 2 years ago

@trilinos/panzer

PhilMiller commented 2 years ago

The full set of things I've changed to drive the build to completion on Intel 2021.3.0 can be found here: https://github.com/PhilMiller/Trilinos/tree/pm/intel19

PhilMiller commented 2 years ago

The code as modified to work around the compiler failure in that one test passes the rest of the suite run by ctest

PhilMiller commented 2 years ago

With CReduce, my test case is down to 'just' 2 MB, and still falling pretty rapidly. Hopefully, I'll have something reasonable by tomorrow or later in the week.

nmhamster commented 2 years ago

@PhilMiller - can you try using the openmpi/4.0.5/intel-oneapi/2021.4.0 module on the Blake test bed to see if this is an issue in Intel 2021.4 OneAPI as well please (I think this will help with the bug report).

PhilMiller commented 2 years ago

For posterity, the test case failed with the same error on 2021.4.0

christiealappatt commented 2 years ago

I got the same error (internal error: 101003_1112). For me the Trilinos build failed when compiling kokkos/core/unit_test/serial/TestSerial_LocalDeepCopy.cpp. All the Intel compiler versions greater than 19.1 (tested till v 2022.0.2) produces the same error when compiled with optimization flag greater than O2. The problem doesn't occur for Intel versions lower than and equal to 19.0

It seems the error occurs with files having too many templates.

github-actions[bot] commented 1 year ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

PhilMiller commented 1 year ago

Nope, Intel's classic compiler is still fragile around this.

github-actions[bot] commented 7 months ago

This issue has had no activity for 365 days and is marked for closure. It will be closed after an additional 30 days of inactivity. If you would like to keep this issue open please add a comment and/or remove the MARKED_FOR_CLOSURE label. If this issue should be kept open even with no activity beyond the time limits you can add the label DO_NOT_AUTOCLOSE. If it is ok for this issue to be closed, feel free to go ahead and close it. Please do not add any comments or change any labels or otherwise touch this issue unless your intention is to reset the inactivity counter for an additional year.

github-actions[bot] commented 6 months ago

This issue was closed due to inactivity for 395 days.

trilinos / Trilinos