E3SM-Project / spack

A flexible package manager that supports multiple versions, configurations, platforms, and compilers.
https://spack.io
Other
1 stars 2 forks source link

Albany no longer building with spack on Anvil #12

Closed xylar closed 1 month ago

xylar commented 1 year ago

@matthewhoffman and I are seeing the following when we build on Albany with Gnu and OpenMPI:

Spack error message ``` ==> Installing albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o ==> No binary for albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o found: installing from source ==> No patches needed for albany ==> albany: Executing phase: 'cmake' ==> albany: Executing phase: 'build' ==> Error: ProcessError: Command exited with status 2: 'make' '-j16' 4 errors found in build log: 354 [ 31%] Building CXX object src/CMakeFiles/albanyLib.dir/disc/stk/Al bany_IossSTKMeshStruct.cpp.o 355 cd /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/ spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-b uild-hflw55t/src && /gpfs/fs1/software/centos7/spack-latest/opt/spa ck/linux-centos7-x86_64/gcc-8.2.0/openmpi-4.1.1-x5n4m36/bin/mpic++ -DALBANY_STK_EXPR_EVAL -DalbanyLib_EXPORTS -I/lcrc/group/e3sm/ac.xy lar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany-develo p-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-build-hflw55t/src -I/lcrc/ group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-sta ge-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src -I /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spa ck-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/ src/evaluators -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davi s/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wc egai5o/spack-src/src/evaluators/bc -I/lcrc/group/e3sm/ac.xylar/spac k_temp/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55 t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/gather -I/lcrc/ group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-sta ge-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/ev aluators/interpolation -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xa say-davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjf itbyl3wcegai5o/spack-src/src/evaluators/response -I/lcrc/group/e3sm /ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany- develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/s catter -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack- stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/s pack-src/src/evaluators/state -I/lcrc/group/e3sm/ac.xylar/spack_tem p/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t6asc fv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/utility -I/lcrc/grou p/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-a lbany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/proble ms -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stag e/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack -src/src/responses -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay- davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitby l3wcegai5o/spack-src/src/disc/stk -I/lcrc/group/e3sm/ac.xylar/spack _temp/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t 6ascfv7jdjfitbyl3wcegai5o/spack-src/src/disc -I/lcrc/group/e3sm/ac. xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany-deve lop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/utility -isystem /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_mache_1.12.0/opt/sp ack/linux-centos7-sandybridge/gcc-8.2.0/trilinos-for-albany-develop -geu2ne7m7fvplmdn5s7r3gzraclf6d7t/include -isystem /lcrc/group/e3sm /ac.xylar/spack_test/spack_for_mache_1.12.0/opt/spack/linux-centos7 -sandybridge/gcc-8.2.0/boost-1.80.0-i6guysn34mbdsncduqpgir432rtxrnq k/include -isystem /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_m ache_1.12.0/opt/spack/linux-centos7-sandybridge/gcc-8.2.0/metis-5.1 .0-xomfdtvdxfc2efzs2wn6sr74e5ccg7ps/include -isystem /gpfs/fs1/soft ware/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/ hdf5-1.10.7-w35u23y/include -isystem /lcrc/group/e3sm/ac.xylar/spac k_test/spack_for_mache_1.12.0/opt/spack/linux-centos7-sandybridge/g cc-8.2.0/parmetis-4.0.3-lxu66szm7wjhi7djn2is4gabcbebkvpv/include -i system /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-cent os7-x86_64/gcc-8.2.0/netcdf-c-4.4.1-mtfptpl/include -isystem /gpfs/ fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gc c-8.2.0/parallel-netcdf-1.11.0-a7ohxsg/include -O3 -DNDEBUG -fPIC - std=c++17 -MD -MT src/CMakeFiles/albanyLib.dir/disc/stk/Albany_Ioss STKMeshStruct.cpp.o -MF CMakeFiles/albanyLib.dir/disc/stk/Albany_Io ssSTKMeshStruct.cpp.o.d -o CMakeFiles/albanyLib.dir/disc/stk/Albany _IossSTKMeshStruct.cpp.o -c /lcrc/group/e3sm/ac.xylar/spack_temp/ac .xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7j djfitbyl3wcegai5o/spack-src/src/disc/stk/Albany_IossSTKMeshStruct.c pp 356 [ 32%] Building CXX object src/CMakeFiles/albanyLib.dir/disc/stk/Al bany_MultiSTKFieldContainer.cpp.o 357 cd /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/ spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-b uild-hflw55t/src && /gpfs/fs1/software/centos7/spack-latest/opt/spa ck/linux-centos7-x86_64/gcc-8.2.0/openmpi-4.1.1-x5n4m36/bin/mpic++ -DALBANY_STK_EXPR_EVAL -DalbanyLib_EXPORTS -I/lcrc/group/e3sm/ac.xy lar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany-develo p-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-build-hflw55t/src -I/lcrc/ group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-sta ge-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src -I /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spa ck-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/ src/evaluators -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davi s/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wc egai5o/spack-src/src/evaluators/bc -I/lcrc/group/e3sm/ac.xylar/spac k_temp/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55 t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/gather -I/lcrc/ group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-sta ge-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/ev aluators/interpolation -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xa say-davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjf itbyl3wcegai5o/spack-src/src/evaluators/response -I/lcrc/group/e3sm /ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany- develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/s catter -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack- stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/s pack-src/src/evaluators/state -I/lcrc/group/e3sm/ac.xylar/spack_tem p/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t6asc fv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/utility -I/lcrc/grou p/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-a lbany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/proble ms -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stag e/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack -src/src/responses -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay- davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitby l3wcegai5o/spack-src/src/disc/stk -I/lcrc/group/e3sm/ac.xylar/spack _temp/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t 6ascfv7jdjfitbyl3wcegai5o/spack-src/src/disc -I/lcrc/group/e3sm/ac. xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany-deve lop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/utility -isystem /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_mache_1.12.0/opt/sp ack/linux-centos7-sandybridge/gcc-8.2.0/trilinos-for-albany-develop -geu2ne7m7fvplmdn5s7r3gzraclf6d7t/include -isystem /lcrc/group/e3sm /ac.xylar/spack_test/spack_for_mache_1.12.0/opt/spack/linux-centos7 -sandybridge/gcc-8.2.0/boost-1.80.0-i6guysn34mbdsncduqpgir432rtxrnq k/include -isystem /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_m ache_1.12.0/opt/spack/linux-centos7-sandybridge/gcc-8.2.0/metis-5.1 .0-xomfdtvdxfc2efzs2wn6sr74e5ccg7ps/include -isystem /gpfs/fs1/soft ware/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/ hdf5-1.10.7-w35u23y/include -isystem /lcrc/group/e3sm/ac.xylar/spac k_test/spack_for_mache_1.12.0/opt/spack/linux-centos7-sandybridge/g cc-8.2.0/parmetis-4.0.3-lxu66szm7wjhi7djn2is4gabcbebkvpv/include -i system /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-cent os7-x86_64/gcc-8.2.0/netcdf-c-4.4.1-mtfptpl/include -isystem /gpfs/ fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gc c-8.2.0/parallel-netcdf-1.11.0-a7ohxsg/include -O3 -DNDEBUG -fPIC - std=c++17 -MD -MT src/CMakeFiles/albanyLib.dir/disc/stk/Albany_Mult iSTKFieldContainer.cpp.o -MF CMakeFiles/albanyLib.dir/disc/stk/Alba ny_MultiSTKFieldContainer.cpp.o.d -o CMakeFiles/albanyLib.dir/disc/ stk/Albany_MultiSTKFieldContainer.cpp.o -c /lcrc/group/e3sm/ac.xyla r/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany-develop- hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/disc/stk/Albany_Mult iSTKFieldContainer.cpp 358 /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spa ck-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/ src/disc/stk/Albany_ExtrudedSTKMeshStruct.cpp: In instantiation of 'Albany::ExtrudedSTKMeshStruct::extrudeBasalFields(const std::vecto r&, const std::vector&, GO, G O):: [with auto:6 = stk::mesh ::Field*; auto:7 = stk::mesh::Field*]': 359 /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spa ck-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/ src/disc/stk/Albany_ExtrudedSTKMeshStruct.cpp:1075:33: required f rom here >> 360 /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spa ck-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/ src/disc/stk/Albany_ExtrudedSTKMeshStruct.cpp:1057:36: internal com piler error: in tsubst_copy, at cp/pt.c:15478 361 int node3dId = Ordering==COLUMN ? node2dId*(numLayers+1) +il+1 : il*maxGlobalNodes2dId+node2dId + 1; 362 ^~~~~~ 363 [ 32%] Building CXX object src/CMakeFiles/albanyLib.dir/disc/stk/Al bany_OrdinarySTKFieldContainer.cpp.o 364 cd /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/ spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-b uild-hflw55t/src && /gpfs/fs1/software/centos7/spack-latest/opt/spa ck/linux-centos7-x86_64/gcc-8.2.0/openmpi-4.1.1-x5n4m36/bin/mpic++ -DALBANY_STK_EXPR_EVAL -DalbanyLib_EXPORTS -I/lcrc/group/e3sm/ac.xy lar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany-develo p-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-build-hflw55t/src -I/lcrc/ group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-sta ge-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src -I /lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spa ck-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/ src/evaluators -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davi s/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wc egai5o/spack-src/src/evaluators/bc -I/lcrc/group/e3sm/ac.xylar/spac k_temp/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55 t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/gather -I/lcrc/ group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-sta ge-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/ev aluators/interpolation -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xa say-davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjf itbyl3wcegai5o/spack-src/src/evaluators/response -I/lcrc/group/e3sm /ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany- develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/s catter -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack- stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/s pack-src/src/evaluators/state -I/lcrc/group/e3sm/ac.xylar/spack_tem p/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t6asc fv7jdjfitbyl3wcegai5o/spack-src/src/evaluators/utility -I/lcrc/grou p/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-a lbany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/proble ms -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay-davis/spack-stag e/spack-stage-albany-develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack -src/src/responses -I/lcrc/group/e3sm/ac.xylar/spack_temp/ac.xasay- davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7jdjfitby l3wcegai5o/spack-src/src/disc/stk -I/lcrc/group/e3sm/ac.xylar/spack _temp/ac.xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t 6ascfv7jdjfitbyl3wcegai5o/spack-src/src/disc -I/lcrc/group/e3sm/ac. xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany-deve lop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/utility -isystem /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_mache_1.12.0/opt/sp ack/linux-centos7-sandybridge/gcc-8.2.0/trilinos-for-albany-develop -geu2ne7m7fvplmdn5s7r3gzraclf6d7t/include -isystem /lcrc/group/e3sm /ac.xylar/spack_test/spack_for_mache_1.12.0/opt/spack/linux-centos7 -sandybridge/gcc-8.2.0/boost-1.80.0-i6guysn34mbdsncduqpgir432rtxrnq k/include -isystem /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_m ache_1.12.0/opt/spack/linux-centos7-sandybridge/gcc-8.2.0/metis-5.1 .0-xomfdtvdxfc2efzs2wn6sr74e5ccg7ps/include -isystem /gpfs/fs1/soft ware/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/ hdf5-1.10.7-w35u23y/include -isystem /lcrc/group/e3sm/ac.xylar/spac k_test/spack_for_mache_1.12.0/opt/spack/linux-centos7-sandybridge/g cc-8.2.0/parmetis-4.0.3-lxu66szm7wjhi7djn2is4gabcbebkvpv/include -i system /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-cent os7-x86_64/gcc-8.2.0/netcdf-c-4.4.1-mtfptpl/include -isystem /gpfs/ fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gc c-8.2.0/parallel-netcdf-1.11.0-a7ohxsg/include -O3 -DNDEBUG -fPIC - std=c++17 -MD -MT src/CMakeFiles/albanyLib.dir/disc/stk/Albany_Ordi narySTKFieldContainer.cpp.o -MF CMakeFiles/albanyLib.dir/disc/stk/A lbany_OrdinarySTKFieldContainer.cpp.o.d -o CMakeFiles/albanyLib.dir /disc/stk/Albany_OrdinarySTKFieldContainer.cpp.o -c /lcrc/group/e3s m/ac.xylar/spack_temp/ac.xasay-davis/spack-stage/spack-stage-albany -develop-hflw55t6ascfv7jdjfitbyl3wcegai5o/spack-src/src/disc/stk/Al bany_OrdinarySTKFieldContainer.cpp 365 0x5962ac tsubst_copy 366 /blues/gpfs/software/centos7/spack-latest/var/spack/stage/gcc-8.2. 0-xhxgy33yu3xpdtjpcw3f4g7wck2ptv7s/gcc-8.2.0/gcc/cp/pt.c:15476 ... 403 0x6bc878 instantiate_decl(tree_node*, bool, bool) 404 /blues/gpfs/software/centos7/spack-latest/var/spack/stage/gcc-8.2. 0-xhxgy33yu3xpdtjpcw3f4g7wck2ptv7s/gcc-8.2.0/gcc/cp/pt.c:24053 405 Please submit a full bug report, 406 with preprocessed source if appropriate. 407 Please include the complete backtrace with any bug report. 408 See for instructions. >> 409 make[2]: *** [src/CMakeFiles/albanyLib.dir/disc/stk/Albany_Extruded STKMeshStruct.cpp.o] Error 1 410 make[2]: *** Waiting for unfinished jobs.... 411 make[2]: Leaving directory `/lcrc/group/e3sm/ac.xylar/spack_temp/ac .xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7j djfitbyl3wcegai5o/spack-build-hflw55t' >> 412 make[1]: *** [src/CMakeFiles/albanyLib.dir/all] Error 2 413 make[1]: Leaving directory `/lcrc/group/e3sm/ac.xylar/spack_temp/ac .xasay-davis/spack-stage/spack-stage-albany-develop-hflw55t6ascfv7j djfitbyl3wcegai5o/spack-build-hflw55t' >> 414 make: *** [all] Error 2 ```

The change seems to be somewhat recent -- thing worked as of January 12th.

Here is what I ran:

Bash script ```bash #!/bin/bash module purge module load cmake/3.20.3-vedypwm module load perl/5.30.3-co3kopi module load gcc/8.2.0-xhxgy33 module load openmpi/4.1.1-x5n4m36 module load intel-mkl/2020.4.304-d6zw4xa module load hdf5/1.10.7-w35u23y module load netcdf-c/4.4.1-mtfptpl module load netcdf-fortran/4.4.4-5yd6dos module load parallel-netcdf/1.11.0-a7ohxsg export TMPDIR=/lcrc/group/e3sm/ac.xylar/spack_temp set -e if [ -d /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_mache_1.12.0 ]; then cd /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_mache_1.12.0 git fetch origin git reset --hard origin/spack_for_mache_1.12.0 else git clone -b spack_for_mache_1.12.0 git@github.com:E3SM-Project/spack.git /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_mache_1.12.0 cd /lcrc/group/e3sm/ac.xylar/spack_test/spack_for_mache_1.12.0 fi source share/spack/setup-env.sh spack env remove -y dev_compass_1_2_0-alpha_4_gnu_openmpi_albany >& /dev/null && \ echo "recreating environment: dev_compass_1_2_0-alpha_4_gnu_openmpi_albany" || \ echo "creating new environment: dev_compass_1_2_0-alpha_4_gnu_openmpi_albany" spack env create dev_compass_1_2_0-alpha_4_gnu_openmpi_albany /gpfs/fs1/home/ac.xylar/mpas-work/compass/simplify_local_mache/conda/build_anvil_gnu_openmpi_albany/dev_compass_1_2_0-alpha_4_gnu_openmpi_albany.yaml spack env activate dev_compass_1_2_0-alpha_4_gnu_openmpi_albany ```

Here is the yaml file:

dev_compass_1_2_0-alpha_4_gnu_openmpi_albany.yaml ```yaml spack: specs: - cmake - gcc - openmpi - intel-mkl - hdf5 - netcdf-c - netcdf-fortran - parallel-netcdf - esmf@8.2.0+mpi+netcdf~pio+pnetcdf - scorpio@1.3.2+pnetcdf~timing+internal-timing~tools+malloc - albany@develop+mpas concretizer: unify: true packages: all: compiler: [gcc@8.2.0] providers: mpi: [openmpi@4.1.1] lapack: [intel-mkl@2020.4.304] bison: externals: - spec: bison@3.0.4 prefix: /usr buildable: false bzip2: externals: - spec: bzip2@1.0.6 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/bzip2-1.0.6-mbwr6sk buildable: false curl: externals: - spec: curl@7.72.0 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/curl-7.72.0-bnppi6e buildable: false diffutils: externals: - spec: diffutils@3.7 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/diffutils-3.7-lmwmgqg buildable: false findutils: externals: - spec: findutils@4.6.0 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/findutils-4.6.0-ef3lzvf buildable: false gettext: externals: - spec: gettext@0.19.8.1 prefix: /usr buildable: false libiconv: externals: - spec: libiconv@1.16 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/intel-20.0.4/libiconv-1.16-gcutq6m buildable: false libxml2: externals: - spec: libxml2@2.9.10 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/intel-20.0.4/libxml2-2.9.10-dwvqy3m buildable: false m4: externals: - spec: m4@1.4.16 prefix: /usr buildable: false openssl: externals: - spec: openssl@1.0.2k prefix: /usr buildable: false tar: externals: - spec: tar@1.26 prefix: /usr buildable: false xz: externals: - spec: xz@5.2.5 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/intel-20.0.4/xz-5.2.5-6uopvxd buildable: false zlib: externals: - spec: zlib@1.2.11 prefix: /gpfs/fs1/soft/chrysalis/spack/opt/spack/linux-centos8-x86_64/intel-20.0.4/zlib-1.2.11-dudhhig buildable: false cmake: externals: - spec: cmake@3.20.3 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/cmake-3.20.3-vedypwm modules: - cmake/3.20.3-vedypwm buildable: false perl: externals: - spec: perl@5.30.3 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/perl-5.30.3-co3kopi modules: - perl/5.30.3-co3kopi buildable: false gcc: externals: - spec: gcc@8.2.0 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/gcc-8.2.0-xhxgy33 modules: - gcc/8.2.0-xhxgy33 buildable: false openmpi: externals: - spec: openmpi@4.1.1 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/openmpi-4.1.1-x5n4m36 modules: - openmpi/4.1.1-x5n4m36 buildable: false intel-mkl: externals: - spec: intel-mkl@2020.4.304 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/intel-mkl-2020.4.304-d6zw4xa modules: - intel-mkl/2020.4.304-d6zw4xa buildable: false hdf5: externals: - spec: hdf5@1.10.7+cxx+fortran+hl+mpi prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/hdf5-1.10.7-w35u23y modules: - hdf5/1.10.7-w35u23y buildable: false netcdf-c: externals: - spec: netcdf-c@4.4.1+mpi~parallel-netcdf prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/netcdf-c-4.4.1-mtfptpl modules: - netcdf-c/4.4.1-mtfptpl buildable: false netcdf-fortran: externals: - spec: netcdf-fortran@4.4.4 prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/netcdf-fortran-4.4.4-5yd6dos modules: - netcdf-fortran/4.4.4-5yd6dos buildable: false parallel-netcdf: externals: - spec: parallel-netcdf@1.11.0+cxx+fortran prefix: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-8.2.0/parallel-netcdf-1.11.0-a7ohxsg modules: - parallel-netcdf/1.11.0-a7ohxsg buildable: false config: install_missing_compilers: false compilers: - compiler: spec: gcc@8.2.0 paths: cc: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/gcc-8.2.0-xhxgy33/bin/gcc cxx: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/gcc-8.2.0-xhxgy33/bin/g++ f77: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/gcc-8.2.0-xhxgy33/bin/gfortran fc: /gpfs/fs1/software/centos7/spack-latest/opt/spack/linux-centos7-x86_64/gcc-6.5.0/gcc-8.2.0-xhxgy33/bin/gfortran flags: {} operating_system: centos7 target: x86_64 modules: [] environment: {} extra_rpaths: [] ```
xylar commented 1 year ago

Note that we're using a relatively old gcc (8.2.0).

xylar commented 1 year ago

There are newer compilers and libraries available. I can try those and see if they work okay. We usually try to stay in sync with E3SM but we don't have to.

xylar commented 1 year ago

As expected, I'm seeing the same error with mvapich instead of openmpi, so it isn't an MPI thing.

xylar commented 1 year ago

I can't find a more up-to-date set of compilers and MPI on Anvil to test with.

ikalash commented 1 year ago

Thanks for checking all of this @xylar . I am sad to say that 'internal compiler error' typically means there is a compiler bug, at least in my experience (and people say it online too: https://stackoverflow.com/questions/12405606/what-do-internal-compiler-error-messages-mean-and-what-can-i-do ). @jewatkins , @mperego , @bartgol , @kliegeois - is this consistent with your experience? I will say that gcc-8.2.0 is pretty old at this point... it would be interesting to see what happens with newer compilers.

I'm not sure how I can help with this. I could build a gcc-8.2.0 compiler on one of my machines and see if the error happens for me or not, to try to reproduce it. That's the only thing that comes to mind now...

bartgol commented 1 year ago

In my experience, ICEs are very susceptible to the most ridicolous microchanges. The way I usually tackle them is by isolating the code that gives the ICE, and try to manipulate it slightly (add parentheses, remove const qualifiers, reorder sums,...). If that doesn't work, I try to move the function that causes issues to a separate cpp file. All of these should have no impact from the syntax point of view, but ICEs are usually a failure of the compiler to correctly parse, so they are legit solutions.

@xylar , I would try to modify the line

          int node3dId = Ordering==COLUMN ? node2dId*(numLayers+1)+il+1 : il*maxGlobalNodes2dId+node2dId + 1;

in some silly way, like

          int node3dId = Ordering==COLUMN ? node2dId*(numLayers+1)+il+1 :  (il*maxGlobalNodes2dId+node2dId + 1);

or

          int node3dId;
         switch (Ordering) {
            case COLUMN: node3dId = node2dId*(numLayers+1)+il+1; break;
            case LAYER:  node3dId = il*maxGlobalNodes2dId+node2dId + 1; break;
         }

Whatever works to get the compiler to generate the correct parsing tree.

ikalash commented 1 year ago

@bartgol : it sounds like you're suggesting to try to reproduce the problem and come up with a sort of hack to avoid the compiler error? I could try building gcc-8.2.0 to see if I get the same error. Please stay tuned. Other suggestions are welcome...

bartgol commented 1 year ago

Well, I don't suggest to try to reproduce it. I suggest to try some mods like those on the machine that has the issue.

You can, of course, try to reproduce, but I would not be surprised if, even with the same compilers, you don't get it on another machine.

xylar commented 1 year ago

Okay, thank you both. I don't think it's worth investigating the specific error or fiddling with that line of code. I'll try compiling new compilers, MPI and the whole gamut and we'll see how it goes.

ikalash commented 1 year ago

Ok. I actually tried compiling with a gcc-8.2.0 compiler but got other errors not related to this. The compiler was from a module on one of our clusters and I think it requires other modules to be loaded to compile certain libraries, which I'd have to figure out. Please do let us know if you see this error with other newer compilers.

xylar commented 1 year ago

I've got a fix for this in https://github.com/MPAS-Dev/compass/pull/539. I think we're good. I needed to build OpenMPI with Spack (using a newer compiler) along with all the dependencies. So basically treating Anvil as an unsupported machine. But that works.

ikalash commented 1 year ago

Glad to hear you have a workaround @xylar !

xylar commented 1 year ago

@ikalash and @matthewhoffman, and update on this (as I noted in https://github.com/MPAS-Dev/compass/pull/539). It seems that MPI from spack works okay on a single node but jobs on multiple nodes just hang. I don't really have the time or expertise to debug the MPI build any further. So this means we're currently without a way to support MALI on Anvil, which is a shame. I don't think our project is high enough priority to put pressure on the LCRC folks to upgrade the version of Gnu and MPI for us.

Suggestions are welcome but I'm starting to think we have to just drop Anvil support of MALI.

ikalash commented 1 year ago

Interesting. I am not sure I have any suggestions for this... I can add this to the Agenda for the Albany meeting this Tuesday. If you'd like to discuss it, please try to join.

xylar commented 1 year ago

We have asked for a new set of compilers, MPI libraries, hdf5, netcdf-c netcdf-fortran and parallel-netcdf for both gnu and intel on Anvil. We don't know how long that might take but we hope that it will solve the issue. @jewatkins was involved in deciding which versions of packages (particularly compilers and OpenMPI) we requested, so we hope we'll be in good shape.

xylar commented 1 month ago

I think we'll let Albany on Anvil go. It doesn't seem like there's much of a push to update compilers there.

matthewhoffman commented 1 month ago

Makes sense to me.