E4S-Project / e4s

E4S for Spack
https://e4s.readthedocs.io
MIT License
30 stars 13 forks source link

[support]: Bugs from Perlmutter E4S 21.11 tests using E4S Testsuite #64

Closed shahzebsiddiqui closed 2 years ago

shahzebsiddiqui commented 2 years ago

Name of Software

testsuite

URL for Software

https://github.com/E4S-Project/testsuite

Contact Details

shahzebsiddiqui@lbl.gov

HPC System

Perlmutter

Request Description

We have this test https://software.nersc.gov/NERSC/buildtest-nersc/-/blob/devel/buildspecs/e4s/E4S-Testsuite/e4s_21.11_perlmutter.yml that we run against the Perlmutter E4S stack using buildtest. This runs several tests for different products and we noticed several errors that can be seen in CDASH https://my.cdash.org/test/56389573.

Relevant log output

Cloning into 'testsuite'...
Updating files:  26% (3751/13920)
Updating files:  27% (3759/13920)
Updating files:  28% (3898/13920)
Updating files:  29% (4037/13920)
Updating files:  30% (4176/13920)
Updating files:  31% (4316/13920)
Updating files:  32% (4455/13920)
Updating files:  33% (4594/13920)
Updating files:  33% (4645/13920)
Updating files:  34% (4733/13920)
Updating files:  35% (4872/13920)
Updating files:  36% (5012/13920)
Updating files:  37% (5151/13920)
Updating files:  38% (5290/13920)
Updating files:  39% (5429/13920)
Updating files:  40% (5568/13920)
Updating files:  41% (5708/13920)
Updating files:  42% (5847/13920)
Updating files:  43% (5986/13920)
Updating files:  44% (6125/13920)
Updating files:  45% (6264/13920)
Updating files:  46% (6404/13920)
Updating files:  47% (6543/13920)
Updating files:  48% (6682/13920)
Updating files:  49% (6821/13920)
Updating files:  50% (6960/13920)
Updating files:  51% (7100/13920)
Updating files:  52% (7239/13920)
Updating files:  53% (7378/13920)
Updating files:  54% (7517/13920)
Updating files:  55% (7656/13920)
Updating files:  56% (7796/13920)
Updating files:  56% (7855/13920)
Updating files:  57% (7935/13920)
Updating files:  58% (8074/13920)
Updating files:  59% (8213/13920)
Updating files:  60% (8352/13920)
Updating files:  60% (8387/13920)
Updating files:  61% (8492/13920)
Updating files:  62% (8631/13920)
Updating files:  62% (8729/13920)
Updating files:  63% (8770/13920)
Updating files:  64% (8909/13920)
Updating files:  65% (9048/13920)
Updating files:  66% (9188/13920)
Updating files:  67% (9327/13920)
Updating files:  68% (9466/13920)
Updating files:  69% (9605/13920)
Updating files:  70% (9744/13920)
Updating files:  71% (9884/13920)
Updating files:  72% (10023/13920)
Updating files:  72% (10155/13920)
Updating files:  73% (10162/13920)
Updating files:  74% (10301/13920)
Updating files:  75% (10440/13920)
Updating files:  76% (10580/13920)
Updating files:  77% (10719/13920)
Updating files:  78% (10858/13920)
Updating files:  79% (10997/13920)
Updating files:  80% (11136/13920)
Updating files:  81% (11276/13920)
Updating files:  82% (11415/13920)
Updating files:  82% (11423/13920)
Updating files:  83% (11554/13920)
Updating files:  84% (11693/13920)
Updating files:  85% (11832/13920)
Updating files:  86% (11972/13920)
Updating files:  87% (12111/13920)
Updating files:  88% (12250/13920)
Updating files:  89% (12389/13920)
Updating files:  90% (12528/13920)
Updating files:  91% (12668/13920)
Updating files:  92% (12807/13920)
Updating files:  93% (12946/13920)
Updating files:  94% (13085/13920)
Updating files:  95% (13224/13920)
Updating files:  96% (13364/13920)
Updating files:  97% (13503/13920)
Updating files:  98% (13642/13920)
Updating files:  98% (13662/13920)
Updating files:  99% (13781/13920)
Updating files: 100% (13920/13920)
Updating files: 100% (13920/13920), done.
3ajgfaj
Cleaning /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/gasnet
---CLEANUP LOG---
Compiling /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/gasnet
---COMPILE LOG---
Skipping load: Environment already setup
cc  `pkg-config --cflags gasnet-smp-par` -I. -c testbarrier.c
Package gasnet-smp-par was not found in the pkg-config search path.
Perhaps you should add the directory containing `gasnet-smp-par.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gasnet-smp-par' found
testbarrier.c:7:10: fatal error: gasnetex.h: No such file or directory
    7 | #include <gasnetex.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:24: testbarrier.o] Error 1
cc  `pkg-config --cflags gasnet-smp-par` -I. -c testbarrier.c
Package gasnet-smp-par was not found in the pkg-config search path.
Perhaps you should add the directory containing `gasnet-smp-par.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gasnet-smp-par' found
testbarrier.c:7:10: fatal error: gasnetex.h: No such file or directory
    7 | #include <gasnetex.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:24: testbarrier.o] Error 1
Compile failed
7a2enxi
Cleaning /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/kokkos
---CLEANUP LOG---
Compiling /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/kokkos
---COMPILE LOG---
Skipping load: Environment already setup
+ mkdir -p build
+ cd build
+ cmake ..
-- The CXX compiler identification is GNU 11.2.0
-- The Fortran compiler identification is GNU 11.2.0
-- Check for working CXX compiler: /opt/cray/pe/gcc/11.2.0/bin/c++
-- Check for working CXX compiler: /opt/cray/pe/gcc/11.2.0/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working Fortran compiler: /opt/cray/pe/gcc/11.2.0/bin/gfortran
-- Check for working Fortran compiler: /opt/cray/pe/gcc/11.2.0/bin/gfortran - works
-- Detecting Fortran compiler ABI info
-- Detecting Fortran compiler ABI info - done
-- Checking whether /opt/cray/pe/gcc/11.2.0/bin/gfortran supports Fortran 90
-- Checking whether /opt/cray/pe/gcc/11.2.0/bin/gfortran supports Fortran 90 - yes
-- Setting policy CMP0074 to use <Package>_ROOT variables
-- Enabled Kokkos devices: OPENMP;SERIAL
-- Configuring done
CMake Error at CMakeLists.txt:31 (target_link_libraries):
  Error evaluating generator expression:

    $<LINK_LANGUAGE:CXX>

  Expression did not evaluate to a known generator expression

CMake Error at CMakeLists.txt:35 (target_link_libraries):
  Error evaluating generator expression:

    $<LINK_LANGUAGE:CXX>

  Expression did not evaluate to a known generator expression

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.
Compile failed
pio2jf2
Cleaning /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/superlu
---CLEANUP LOG---
Compiling /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/superlu
---COMPILE LOG---
Skipping load: Environment already setup
+ cc -g ./c_sample.c -I/global/common/software/spackecp/perlmutter/e4s-21.11/software/cray-sles15-zen3/gcc-11.2.0/superlu-5.3.0-pio2jf2ule3b77kfvano5fu2crumrmp5/include/ -L/global/common/software/spackecp/perlmutter/e4s-21.11/software/cray-sles15-zen3/gcc-11.2.0/superlu-5.3.0-pio2jf2ule3b77kfvano5fu2crumrmp5/lib64 -L/global/common/software/spackecp/perlmutter/e4s-21.11/software/cray-sles15-zen3/gcc-11.2.0/superlu-5.3.0-pio2jf2ule3b77kfvano5fu2crumrmp5/lib -L/lib -lsuperlu -lopenblas -lm -o c_sample
Running /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/superlu
Skipping load: Environment already setup
+ ./c_sample
31 May 2022 09:23:44 AM

C_SAMPLE:
  C version
  SUPERLU solves a single precision complex linear system.

  CC matrix:
     #     I     J         A.r               A.i
  ----  ----  ----  ----------------  ----------------

     0     0     0                19                 0
     1     1     0                12                 0
     2     4     0                12                 0
     3     1     1                21                 0
     4     2     1                12                 0
     5     4     1                12                 0
     6     0     2                21                 0
     7     2     2                16                 0
     8     0     3                21                 0
     9     3     3                 5                 0
    10     3     4                21                 0
    11     4     4                18                 0

CompCol matrix A:
Stype 0, Dtype 2, Mtype 0
nrow 5, ncol 5, nnz 12
nzval: 19.000000  0.000000  12.000000  0.000000  12.000000  0.000000  21.000000  0.000000  12.000000  0.000000  12.000000  0.000000  21.000000  0.000000  16.000000  0.000000  21.000000  0.000000  5.000000  0.000000  21.000000  0.000000  18.000000  0.000000  
rowind: 0  1  4  1  2  4  0  2  0  3  3  4  
colptr: 0  3  6  8  10  12  

CompCol matrix U:
Stype 0, Dtype 2, Mtype 4
nrow 5, ncol 5, nnz 15
nzval: 
rowind: 
colptr: 0  0  0  0  0  0  

SuperNode matrix L:
Stype 3, Dtype 2, Mtype 1
nrow 5, ncol 5, nnz 15, nsuper 0
nzval:
0   0   1.900000e+01    0.000000e+00
1   0   6.315789e-01    0.000000e+00
2   0   0.000000e+00    0.000000e+00
3   0   0.000000e+00    0.000000e+00
4   0   6.315789e-01    0.000000e+00
0   1   0.000000e+00    0.000000e+00
1   1   0.000000e+00    6.315789e-01
2   1   0.000000e+00    0.000000e+00
3   1   0.000000e+00    2.100000e+01
4   1   0.000000e+00    5.714286e-01
0   2   0.000000e+00    0.000000e+00
1   2   2.100000e+01    0.000000e+00
2   2   5.714286e-01    0.000000e+00
3   2   0.000000e+00    0.000000e+00
4   2   5.714286e-01    0.000000e+00
0   3   0.000000e+00    0.000000e+00
1   3   0.000000e+00    5.714286e-01
2   3   0.000000e+00    2.100000e+01
3   3   0.000000e+00    -1.326316e+01
4   3   0.000000e+00    2.357895e+01
0   4   2.100000e+01    0.000000e+00
1   4   -1.326316e+01   0.000000e+00
2   4   2.357895e+01    0.000000e+00
3   4   0.000000e+00    0.000000e+00
4   4   -2.410714e-01   0.000000e+00

nzval_colptr: 0  5  10  15  20  25  
rowind: 0  1  2  3  4  
rowind_colptr: 0  5  5  5  5  5  
col_to_sup: 0  0  0  0  0  
sup_to_col: 0  5  

perm_r
0   0
1   1
2   2
3   3
4   4

  Computed solution:

-0.03125  0
0.0654762  -0
0.0133929  -0
0.0625  0
0.0327381  0

  Product A*X:

1  0
1  0
1  0
1  0
1  0

C_SAMPLE:
  Normal end of execution.

31 May 2022 09:23:44 AM
Success
bwwdj5i
Running /global/cfs/cdirs/m3503/buildtest/runs/perlmutter_e4s/2022-05-31/perlmutter.slurm.regular/e4s_21.11_perlmutter/e4s_testsuite_for_e4s_21.11/ba52b55b/stage/testsuite/validation_tests/superlu-dist
Skipping load: Environment already setup
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16' 'pddrive'

16 errors found in test log:
     5     ==> [2022-05-31-09:24:05.390602] FILTER FILE: /global/common/softwar
           e/spackecp/perlmutter/e4s-21.11/software/cray-sles15-zen3/gcc-11.2.0
           /superlu-dist-6.4.0-bwwdj5i4e7calhpgeew52y4lmc27qwbx/.spack/test/mak
           e.inc [replacing "^LIBS[^\+]+=.+"]
     6     ==> [2022-05-31-09:24:05.393830] FILTER FILE: /global/common/softwar
           e/spackecp/perlmutter/e4s-21.11/software/cray-sles15-zen3/gcc-11.2.0
           /superlu-dist-6.4.0-bwwdj5i4e7calhpgeew52y4lmc27qwbx/.spack/test/mak
           e.inc [replacing "^LOADOPTS.+"]
     7     ==> [2022-05-31-09:24:05.395456] 'make' '-j16' 'pddrive'
     8     /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/bin/mpicxx -Wl,-rpath,/global/
           common/software/spackecp/perlmutter/e4s-21.11/software/cray-sles15-z
           en3/gcc-11.2.0/superlu-dist-6.4.0-bwwdj5i4e7calhpgeew52y4lmc27qwbx/l
           ib,-rpath,/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib,-rpath,/o
           pt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib,-rpath,/global/common
           /software/spackecp/perlmutter/e4s-21.11/software/cray-sles15-zen3/gc
           c-11.2.0/parmetis-4.0.3-lq5ude6n7ksg43sazobsy6gghvhj4vhy/lib,-rpath,
           /global/common/software/spackecp/perlmutter/e4s-21.11/software/cray-
           sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagulm7tihcav7q/
           lib pddrive.o dcreate_matrix.o sp_ienv.o  /global/common/software/sp
           ackecp/perlmutter/e4s-21.11/software/cray-sles15-zen3/gcc-11.2.0/sup
           erlu-dist-6.4.0-bwwdj5i4e7calhpgeew52y4lmc27qwbx/lib/libsuperlu_dist
           .so -L/opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib -lsci_gnu -L/
           opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib -lsci_gnu -L/global/
           common/software/spackecp/perlmutter/e4s-21.11/software/cray-sles15-z
           en3/gcc-11.2.0/parmetis-4.0.3-lq5ude6n7ksg43sazobsy6gghvhj4vhy/lib -
           lparmetis -L/global/common/software/spackecp/perlmutter/e4s-21.11/so
           ftware/cray-sles15-zen3/gcc-11.2.0/metis-5.1.0-lxe5bhakcmkcf7zuqcagu
           lm7tihcav7q/lib -lmetis  -lm -o pddrive
     9     /usr/bin/ld: warning: libgfortran.so.5, needed by /opt/cray/pe/libsc
           i/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so, not found (try using -
           rpath or -rpath-link)
     10    /usr/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/mpich/8
           .1.13/ofi/gnu/9.1/lib/libmpi_gnu_91.so, not found (try using -rpath 
           or -rpath-link)
  >> 11    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_concat_string@@GFORTRAN_8'
  >> 12    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_os_error@@GFORTRAN_8'
  >> 13    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_compare_string@@GFORTRAN_8
           '
  >> 14    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_st_write_done@@GFORTRAN_8'
  >> 15    /usr/bin/ld: /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpi_gnu_91
           .so: undefined reference to `fi_strerror@@FABRIC_1.0'
  >> 16    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_runtime_error@@GFORTRAN_8'
  >> 17    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_transfer_real_write@@GFORT
           RAN_8'
  >> 18    /usr/bin/ld: /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpi_gnu_91
           .so: undefined reference to `fi_dupinfo@@FABRIC_1.3'
  >> 19    /usr/bin/ld: /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpi_gnu_91
           .so: undefined reference to `fi_fabric@@FABRIC_1.1'
  >> 20    /usr/bin/ld: /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpi_gnu_91
           .so: undefined reference to `fi_getinfo@@FABRIC_1.3'
  >> 21    /usr/bin/ld: /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpi_gnu_91
           .so: undefined reference to `fi_freeinfo@@FABRIC_1.3'
  >> 22    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_st_write@@GFORTRAN_8'
  >> 23    /usr/bin/ld: /opt/cray/pe/mpich/8.1.13/ofi/gnu/9.1/lib/libmpi_gnu_91
           .so: undefined reference to `fi_version@@FABRIC_1.0'
  >> 24    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_transfer_character_write@@
           GFORTRAN_8'
  >> 25    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_get_environment_variable_
           i4@@GFORTRAN_8'
  >> 26    collect2: error: ld returned 1 exit status
     27    make: *** [Makefile:69: pddrive] Error 1

See test log for details:
  /global/homes/e/e4s/.spack/test/pxtixayt22tcl6suoglgbcn7nkw2oze4/superlu-dist-6.4.0-bwwdj5i-test-out.txt

==> Error: 1 test(s) in the suite failed.

Run failed

Reproduce Bug

Take a look at https://software.nersc.gov/NERSC/buildtest-nersc/-/blob/devel/.gitlab-ci.yml.

See the content of this section

perlmutter_e4s:
  stage: test
  tags: [perlmutter-e4s]
  only:
    refs:
      - schedules
    variables:
      - $TESTNAME == "perlmutter_e4s"
  before_script:
    - *setup-buildtest   
  script:
    - buildtest buildspec find --rebuild --root $CI_PROJECT_DIR/buildspecs
    - buildtest build -t e4s --account m3503_g --testdir $CFS/m3503/buildtest/runs/$CI_JOB_NAME/$(date +%F)
    - buildtest report --filter state=FAIL
    - buildtest cdash upload e4s
  after_script:
    - *cleanup-buildtest
  artifacts:
    paths:
      - $CI_PROJECT_DIR/.artifacts

If you want to rerun the pipeline just go to https://software.nersc.gov/NERSC/buildtest-nersc/-/pipeline_schedules and press "Play" under Scheduled Pipeline

Screen Shot 2022-05-31 at 12 51 00 PM

wspear commented 2 years ago

This covers the run of the e4s testsuite's gasnet, kokkos, superlu and superlu-dist tests on perlmutter, targeting e4s 21.11.

Gasnet test code failed to compile like:

cc  `pkg-config --cflags gasnet-smp-par` -I. -c testbarrier.c
Package gasnet-smp-par was not found in the pkg-config search path.
Perhaps you should add the directory containing `gasnet-smp-par.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gasnet-smp-par' found
testbarrier.c:7:10: fatal error: gasnetex.h: No such file or directory
    7 | #include <gasnetex.h>
      |          ^~~~~~~~~~~~
compilation terminated.
make: *** [Makefile:24: testbarrier.o] Error 1
cc  `pkg-config --cflags gasnet-smp-par` -I. -c testbarrier.c
Package gasnet-smp-par was not found in the pkg-config search path.
Perhaps you should add the directory containing `gasnet-smp-par.pc'
to the PKG_CONFIG_PATH environment variable
No package 'gasnet-smp-par' found
testbarrier.c:7:10: fatal error: gasnetex.h: No such file or directory
    7 | #include <gasnetex.h>
      |          ^~~~~~~~~~~~
compilation terminated.

Kokkos test code cmake failed like:


-- Configuring done
CMake Error at CMakeLists.txt:31 (target_link_libraries):
  Error evaluating generator expression:

    $<LINK_LANGUAGE:CXX>

  Expression did not evaluate to a known generator expression

CMake Error at CMakeLists.txt:35 (target_link_libraries):
  Error evaluating generator expression:

    $<LINK_LANGUAGE:CXX>

  Expression did not evaluate to a known generator expression

-- Generating done
CMake Generate step failed.  Build files cannot be regenerated correctly.

Superlu dist spack test failed like:

     9     /usr/bin/ld: warning: libgfortran.so.5, needed by /opt/cray/pe/libsc
           i/21.08.1.2/GNU/9.1/x86_64/lib/libsci_gnu.so, not found (try using -
           rpath or -rpath-link)
     10    /usr/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/mpich/8
           .1.13/ofi/gnu/9.1/lib/libmpi_gnu_91.so, not found (try using -rpath 
           or -rpath-link)
  >> 11    /usr/bin/ld: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/lib/libsci
           _gnu.so: undefined reference to `_gfortran_concat_string@@GFORTRAN_8'

With numerous additional undefined symbols.

Todo: Investigate and correct any relevant problems with test construction. Determine if any package-side issues are still present as of the 22.05 e4s release and escalate as needed.

shahzebsiddiqui commented 2 years ago

Thanks for looking into this. I just realized that it may have been good to post this issue directly in https://github.com/E4S-Project/testsuite instead of this project.

wspear commented 2 years ago

I don't think the testsuite repo should get bug reports unless they're with the functionality of the testsuite itself or they've already been determined to be failures caused by the construction of the test. In general issues identified by the testsuite should go to the e4s tracker for triage as you've done here.

wspear commented 2 years ago

The superlu-dist error seems to be fixed in 22.05 but another error in the package's test configuration was present. This PR should have it running: https://github.com/spack/spack/pull/30970

wspear commented 2 years ago

The gasnet and kokkos tests run successfully on the e4s 22.05 test deployment on perlmutter. I will check these again on the official NERSC 22.05 deployment when it becomes available. The superlu-dist test will fail for 22.05 unless we backport the above PR, once it's merged.

wspear commented 2 years ago

This issue is resolved by: https://github.com/E4S-Project/testsuite/commit/3d274f5e88e00bc81149c2fb19b71378bcfa66e2 https://software.nersc.gov/NERSC/buildtest-nersc/-/merge_requests/53 https://software.nersc.gov/NERSC/buildtest-nersc/-/merge_requests/61 https://software.nersc.gov/NERSC/buildtest-nersc/-/merge_requests/50

With this buildspec split into specs for the individual tests there may be new issues to report on a per-test basis.