search

flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 49 forks source link

flux-core@0.28.0%gcc@11.2.0 on perlmutter issue with libzmq for E4S/21.08 #3923

Open shahzebsiddiqui opened 2 years ago

shahzebsiddiqui commented 2 years ago

I am trying to build e4s/21.08 on perlmutter which is based on spack branch https://github.com/spack/spack/tree/e4s-21.08. Here is my concretized output for flux-core I am using this with gcc@11.2.0 compiler. I got all the dependencies built

siddiq90@login37> spack spec -I flux-core
Input spec
--------------------------------
 -   flux-core

Concretized
--------------------------------
 -   flux-core@0.28.0%gcc@11.2.0~cuda~docs arch=cray-sles15-zen3
[+]      ^czmq@4.1.1%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^autoconf@2.69%gcc@11.2.0 arch=cray-sles15-zen3
[+]              ^m4@1.4.18%gcc@11.2.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=cray-sles15-zen3
[+]              ^perl@5.34.0%gcc@11.2.0+cpanm+shared+threads arch=cray-sles15-zen3
[+]                  ^berkeley-db@18.1.40%gcc@11.2.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=cray-sles15-zen3
[+]                  ^bzip2@1.0.6%gcc@11.2.0~debug~pic+shared arch=cray-sles15-zen3
[+]                  ^gdbm@1.19%gcc@11.2.0 arch=cray-sles15-zen3
[+]                      ^readline@8.1%gcc@11.2.0 arch=cray-sles15-zen3
[+]                          ^ncurses@6.2%gcc@11.2.0~symlinks+termlib abi=none arch=cray-sles15-zen3
[+]                              ^pkg-config@0.29.2%gcc@11.2.0+internal_glib arch=cray-sles15-zen3
[+]                  ^zlib@1.2.11%gcc@11.2.0+optimize+pic+shared arch=cray-sles15-zen3
[+]          ^automake@1.16.3%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^libtool@2.4.6%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^libzmq@4.3.3%gcc@11.2.0~drafts+libsodium patches=8ebde83ee148989f9118d36ebaf256532627b8a6e7a486842110623331972edb arch=cray-sles15-zen3
[+]              ^libbsd@0.11.3%gcc@11.2.0 arch=cray-sles15-zen3
[+]                  ^libmd@1.0.3%gcc@11.2.0 arch=cray-sles15-zen3
[+]              ^libsodium@1.0.18%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^util-linux-uuid@2.36.2%gcc@11.2.0 arch=cray-sles15-zen3
[+]      ^hwloc@2.5.0%gcc@11.2.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=cray-sles15-zen3
[+]          ^libpciaccess@0.16%gcc@11.2.0 arch=cray-sles15-zen3
[+]              ^util-macros@1.19.3%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^libxml2@2.9.10%gcc@11.2.0~python arch=cray-sles15-zen3
[+]              ^libiconv@1.16%gcc@11.2.0 libs=shared,static arch=cray-sles15-zen3
[+]              ^xz@5.2.5%gcc@11.2.0~pic libs=shared,static arch=cray-sles15-zen3
[+]      ^jansson@2.13.1%gcc@11.2.0~ipo+shared build_type=RelWithDebInfo arch=cray-sles15-zen3
[+]          ^cmake@3.21.1%gcc@11.2.0~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=cray-sles15-zen3
[+]              ^openssl@1.1.1k%gcc@11.2.0~docs+systemcerts arch=cray-sles15-zen3
[+]      ^lua@5.3.5%gcc@11.2.0~pcfile+shared arch=cray-sles15-zen3
[+]          ^unzip@6.0%gcc@11.2.0 arch=cray-sles15-zen3
[+]      ^lua-luaposix@35.0%gcc@11.2.0 arch=cray-sles15-zen3
[+]      ^lz4@1.9.3%gcc@11.2.0 libs=shared,static arch=cray-sles15-zen3
[+]      ^py-cffi@1.14.3%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^libffi@3.3%gcc@11.2.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=cray-sles15-zen3
[+]          ^py-pycparser@2.20%gcc@11.2.0 arch=cray-sles15-zen3
[+]              ^py-setuptools@57.4.0%gcc@11.2.0 arch=cray-sles15-zen3
[+]                  ^python@3.8.8%gcc@11.2.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,ebdca648c9c1d25f586d7e2a495b62e6d91973b55264a13d89eda1beff72ef56 arch=cray-sles15-zen3
[+]      ^py-jsonschema@3.2.0%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^py-attrs@20.3.0%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^py-pyrsistent@0.15.7%gcc@11.2.0 arch=cray-sles15-zen3
[+]              ^py-six@1.15.0%gcc@11.2.0 arch=cray-sles15-zen3
[+]          ^py-setuptools-scm@6.0.1%gcc@11.2.0+toml arch=cray-sles15-zen3
[+]              ^py-toml@0.10.2%gcc@11.2.0 arch=cray-sles15-zen3
[+]      ^py-pyyaml@5.3.1%gcc@11.2.0+libyaml arch=cray-sles15-zen3
[+]          ^libyaml@0.2.5%gcc@11.2.0 arch=cray-sles15-zen3
[+]      ^yaml-cpp@0.7.0%gcc@11.2.0~ipo+pic+shared~tests build_type=RelWithDebInfo arch=cray-sles15-zen3

The error i am getting is the following

==> Installing flux-core-0.28.0-ct3y434mfjw7f7vfxnxwmgbv4kcksptf
==> No binary for flux-core-0.28.0-ct3y434mfjw7f7vfxnxwmgbv4kcksptf found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/9a/9a784def7186b0036091bd8d6d8fe5bc3425ab2927e1465e1c9ad266631c285d.tar.gz
==> No patches needed for flux-core
==> flux-core: Executing phase: 'autoreconf'
==> flux-core: Executing phase: 'configure'
==> flux-core: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16' 'V=1'

2 errors found in build log:
     1748    libtool: link: /global/common/software/spackecp/perlmutter/e4s-21.08/spack/lib/spack/env/gcc/gcc -Wall -Werror -Wno-strict-aliasing -Wno-error=d
             eprecated-declarations -Werror=missing-field-initializers -g -O2 -o .libs/bulk-exec test/bulk-exec.o  ./.libs/libbulk-exec.a ../../../src/common
             /.libs/libflux-internal.a -L/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder
             __/__spack_/cray-sles15-zen3/gcc-11.2.0/jansson-2.13.1-ft5r3s7dyklzkkubzhq6fvbxp37wdkvi/lib -ljansson -L/tmp/spack/__spack_path_placeholder__/__
             spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/czmq-4.1.1-w5zj2yl76vjsr2bll
             63btpiy7aqnzi67/lib -L/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__s
             pack_/cray-sles15-zen3/gcc-11.2.0/libzmq-4.3.3-sbpscdx5msmvwp3sjdifj7s4mqnjwmw3/lib -L/tmp/spack/__spack_path_placeholder__/__spack_path_placeho
             lder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/util-linux-uuid-2.36.2-54hdrvgbqarvnutpudjwmsj
             or2xqyv33/lib -luuid -lpthread -ldl -L/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_p
             laceholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/libsodium-1.0.18-6bwtrcknhtjvpf3qqi6vtrh4q4u5ipj4/lib -lsodium ../../../src/common/.libs/libfl
             ux-core.so ../../../src/common/.libs/libflux-idset.so ../../../src/common/.libs/libflux-optparse.so ../../../src/common/libutil/.libs/libutil.a 
             -lczmq -lzmq -lm -Wl,-rpath -Wl,/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeho
             lder__/__spack_/cray-sles15-zen3/gcc-11.2.0/flux-core-0.28.0-ct3y434mfjw7f7vfxnxwmgbv4kcksptf/lib
     1749    libtool: link: /usr/bin/nm -B  .libs/job-exec.o .libs/rset.o .libs/testexec.o .libs/exec.o   ../../../src/common/libjob/.libs/libjob.a ../../../
             src/common/.libs/libflux-internal.a ./.libs/libbulk-exec.a | sed -n -e 's/^.*[  ]\([ABCDGIRSTW][ABCDGIRSTW]*\)[     ][  ]*\([_A-Za-z][_A-Za-z0-9]*\
             )$/\1 \2 \2/p' | sed '/ __gnu_lto/d' | /usr/bin/sed 's/.* //' | sort | uniq > .libs/job-exec.exp
     1750    libtool: link: /usr/bin/grep -E -e "^mod_(main|name|service)$" ".libs/job-exec.exp" > ".libs/job-exec.expT"
     1751    libtool: link: mv -f ".libs/job-exec.expT" ".libs/job-exec.exp"
     1752    libtool: link: echo "{ global:" > .libs/job-exec.ver
     1753    libtool: link:  cat .libs/job-exec.exp | sed -e "s/\(.*\)/\1;/" >> .libs/job-exec.ver
  >> 1754    /usr/bin/ld: /tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cra
             y-sles15-zen3/gcc-11.2.0/libzmq-4.3.3-sbpscdx5msmvwp3sjdifj7s4mqnjwmw3/lib/libzmq.so: undefined reference to `std::__throw_bad_array_new_length(
             )@@GLIBCXX_3.4.29'
  >> 1755    collect2: error: ld returned 1 exit status
     1756    make[3]: *** [Makefile:880: bulk-exec] Error 1
     1757    make[3]: *** Waiting for unfinished jobs....
     1758    libtool: link:  echo "local: *; };" >> .libs/job-exec.ver
     1759    libtool: link:  /global/common/software/spackecp/perlmutter/e4s-21.08/spack/lib/spack/env/gcc/gcc -shared  -fPIC -DPIC  .libs/job-exec.o .libs/r
             set.o .libs/testexec.o .libs/exec.o  -Wl,--whole-archive ../../../src/common/libjob/.libs/libjob.a ../../../src/common/.libs/libflux-internal.a 
             ./.libs/libbulk-exec.a -Wl,--no-whole-archive  -Wl,-rpath -Wl,/tmp/spack-stage/siddiq90/spack-stage-flux-core-0.28.0-ct3y434mfjw7f7vfxnxwmgbv4kc
             ksptf/spack-src/spack-build/src/common/.libs -Wl,-rpath -Wl,/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh
             older__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/flux-core-0.28.0-ct3y434mfjw7f7vfxnxwmgbv4kcksptf/lib -L/tmp/spack/__spa
             ck_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/jans
             son-2.13.1-ft5r3s7dyklzkkubzhq6fvbxp37wdkvi/lib -ljansson -L/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeh
             older__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/czmq-4.1.1-w5zj2yl76vjsr2bll63btpiy7aqnzi67/lib -L/tmp/spack/__spack_pat
             h_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/libzmq-4.3
             .3-sbpscdx5msmvwp3sjdifj7s4mqnjwmw3/lib -L/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_pa
             th_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/util-linux-uuid-2.36.2-54hdrvgbqarvnutpudjwmsjor2xqyv33/lib -luuid -lpthread -ldl -L/tmp/s
             pack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-1
             1.2.0/libsodium-1.0.18-6bwtrcknhtjvpf3qqi6vtrh4q4u5ipj4/lib -lsodium ../../../src/common/.libs/libflux-core.so -lczmq -lzmq -lm  -g -O2 -Wl,--no
             -undefined   -Wl,-soname -Wl,job-exec.so -Wl,-version-script -Wl,.libs/job-exec.ver -o .libs/job-exec.so
     1760    libtool: link: ( cd ".libs" && rm -f "job-exec.la" && ln -s "../job-exec.la" "job-exec.la" )
     1761    make[3]: Leaving directory '/tmp/spack-stage/siddiq90/spack-stage-flux-core-0.28.0-ct3y434mfjw7f7vfxnxwmgbv4kcksptf/spack-src/spack-build/src/mo
             dules/job-exec'

See build log for details:
  /tmp/spack-stage/siddiq90/spack-stage-flux-core-0.28.0-ct3y434mfjw7f7vfxnxwmgbv4kcksptf/spack-build-out.txt

Here is a copy of my spack.yaml which is in development you can remove all the specs except for flux-core and try building this on Perlmutter. I could try a different compiler but haven't got to it yet.

spack:
  view: false
  config:
    install_tree:
      root: /tmp/spack
      padded_length: 127
    build_stage: $tempdir/spack-stage
    misc_cache: $tempdir/misc_cache
  mirrors:
    perlmutter-e4s-21.08: /global/common/software/spackecp/mirrors/perlmutter-e4s-21.08
  definitions:
  - cuda_specs:  
    #- amrex@21.10 +openmp +pic +shared +cuda cuda_arch=80
    #- caliper@2.6.0 +cuda cuda_arch=80
    #- chai@2.3.0 +cuda ~benchmarks ~tests cuda_arch=80 ^umpire@4.1.2 ~shared concretization issue
    #- ginkgo@1.3.0 +cuda cuda_arch=80
    # - hpx@1.7.1 +cuda cuda_arch=80
    #- hypre@2.22.0 +cuda cuda_arch=80
    #- magma@2.6.1 +cuda cuda_arch=80
    #- mfem@4.3.0 +cuda cuda_arch=80
    #    - parsec@3.0.2012 +cuda cuda_arch=80
    # - raja@0.13.0 +cuda cuda_arch=80 see issue https://github.com/spack/spack/issues/26995
    #- slate +cuda cuda_arch=80
    #- strumpack +cuda ~slate cuda_arch=80
    # - sundials +cuda cuda_arch=80
    #- superlu-dist@6.4.0 +cuda cuda_arch=80
    #- umpire@4.1.2 +cuda ~shared cuda_arch=80
    - zfp@0.5.5 +cuda cuda_arch=80
  - gcc_cuda_specs:
          #- magma@2.6.1 +cuda cuda_arch=80
    - raja@0.13.0 +cuda cuda_arch=80 
      #- slate +cuda cuda_arch=80
    - sundials +cuda cuda_arch=80
    - superlu-dist@6.4.0 +cuda cuda_arch=80
  - e4s_specs:
    - adios2@2.7.1 ~dataman
    - aml@0.1.0
    #- amrex@21.08 fails to find cray-libsci header files
    # - arborx@1.0 failed install
    - argobots@1.1
    #- ascent@0.7.1 build failed
    #- bolt@2.0
    - darshan-runtime@3.3.1 scheduler=slurm ~mpi
    - darshan-util@3.3.1
    #- dyninst@11.0.1 problem installing elfutils
    #- flecsi@1.4.2 issue building parmetis
    - flit@2.1.0
    #- flux-core@0.28.0  /usr/bin/ld: /tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-11.2.0/libzmq-4.3.3-sbpscdx5msmvwp3sjdifj7s4mqnjwmw3/lib/libzmq.so: undefined reference to `std::__throw_bad_array_new_length()@@GLIBCXX_3.4.29'
    #- fortrilinos@2.0.0 skip for now
    - gasnet@2021.3.0
    # - geopm@1.1.0 skip for now
    - ginkgo@1.3.0
    - hdf5@1.12.0 ~mpi 
    #- heffte@2.1.0 +fftw build failure on fftw
    #- hpctoolkit@2021.05.15 depends on dyninst which failed
    #- hypre@2.22.0
    #- kokkos@3.4.00 +openmp #see https://github.com/spack/spack/issues/26950
    # - kokkos-kernels@3.2.00 +openmp
    - legion@21.03.0
    - metall@0.15
    #- mfem@4.3.0 install failed
    #- netlib-scalapack@2.1.0 build failure
    # - omega-h@9.32.5 install failed
    - openmpi@4.1.1
    - papi@6.0.0.1 
    #- papyrus@1.0.1 could not find MPI libraries
    # - parallel-netcdf@1.12.2 Skip package 
    #- paraview@5.9.1 install failed
      #- parsec@3.0.2012 ~cuda failed install
    #- petsc@3.15.3 ~hypre ~hdf5 build fail
    - qthreads@1.16 scheduler=distrib
    - raja@0.13.0
    #- rempi@1.1.0 build fail
    #- scr@3.0rc1
    - slate@2021.05.02 ~cuda ~mpi   
    #- strumpack@5.1.1 ~slate build failure
    #- sundials@5.7.0 build failure
    - superlu@5.2.2
    #- superlu-dist@6.4.0 +openmp build failure
    - swig@4.0.2
    - swig@4.0.2-fortran
    - sz@2.1.12 +fortran
     #- trilinos@13.0.1 +amesos +amesos2 +anasazi +aztec +belos +boost +epetra +epetraext +ifpack +ifpack2 +intrepid +intrepid2 +isorropia +kokkos +ml +minitensor +muelu +nox +piro +phalanx +rol +rythmos +sacado +stk +shards +shylu +stokhos +stratimikos +teko +tempus +tpetra +trilinoscouplings +zoltan +zoltan2 +superlu-dist gotype=long_long
    - umap@2.1.0
    - umpire@4.1.2 +openmp
    #- unifyfs@0.9.1 failed to find MPI 
    #- upcxx@2021.3.0 build failed
    #- veloc@1.4 build failed
    - vtk-m@1.6.0
    - zfp@0.5.5
  - aux_specs:
    #- axom@0.5.0 ^umpire@4.1.2 failed to install conduit
    #- conduit@0.7.2 ~hdf5 ~mpi build failure
    - faodel@1.1906.1 ~mpi
    #- globalarrays@5.8 failed configure: error: could not link simple Fortran MPI program
    - gotcha@1.0.3
    - hpx@1.7.1
    - libnrm@0.1.0 
    #- libquo@1.3.1  configure: error: *** CC cannot compile MPI apps. set CC to something that can or set CPPFLAGS and LDFLAGS appropriately
    #- loki@0.1.7 ../include/loki/SmallObj.h:462:57: error: ISO C++17 does not allow dynamic exception specifications
    - mercury@2.0.1
    - mpark-variant@1.4.0
    #- mpifileutils@0.11 ~xattr failed to build libcircle
    - ninja@1.10.2
    - openpmd-api@0.13.4 ~hdf5
    - pdt@3.25.1
    #- phist@1.9.4 build failed on MPI_Allreduce
    #- plasma@20.9.20 failed to find blas libraries
    - precice@2.2.1 ~petsc ~mpi
    #- pumi@2.2.6 build failed
    - py-warpx@21.08 ~mpi ^warpx~mpi dims=2 ^fftw~mpi
    - py-warpx@21.08 ~mpi ^warpx~mpi dims=3 ^fftw~mpi
    - py-warpx@21.08 ~mpi ^warpx~mpi dims=rz ^fftw~mpi
    #- slepc@3.15.1 ~arpack unable to build petsc 
    - stc@0.9.0
    - turbine@1.3.0
    - warpx~mpi dims=rz ^fftw~mpi
    - warpx~mpi dims=2 ^fftw~mpi
    - warpx~mpi dims=3 ^fftw~mpi

  - compilers: ['%nvhpc@21.7', '%gcc@11.2.0']
  specs:
  - matrix:
    - [$cuda_specs]
    - ['%nvhpc']
    - [^cmake%gcc]
  - matrix:
    - [$e4s_specs]
    - ['%gcc@11.2.0']
  - matrix:
    - [$gcc_cuda_specs]
    - ['%gcc@11.2.0']
  #- matrix:
  #  - [$e4s_specs, $cuda_specs]
  #  - [$compilers]
  #  - [^nvhpc, ^cray-libsci]
  #  exclude:
  #    - '%gcc ^nvhpc'
  #    - '%nvhpc ^cray-libsci'  
  - matrix:
    - [$aux_specs]
    - ['%gcc@11.2.0']
  compilers:
  - compiler:
      spec: gcc@10.3.0
      paths:
        cc: /opt/cray/pe/gcc/10.3.0/bin/gcc
        cxx: /opt/cray/pe/gcc/10.3.0/bin/g++
        f77: /opt/cray/pe/gcc/10.3.0/bin/gfortran
        fc: /opt/cray/pe/gcc/10.3.0/bin/gfortran
      operating_system: sles15
      modules:
      - PrgEnv-gnu/8.1.0
      - gcc/10.3.0
  - compiler:
      spec: gcc@11.2.0
      paths:
        cc: /opt/cray/pe/gcc/11.2.0/bin/gcc
        cxx: /opt/cray/pe/gcc/11.2.0/bin/g++
        f77: /opt/cray/pe/gcc/11.2.0/bin/gfortran
        fc: /opt/cray/pe/gcc/11.2.0/bin/gfortran
      operating_system: sles15
      modules:
      - PrgEnv-gnu/8.1.0
      - gcc/11.2.0
  - compiler:
      spec: nvhpc@21.7
      paths:
        cc: /opt/nvidia/hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc
        cxx: /opt/nvidia/hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvc++
        f77: /opt/nvidia/hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
        fc: /opt/nvidia/hpc_sdk/Linux_x86_64/21.7/compilers/bin/nvfortran
      modules:
      - nvhpc-nompi/21.7
      flags: {}
      operating_system: sles15
      target: any
      environment: {}
      extra_rpaths: []
  packages:
    all:
      providers:
        compiler: [gcc@11.3.0, nvhpc@21.7]
        mpi: [cray-mpich]
        fftw-api: [cray-fftw]
        blas: [cray-libsci, nvhpc]
        scalapack: [cray-libsci]
        pkgconfig: [pkg-config]
    amrex:
      variants: +openmp +pic +fortran
    automake:
      version: [1.16.3]
    bzip2:
      version: [1.0.6]
      buildable: false
      externals:
      - spec: bzip2@1.0.6
        prefix: /usr
    cray-mpich:
      buildable: false
      externals:
      - spec: cray-mpich@8.1.0
        prefix: /opt/cray/pe/mpich/8.1.9/ofi/GNU/9.1/
        modules:
        - cray-mpich/8.1.9
    cray-libsci:
      buildable: false
      externals:
      - spec: cray-libsci@20.06.1
        prefix: /opt/cray/pe/libsci/21.08.1.2/GNU/9.1/x86_64/
        modules:
        - cray-libsci/21.08.1.2
    cuda:
      buildable: false
      version: [11.4.0]
      externals:
      - spec: cuda@11.4.0%gcc
        modules:
        - PrgEnv-gnu
        - nvhpc-nompi/21.7
    diffutils:
      version: [3.6]
      buildable: false
      externals:
      - spec: diffutils@3.6
        prefix: /usr
    cray-fftw:
      version: [3.3.8.8]
      buildable: false
      externals:
      - spec: cray-fftw@3.3.8.8
        modules:
        - cray-fftw/3.3.8.11
    hdf5:
      variants: +cxx +fortran ~mpi
    gettext:
      version: [0.19.8.1]
      buildable: false
      externals:
      - spec: gettext
        prefix: /usr
    git:
      version: [2.26.2]
      buildable: false
      externals:
      - spec: git@2.26.2
        prefix: /usr
    libfabric:
      buildable: false
      externals:
      - spec: libfabric@1.11.0
        modules:
        - libfabric/1.11.0.4.79
    m4:
      buildable: false
      externals:
      - spec: m4@1.4.18
        prefix: /usr
   #nvhpc:
   #   buildable: false
   #   externals:
   #   - spec: nvhpc@21.7
   #     modules:
   #     - nvhpc/21.7
    openssl:
      buildable: false
      externals:
      - spec: openssl@1.1.1k
        prefix: /usr
    openssh:
      version: [7.9p1]
      buildable: false
      externals:
      - spec: openssh
        prefix: /usr
    openmpi:
      version: [4.0.5]
      variants: +cxx schedulers=slurm
    pdsh:
      buildable: false
      externals:
      - spec: pdsh@2.31
        prefix: /usr
    petsc:
      variants: ~hypre ~hdf5 ~metis ~superlu-dist ~mpi
    python:
      buildable: false
      externals:
      - spec: python@3.8.8
        modules:
        - python/3.9-anaconda-2021.05
    slurm:
      buildable: false
      version: [20-11-5-1]
      externals:
      - spec: slurm
        prefix: /usr
    tar:
      version: [1.3]
      buildable: false
      externals:
      - spec: tar@1.30
        prefix: /usr
    unzip:
      version: [6.0]
      buildable: false
      externals:
      - spec: unzip@6.0
        prefix: /usr
    zsh:
      version: [5.6]
      buildable: false
      externals:
      - spec: zsh@5.6
        prefix: /usr
grondo commented 2 years ago

Thanks! Let us know if using flux-core@0.30.0 from spack's develop branch still shows this issue.

Unfortunately, I don't have a place to try out gcc@11.2.0 at the moment to attempt to reproduce.

shahzebsiddiqui commented 2 years ago

Hi @grondo

i cherry-picked this PR https://github.com/spack/spack/pull/26840 and tried installing flux-core@0.30.0 and i am getting the same error. If you have access to Perlmutter then you can see our gcc/11.2.0 module. I can try a different compiler version and see if it works or wait till next time. It would be helpful if one of you can build the package on perlmutter before there is a next release of e4s.

dongahn commented 2 years ago

Tagging @jameshcorbett as FYI.

shahzebsiddiqui commented 2 years ago

good news i got flux-core installed via gcc@10.3.0 so issue still stand with gcc@11.x 

-- cray-sles15-zen3 / gcc@10.3.0 --------------------------------
flux-core@0.30.0~cuda~docs patches=0329f4b38b5ce4e1a3dee212cb93ce40388d1bde47c82a7a8e73bc7c38274c81
dongahn commented 2 years ago

Great finding. Thanks @shahzebsiddiqui!

shahzebsiddiqui commented 2 years ago

question do we need to install flux-sched or can flux-core be installed by itself. I am trying to figure out if we need both tools installed on Perlmutter.

I take it flux-sched is the scheduler

dongahn commented 2 years ago

@shahzebsiddiqui:

question do we need to install flux-sched or can flux-core be installed by itself. I am trying to figure out if we need both tools installed on Perlmutter.

flux-core can stand alone but you will need advanced scheduler for Permutter. flux-sched has fluxion graph based scheduler: https://github.com/flux-framework/flux-sched. It is the one that has been used for all hero runs including MuMMI cancer research workflow at ORNL across all 4000 nodes. Please see our SC21 paper that will be presented Tue at 10:30AM.

Other resources:

jameshcorbett commented 2 years ago

If you want to run some basic tests with just flux-core, you could try running something like flux start flux mini run hostname or more generally flux start flux mini run [COMMAND].

To get Flux working with Cray Shasta MPI we will need flux-coral2 as well, but my guess is that I would handle that right @dongahn ?

shahzebsiddiqui commented 2 years ago

i think it worked but got some warnings which would be related to lfconfig

siddiq90@login37> flux start flux mini run hostname
sh: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
sh: ldconfig: command not found
2021-10-29T20:24:12.341575Z broker.err[0]: rc1.0: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
2021-10-29T20:24:12.350073Z broker.err[0]: rc1.0: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
2021-10-29T20:24:12.746518Z broker.err[0]: rc1.0: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
login37
2021-10-29T20:24:13.579384Z broker.err[0]: cleanup.0: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
2021-10-29T20:24:13.605998Z broker.err[0]: cleanup.1: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
2021-10-29T20:24:13.630464Z broker.err[0]: cleanup.2: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
2021-10-29T20:24:13.667486Z broker.err[0]: rc3.0: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
2021-10-29T20:24:13.675539Z broker.err[0]: rc3.0: /bin/bash: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)
siddiq90@login37> hostname -f
login37
shahzebsiddiqui commented 2 years ago

On a unrelated note i tried installed flux-sched via spack install flux-sched%gcc@10.3.0 since flux-core worked but it failed on finding readline library which i see is available

siddiq90@login37> ls $(spack location -i readline@8.1%gcc@10.3.0)/include/readline
chardefs.h  history.h  keymaps.h  readline.h  rlconf.h  rlstdc.h  rltypedefs.h  tilde.h

This is the error message

==> Installing flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr
==> No binary for flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/8d/8dffa8eaec95a81286f621639ef851c52dc4c562d365971233bbd91100c31ed2.tar.gz
==> Ran patch() for flux-sched
==> flux-sched: Executing phase: 'autoreconf'
==> flux-sched: Executing phase: 'configure'
==> flux-sched: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j16' 'V=1'

1 error found in build log:
     794    make[3]: Entering directory '/tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/spack-build/resou
            rce/utilities'
     795    Making all in .
     796    make[4]: Entering directory '/tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/spack-build/resou
            rce/utilities'
     797    /global/common/software/spackecp/perlmutter/e4s-21.08/spack/lib/spack/env/gcc/g++ -DHAVE_CONFIG_H -I. -I/tmp/spack-stage/siddiq90/spack-stage-flu
            x-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/resource/utilities -I../..  -I/tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-i
            om7hquijxbk7x6poe5kw3d367shunkr/spack-src -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
            ath_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/czmq-4.1.1-i3wpopweibdbirllr56hc3d2ssst2wax/include -I/tmp/spack/__spack_path_placeholder_
            _/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/libzmq-4.3.3-icn2nnt4as6s
            jha7pwt7qrwlrmdj5hkg/include -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placehold
            er__/__spack_/cray-sles15-zen3/gcc-10.3.0/libsodium-1.0.18-edreexoi62k7657aptvd7rtqv3hchjtp/include -I/tmp/spack/__spack_path_placeholder__/__spa
            ck_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/libbsd-0.11.3-ui4milorjaxjxrgjx5
            u4wbrbclxjoeeo/include -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/_
            _spack_/cray-sles15-zen3/gcc-10.3.0/flux-core-0.30.0-5t4tzekzzqzgcpduauchrcq3r2dca27k/include -isystem /tmp/spack/__spack_path_placeholder__/__sp
            ack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/boost-1.76.0-k44pi6a66n3h4yz5ye
            gi5yqqxydffsrg/include  -Wall -Werror -Werror=missing-field-initializers -Wno-error=deprecated-declarations -Wno-unused-local-typedefs -Wno-depre
            cated-declarations -Wno-unused-variable -Wno-error  -Wno-uninitialized -MT grug2dot-grug2dot.o -MD -MP -MF .deps/grug2dot-grug2dot.Tpo -c -o grug
            2dot-grug2dot.o `test -f 'grug2dot.cpp' || echo '/tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-s
            rc/resource/utilities/'`grug2dot.cpp
     798    /global/common/software/spackecp/perlmutter/e4s-21.08/spack/lib/spack/env/gcc/g++ -DHAVE_CONFIG_H -I. -I/tmp/spack-stage/siddiq90/spack-stage-flu
            x-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/resource/utilities -I../..  -I/tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-i
            om7hquijxbk7x6poe5kw3d367shunkr/spack-src -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
            ath_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/czmq-4.1.1-i3wpopweibdbirllr56hc3d2ssst2wax/include -I/tmp/spack/__spack_path_placeholder_
            _/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/libzmq-4.3.3-icn2nnt4as6s
            jha7pwt7qrwlrmdj5hkg/include -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placehold
            er__/__spack_/cray-sles15-zen3/gcc-10.3.0/libsodium-1.0.18-edreexoi62k7657aptvd7rtqv3hchjtp/include -I/tmp/spack/__spack_path_placeholder__/__spa
            ck_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/libbsd-0.11.3-ui4milorjaxjxrgjx5
            u4wbrbclxjoeeo/include -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/_
            _spack_/cray-sles15-zen3/gcc-10.3.0/flux-core-0.30.0-5t4tzekzzqzgcpduauchrcq3r2dca27k/include -isystem /tmp/spack/__spack_path_placeholder__/__sp
            ack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/boost-1.76.0-k44pi6a66n3h4yz5ye
            gi5yqqxydffsrg/include  -Wall -Werror -Werror=missing-field-initializers -Wno-error=deprecated-declarations -Wno-unused-local-typedefs -Wno-depre
            cated-declarations -Wno-unused-variable -Wno-error  -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__
            /__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/flux-core-0.30.0-5t4tzekzzqzgcpduauchrcq3r2dca27k/include -Wno-uninitialized -MT
             resource_query-resource-query.o -MD -MP -MF .deps/resource_query-resource-query.Tpo -c -o resource_query-resource-query.o `test -f 'resource-que
            ry.cpp' || echo '/tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/resource/utilities/'`resource
            -query.cpp
     799    /global/common/software/spackecp/perlmutter/e4s-21.08/spack/lib/spack/env/gcc/g++ -DHAVE_CONFIG_H -I. -I/tmp/spack-stage/siddiq90/spack-stage-flu
            x-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/resource/utilities -I../..  -I/tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-i
            om7hquijxbk7x6poe5kw3d367shunkr/spack-src -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_p
            ath_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/czmq-4.1.1-i3wpopweibdbirllr56hc3d2ssst2wax/include -I/tmp/spack/__spack_path_placeholder_
            _/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/libzmq-4.3.3-icn2nnt4as6s
            jha7pwt7qrwlrmdj5hkg/include -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placehold
            er__/__spack_/cray-sles15-zen3/gcc-10.3.0/libsodium-1.0.18-edreexoi62k7657aptvd7rtqv3hchjtp/include -I/tmp/spack/__spack_path_placeholder__/__spa
            ck_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/libbsd-0.11.3-ui4milorjaxjxrgjx5
            u4wbrbclxjoeeo/include -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/_
            _spack_/cray-sles15-zen3/gcc-10.3.0/flux-core-0.30.0-5t4tzekzzqzgcpduauchrcq3r2dca27k/include -isystem /tmp/spack/__spack_path_placeholder__/__sp
            ack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/boost-1.76.0-k44pi6a66n3h4yz5ye
            gi5yqqxydffsrg/include  -Wall -Werror -Werror=missing-field-initializers -Wno-error=deprecated-declarations -Wno-unused-local-typedefs -Wno-depre
            cated-declarations -Wno-unused-variable -Wno-error  -I/tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__
            /__spack_path_placeholder__/__spack_/cray-sles15-zen3/gcc-10.3.0/flux-core-0.30.0-5t4tzekzzqzgcpduauchrcq3r2dca27k/include -Wno-uninitialized -MT
             resource_query-command.o -MD -MP -MF .deps/resource_query-command.Tpo -c -o resource_query-command.o `test -f 'command.cpp' || echo '/tmp/spack-
            stage/siddiq90/spack-stage-flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/resource/utilities/'`command.cpp
  >> 800    /tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-src/resource/utilities/resource-query.cpp:20:10: f
            atal error: editline/readline.h: No such file or directory
     801       20 | #include <editline/readline.h>
     802          |          ^~~~~~~~~~~~~~~~~~~~~
     803    compilation terminated.
     804    make[4]: *** [Makefile:592: resource_query-resource-query.o] Error 1
     805    make[4]: *** Waiting for unfinished jobs....
     806    In file included from /tmp/spack/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__spack_path_placeholder__/__sp
            ack_/cray-sles15-zen3/gcc-10.3.0/boost-1.76.0-k44pi6a66n3h4yz5yegi5yqqxydffsrg/include/boost/smart_ptr/detail/sp_thread_sleep.hpp:22,

See build log for details:
  /tmp/spack-stage/siddiq90/spack-stage-flux-sched-0.19.0-iom7hquijxbk7x6poe5kw3d367shunkr/spack-build-out.txt

I will remove our flux install from e4s/21.08 until further notice and when its ready for use.

grondo commented 2 years ago

The spack package.py for flux-sched@0.19 requires libedit not libreadline, I wonder how e4s/21.08 is picking up a dependency on libreadline?

shahzebsiddiqui commented 2 years ago

this is the concretized spec for flux-sched i dont see libedit in its dag

nput spec
--------------------------------
flux-sched%gcc@10.3.0

Concretized
--------------------------------
flux-sched@0.17.0%gcc@10.3.0~cuda arch=cray-sles15-zen3
    ^boost@1.76.0%gcc@10.3.0+atomic+chrono~clanglibcpp~container~context~coroutine+date_time~debug+exception~fiber+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave cxxstd=98 patches=57a8401dee8f52b0342e0c8147a5b2db834e8d8f3fbcbbc5950016bd3e9e1ef0 visibility=hidden arch=cray-sles15-zen3
        ^bzip2@1.0.6%gcc@10.3.0~debug~pic+shared arch=cray-sles15-zen3
        ^zlib@1.2.11%gcc@10.3.0+optimize+pic+shared arch=cray-sles15-zen3
    ^flux-core@0.28.0%gcc@10.3.0~cuda~docs arch=cray-sles15-zen3
        ^czmq@4.1.1%gcc@10.3.0 arch=cray-sles15-zen3
            ^autoconf@2.69%gcc@10.3.0 arch=cray-sles15-zen3
                ^m4@1.4.18%gcc@10.3.0+sigsegv patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 arch=cray-sles15-zen3
                ^perl@5.34.0%gcc@10.3.0+cpanm+shared+threads arch=cray-sles15-zen3
                    ^berkeley-db@18.1.40%gcc@10.3.0+cxx~docs+stl patches=b231fcc4d5cff05e5c3a4814f6a5af0e9a966428dc2176540d2c05aff41de522 arch=cray-sles15-zen3
                    ^gdbm@1.19%gcc@10.3.0 arch=cray-sles15-zen3
                        ^readline@8.1%gcc@10.3.0 arch=cray-sles15-zen3
                            ^ncurses@6.2%gcc@10.3.0~symlinks+termlib abi=none arch=cray-sles15-zen3
                                ^pkg-config@0.29.2%gcc@10.3.0+internal_glib arch=cray-sles15-zen3
            ^automake@1.16.3%gcc@10.3.0 arch=cray-sles15-zen3
            ^libtool@2.4.6%gcc@10.3.0 arch=cray-sles15-zen3
            ^libzmq@4.3.3%gcc@10.3.0~drafts+libsodium arch=cray-sles15-zen3
                ^libbsd@0.11.3%gcc@10.3.0 arch=cray-sles15-zen3
                    ^libmd@1.0.3%gcc@10.3.0 arch=cray-sles15-zen3
                ^libsodium@1.0.18%gcc@10.3.0 arch=cray-sles15-zen3
            ^util-linux-uuid@2.36.2%gcc@10.3.0 arch=cray-sles15-zen3
        ^hwloc@2.5.0%gcc@10.3.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~opencl+pci~rocm+shared arch=cray-sles15-zen3
            ^libpciaccess@0.16%gcc@10.3.0 arch=cray-sles15-zen3
                ^util-macros@1.19.3%gcc@10.3.0 arch=cray-sles15-zen3
            ^libxml2@2.9.10%gcc@10.3.0~python arch=cray-sles15-zen3
                ^libiconv@1.16%gcc@10.3.0 libs=shared,static arch=cray-sles15-zen3
                ^xz@5.2.5%gcc@10.3.0~pic libs=shared,static arch=cray-sles15-zen3
        ^jansson@2.13.1%gcc@10.3.0~ipo+shared build_type=RelWithDebInfo arch=cray-sles15-zen3
            ^cmake@3.21.1%gcc@10.3.0~doc+ncurses+openssl+ownlibs~qt build_type=Release arch=cray-sles15-zen3
                ^openssl@1.1.1k%gcc@10.3.0~docs+systemcerts arch=cray-sles15-zen3
        ^lua@5.3.5%gcc@10.3.0~pcfile+shared arch=cray-sles15-zen3
            ^unzip@6.0%gcc@10.3.0 arch=cray-sles15-zen3
        ^lua-luaposix@35.0%gcc@10.3.0 arch=cray-sles15-zen3
        ^lz4@1.9.3%gcc@10.3.0 libs=shared,static arch=cray-sles15-zen3
        ^py-cffi@1.14.3%gcc@10.3.0 arch=cray-sles15-zen3
            ^libffi@3.3%gcc@10.3.0 patches=26f26c6f29a7ce9bf370ad3ab2610f99365b4bdd7b82e7c31df41a3370d685c0 arch=cray-sles15-zen3
            ^py-pycparser@2.20%gcc@10.3.0 arch=cray-sles15-zen3
                ^py-setuptools@57.4.0%gcc@10.3.0 arch=cray-sles15-zen3
                    ^python@3.8.8%gcc@10.3.0+bz2+ctypes+dbm~debug+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93189bc278fbc37a50ed7f183bd8aaf249a8e1670a465f0db6bb4f8cf87,ebdca648c9c1d25f586d7e2a495b62e6d91973b55264a13d89eda1beff72ef56 arch=cray-sles15-zen3
        ^py-jsonschema@3.2.0%gcc@10.3.0 arch=cray-sles15-zen3
            ^py-attrs@20.3.0%gcc@10.3.0 arch=cray-sles15-zen3
            ^py-pyrsistent@0.15.7%gcc@10.3.0 arch=cray-sles15-zen3
                ^py-six@1.15.0%gcc@10.3.0 arch=cray-sles15-zen3
            ^py-setuptools-scm@6.0.1%gcc@10.3.0+toml arch=cray-sles15-zen3
                ^py-toml@0.10.2%gcc@10.3.0 arch=cray-sles15-zen3
        ^py-pyyaml@5.3.1%gcc@10.3.0+libyaml arch=cray-sles15-zen3
            ^libyaml@0.2.5%gcc@10.3.0 arch=cray-sles15-zen3
        ^yaml-cpp@0.7.0%gcc@10.3.0~ipo+pic+shared~tests build_type=RelWithDebInfo arch=cray-sles15-zen3
dongahn commented 2 years ago

To get Flux working with Cray Shasta MPI we will need flux-coral2 as well, but my guess is that I would handle that right @dongahn ?

Yes, it would be great for you to produce a "blessed, fully configured" version for PM. @shahzebsiddiqui's effort is complimentary.

dongahn commented 2 years ago

sh: /global/common/software/nersc/shasta2105/python/3.9-anaconda-2021.05/lib/libtinfo.so.6: no version information available (required by /lib64/libreadline.so.7)

@shahzebsiddiqui: I think this should be harmless. This is typically created when conda brings in a library that Flux depends on, which is built with a different compiler. In this case /lib64/libreadline.so.7 depends on libtinfo.so.6 which is satisfied by conda activatation. But it isn't FULLy compatible with this system library.

dongahn commented 2 years ago

It would be helpful if one of you can build the package on perlmutter before there is a next release of e4s.

ECP ExaWorks is working with e4s and we plan to do this as part of that effort.

dongahn commented 2 years ago

The spack package.py for flux-sched@0.19 requires libedit not libreadline, I wonder how e4s/21.08 is picking up a dependency on libreadline?

Could be an older recipe.

grondo commented 2 years ago

Could be an older recipe.

Ah, ok, the libedit dependency was added in this commit on Sep 21:

https://github.com/spack/spack/commit/22cfc19bcbc9d4df84dbd7f8023fbeb7de6ea94f

Before trying to build flux-core/flux-sched, I suggest updating to the latest recipes from spack's develop branch.

SteVwonder commented 2 years ago

Unfortunately, I don't have a place to try out gcc@11.2.0 at the moment to attempt to reproduce.

FWIW, you can use spack to build a compiler that can then be used in subsequent spack commands:

spack install gcc@11.2.0
spack load gcc@11.2.0
spack compiler find
spack install flux-core %gcc@11.2.0
shahzebsiddiqui commented 2 years ago

Unfortunately, I don't have a place to try out gcc@11.2.0 at the moment to attempt to reproduce.

FWIW, you can use spack to build a compiler that can then be used in subsequent spack commands:

spack install gcc@11.2.0
spack load gcc@11.2.0
spack compiler find
spack install flux-core %gcc@11.2.0

Yes i am aware of this and I have this set in my spack.yaml. flux-core was not an issue it was getting flux-sched installed.

Could be an older recipe.

Ah, ok, the libedit dependency was added in this commit on Sep 21:

spack/spack@22cfc19

Before trying to build flux-core/flux-sched, I suggest updating to the latest recipes from spack's develop branch.

Yeah these kind of changes require waiting for next e4s release. We can't constantly cherry-pick or use spack-develop but its good to know that this fix will address the issue I was having with flex-sched.

In the meantime, i would recommend we get you setup with a spack pipeline at NERSC so you can build the package at develop. Take a look at https://docs.nersc.gov/applications/e4s/spack_gitlab_pipeline/ in our user docs. I can create a project space in gitlab for you start building a pipeline.

Take a look at this https://www.youtube.com/watch?v=eSEiVxXpCDo&t=3705s at 1:00:00 to 1:26:00 which talks about the gitlab spack pipeline.

The use-case is mentioned in the docs. We would like to get builds for your products working at develop and latest release. That way you have confidence that your package can be built at the facility. Note that currently we have runnners on Cori and not Perlmutter though that will be present in the near future.