search

fasrc / User_Codes

73 stars 16 forks source link

Error when building NetCDF Utilities with gcc@12.2.0 in Spack (possible cache not cleared error) #13

Closed yantosca closed 1 year ago

yantosca commented 1 year ago

I was able to build the NCO (netcdf operators) OK using gcc@10.2.0 but not with gcc@12.2.0. I am not sure if this is due to a cached prior setting my Spack environment:

[holy7c12101 RockyLinux]$ spack -C ./spack-env/gnu12 install nco
[+] /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/antlr-2.7.7-vwg4fp5hixtmsrqgbafqwaq4orn7usy4
[+] /usr (external m4-1.4.18-6yrjjvpvae3cu2nv5noeuotvyz2cnkw2)
[+] /n/sw/helmod-rocky8/apps/Core/flex/2.6.4-fasrc01 (external flex-2.6.4-5dga27e2evtxg4grzdy2lijxss6ubs27)
[+] /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/gsl-2.7.1-5zmgtyqtps3owdhvwj4ocjd5yvwnwrbe
[+] /n/sw/helmod-rocky8/apps/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-farc01/netcdf-c/4.9.2-fasrc01 (external netcdf-c-4.9.2-tyyqxxkkdcdynz3szrtx72akxfxkm7f7)
[+] /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/libmd-1.0.4-jxvqhjl7uir2utgpdxzxho744wnbyicj
[+] /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/bison-3.8.2-qharcjky67vjqf65solpjjs2enldp2ly
[+] /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/libbsd-0.11.7-23wv73vhh5xokpsvqbqfucwwetngskdx
[+] /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/expat-2.5.0-qhyjsg6pqbymua74ljvuacveehgtvz4i
[+] /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/udunits-2.2.28-nq7szfdrw5es6cacy6i7xqhr535igadg
==> Installing nco-5.1.4-dsoe76xx3v5jvhsq2zf3zirguoocgpzf
==> No binary for nco-5.1.4-dsoe76xx3v5jvhsq2zf3zirguoocgpzf found: installing from source
==> Using cached archive: /net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/var/spack/cache/_source-cache/archive/4b/4b1ec67b795b985990620be7b7422ecae6da77f5ec93e4407b799f0220dffc88.tar.gz
==> No patches needed for nco
==> nco: Executing phase: 'autoreconf'
==> nco: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    '/tmp/ryantosca/spack-stage/spack-stage-nco-5.1.4-dsoe76xx3v5jvhsq2zf3zirguoocgpzf/spack-src/configure' '--prefix=/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux/spack/opt/spack/linux-rocky8-x86_64/gcc-12.2.0/nco-5.1.4-dsoe76xx3v5jvhsq2zf3zirguoocgpzf' '--disable-doc'

2 errors found in build log:
     154    #################################
     155    #  Test for netCDF             #
     156    ################################
     157    WARNING: NETCDF_LIB location "/n/sw/helmod-rocky8/apps/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-farc01/netcdf-c
            /4.9.2-fasrc01/lib" does not exist!
     158    WARNING: NETCDF_INC location "/n/sw/helmod-rocky8/apps/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-farc01/netcdf-c
            /4.9.2-fasrc01/include" does not exist!
     159    checking for netcdf.h... no
  >> 160    configure: error: in `/tmp/ryantosca/spack-stage/spack-stage-nco-5.1.4-dsoe76xx3v5jvhsq2zf3zirguoocgpzf/spa
            ck-src':
  >> 161    configure: error: cannot find netCDF header
     162    See `config.log' for more details

See build log for details:
  /tmp/ryantosca/spack-stage/spack-stage-nco-5.1.4-dsoe76xx3v5jvhsq2zf3zirguoocgpzf/spack-build-out.txt

Note that the path has a typo: (farc01 instead of fasrc01): 

/n/sw/helmod-rocky8/apps/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-farc01/netcdf-c/4.9.2-fasrc01/lib

I think this was my own typo in my packages.yaml file. But I fixed it and then I regenerated the module files so I think this shouldn't be caching the error. I also did spack clean -a several times which should clean up the build.

I have these modules loaded:

Currently Loaded Modules:
  1) gmp/6.2.1-fasrc01    4) gcc/12.2.0-fasrc01      7) szip/2.1.1-fasrc01      10) netcdf-fortran/4.6.0-fasrc02
  2) mpfr/4.2.0-fasrc01   5) openmpi/4.1.4-fasrc01   8) hdf5/1.14.0-fasrc01     11) flex/2.6.4-fasrc01
  3) mpc/1.3.1-fasrc01    6) zlib/1.2.13-fasrc01     9) netcdf-c/4.9.2-fasrc01  12) cmake/3.25.2-fasrc01

I am building from this folder

/net/holy-nfsisilon/ifs/rc_labs/jacob_lab/Lab/seasfs01/Lab/RockyLinux

with this command

spack -C ./spack-env/gnu12 install nco

and my packages.yaml file is:

#============================================================================
# gnu12/packages.yaml:
#
# Defines package specs for externally-built FASRC modules and for
# Spack modules.  All modules use GNU Compiler Collection 12.2.0.
#
# Settings will prevent Spack from trying to rebuild FASRC modules
# again, which can cause issues with compiling GEOS-Chem and GCHP.
#
#  -- Bob Yantosca (01 May 2023)
#============================================================================

packages:
  #
  # Specs for FASRC-built modules.  These will be assigned "Spack-like"
  # specs (e.g. netcdf-c/4.9.2-fasrc01 => netcdf-c/4.9.2-gcc-12.2.0).
  #
  cmake:
    buildable: False
    externals:
      - spec: "cmake@3.25.2"
        prefix: "/n/sw/helmod-rocky8/apps/Core/cmake/3.25.2-fasrc01"

  gdb:
    buildable: False
    externals:
      - spec: "gdb@8.2.19"
        prefix: "/usr/bin"

  hdf5:
    buildable: False
    externals:
      - spec: "hdf5@1.14.0 %gcc@12.2.0"
        prefix: "/n/sw/helmod-rocky8/apps/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-fasrc01/hdf5/1.14.0-fasrc01"

  netcdf-c:
    buildable: False
    externals:
      - spec: "netcdf-c@4.9.2 ^openmpi@4.1.4"
        prefix: "/n/sw/helmod-rocky8/apps/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-fasrc01/netcdf-c/4.9.2-fasrc01"

  netcdf-fortran:
    buildable: False
    externals:
     - spec: "netcdf-fortran@4.6.0 ^openmpi@4.1.4"
       prefix: "/n/sw/helmod-rocky8/apps/MPI/gcc/12.2.0-fasrc01/openmpi/4.1.4-fasrc01/netcdf-fortran/4.6.0-fasrc02"

  openmpi:
    buildable: False
    externals:
      - spec: "openmpi@4.1.4"
        prefix: "/n/sw/helmod-rocky8/apps/Comp/gcc/12.2.0-fasrc01/openmpi/4.1.4-fasrc01"

  perl:
    buildable: False
    externals:
      - spec: "perl@5.26.3"
        prefix: "/usr/bin"

  szip:
    buildable: False
    externals:
      - spec: "szip@2.1.1"
        prefix: "/n/sw/helmod-rocky8/apps/Core/szip/2.1.1-fasrc01"

  zlib:
    buildable: False
    externals:
      - spec: "zlib@1.2.13"
        prefix: "/n/sw/helmod-rocky8/apps/Core/zlib/1.2.13-fasrc01"
  #
  # Specs for modules that will be built with Spack
  #
  cdo:
    version: ["2.1.0"]
    require: ["netcdf-c@4.9.2", "zlib@1.2.13"]
    variants: "~doc~proj"
    compiler: ["gcc@12.2.0"]

  cgdb:
    version: ["0.7.1"]
    require: "gdb"
    compiler: ["gcc@12.2.0"]

  emacs:
    version: ["28.2"]
    variants: "+gtk+X"
    compiler: ["gcc@12.2.0"]

  nco:
    version: ["5.1.4"]
    require: ["netcdf-c@4.9.2"]
    variants: "~doc"
    compiler: ["gcc@12.2.0"]

  tau:
    version: ["2.32"]
    require: ["openmpi@4.1.0"]
    variants: "~otf2+openmp+papi+mpi"
    compiler: ["gcc@10.2.0"]
  #
  # Settings for all Spack-built packages
  #
  all:
    target: ["x86_64"]
    providers:
      mpi: ["openmpi"]

Thanks for any insights!

yantosca commented 1 year ago

Ah I see it. I had a typo in packages.yaml that caused the netCDF external package to point to the wrong path:

[holy7c12101 RockyLinux]$ spack -C ./spack-env/gnu12 uninstall --dependents netcdf-c /tyyqxxk
==> The following packages will be uninstalled:

    -- linux-rocky8-x86_64 / gcc@12.2.0 -----------------------------
    3uf3yiu cdo@2.1.0  tyyqxxk netcdf-c@4.9.2

==> Do you want to proceed? [y/N] y
==> Successfully uninstalled cdo@2.1.0%gcc@12.2.0~curl~external-grib1+fftw3+hdf5+libxml2~magics+netcdf+openmp~proj~szip+udunits2 build_system=autotools grib2=eccodes arch=linux-rocky8-x86_64/3uf3yiu

This might be a good warning for folks to check their packages.yaml file! Thanks much!