`compilers.yaml` not generated when gcc version is shadowed by system gcc

[bcumming]

When the version of gcc in the gcc compiler toolchain matches the system-provided compiler, the compiler is not found when generating compilers.yaml for any spack environment that requests the gcc compiler toolchain.

This is observed on a system with gcc 11.3 in /usr/bin, when gcc@11.3 toolchain is requested to install nvhpc or an environment in environments.yaml.

It looks like the version found in compilers/gcc environment matches the one in config/compilers.yaml, so it is skipped for some reason. This is either a Spack bug, or misuse of Spack by us.

[sekelle]

I'm getting this issue with compilers.yaml not being generated on clariden, even though the system compiler is still GCC 7.5 and I'm building GCC 11.4. The issue shows up with spack/releases-v0.21.

  Search: 0.00s.  Fetch: 7.26s.  Install: 3m 31.65s.  Extract: 3m 26.77s.  Relocate: 4.83s.  Total: 3m 38.92s
[+] /user-environment/linux-sles15-zen2/gcc-11.4.0/nvhpc-21.7-tqwwxrttfbihyzm55h2sk3fenahph5sv
spack -e ./llvm buildcache create --rebuild-index --allow-root --only=package alpscache \
$(spack --color=never -e ./llvm find --format '{name};{/hash}' | grep -v -E '^(nvhpc);' | cut -d ';' -f2)
==> Warning: The flag `--allow-root` is the default in Spack 0.21, will be removed in Spack 0.22
==> Selected 49 specs to push to file:///iopsstor/scratch/cscs/sebkelle/uenv-cache/user-environment
==> All specs are already in the buildcache. Use --force to overwrite them.
touch llvm/generated/build_cache
make[1]: Leaving directory '/dev/shm/sebkelle/sbuild-bb/compilers'
/dev/shm/sebkelle/sbuild-bb/bwrap-mutable-root.sh --tmpfs ~ --bind /dev/shm/sebkelle/sbuild-bb/tmp /tmp --bind /dev/shm/sebkelle/sbuild-bb/store /user-environment make -C environments
/dev/shm/sebkelle/sbuild-bb/bwrap-mutable-root.sh --tmpfs ~ --bind /dev/shm/sebkelle/sbuild-bb/tmp /tmp --bind /dev/shm/sebkelle/sbuild-bb/store /user-environment make -C generate-config
make[1]: Entering directory '/dev/shm/sebkelle/sbuild-bb/environments'
make[1]: Entering directory '/dev/shm/sebkelle/sbuild-bb/generate-config'
spack config --scope=user add upstreams:system:install_tree:/user-environment
spack compiler find --scope=user $(find  $(spack -e ../compilers/gcc find --format='{prefix}' gcc llvm nvhpc) $(spack -e ../compilers/llvm find --format='{prefix}' gcc llvm nvhpc) '(' -name gcc -o -name clang -o -name nvc ')' -path '*/bin/*' '(' -type f -o -type l ')' -exec dirname {} +)
spack config --scope=user add upstreams:system:install_tree:/user-environment
spack compiler find --scope=user $(find  $(spack -e ../compilers/bootstrap find --format='{prefix}' gcc llvm nvhpc) $(spack -e ../compilers/gcc find --format='{prefix}' gcc llvm nvhpc) $(spack -e ../compilers/llvm find --format='{prefix}' gcc llvm nvhpc) '(' -name gcc -o -name clang -o -name nvc ')' -path '*/bin/*' '(' -type f -o -type l ')' -exec dirname {} +)
mkdir -p /user-environment
touch gcc-nvgpu/packages.yaml
spack config --scope=user add config:install_tree:root:/user-environment
spack compiler find --scope=user $(find  $(spack --color=never -e ../compilers/gcc find --format '{prefix}' gcc@11.3) '(' -name gcc -o -name clang -o -name nvc ')' -path '*/bin/*' '(' -type f -o -type l ')' -exec dirname {} +)
find: ‘==>’: No such file or directory
find: ‘No’: No such file or directory
find: ‘package’: No such file or directory
find: ‘matches’: No such file or directory
find: ‘the’: No such file or directory
find: ‘query:’: No such file or directory
find: ‘gcc@11.3’: No such file or directory
==> Found no new compilers
==> Compilers are defined in the following files:
    /dev/shm/sebkelle/sbuild-bb/config/compilers.yaml
spack -e gcc-nvgpu/ concretize -f
==> Error: Detected 1 missing include path(s):
   /dev/shm/sebkelle/sbuild-bb/environments/gcc-nvgpu/compilers.yaml
Makefile:42: gcc-nvgpu/Makefile: No such file or directory
make[1]: *** [../Make.inc:20: gcc-nvgpu/spack.lock] Error 1
make[1]: Leaving directory '/dev/shm/sebkelle/sbuild-bb/environments'
make: *** [Makefile:50: environments] Error 2
make: *** Waiting for unfinished jobs....
==> Added 3 new compilers to /user-environment/config/compilers.yaml
    nvhpc@21.7  gcc@11.4.0  clang@14.0.6
==> Compilers are defined in the following files:
    /dev/shm/sebkelle/sbuild-bb/config/compilers.yaml  /user-environment/config/compilers.yaml
install -m 644 /dev/shm/sebkelle/sbuild-bb/config/packages.yaml /user-environment/config/packages.yaml
install -m 644 /dev/shm/sebkelle/sbuild-bb/config/repos.yaml /user-environment/config/repos.yaml
==> Added 3 new compilers to /dev/shm/sebkelle/sbuild-bb/modules/compilers.yaml
    nvhpc@21.7  gcc@11.4.0  clang@14.0.6
==> Compilers are defined in the following files:
    /dev/shm/sebkelle/sbuild-bb/config/compilers.yaml  /dev/shm/sebkelle/sbuild-bb/modules/compilers.yaml
make[1]: Leaving directory '/dev/shm/sebkelle/sbuild-bb/generate-config'

Recipe: /iopsstor/scratch/cscs/sebkelle/prgenv-gnu config: ./bin/stack-config -b /dev/shm/sebkelle/sbuild-bb/ -r /iopsstor/scratch/cscs/sebkelle/prgenv-gnu -s ~/soft/alps-cluster-config/clariden -c ~/cache-config.yaml

[bcumming]

@sekelle, these issues are not related. In your case, the environment in environments.yaml requests a specific version of gcc@11.3. Spack v0.21 will build gcc@11.4 (because in compilers.yaml simply requires gcc@11).

The fix is simple: update environment.yaml to request gcc@11.