Namespaces existing only in buildcaches cause concretization to fail

[sleak-lbl]

Steps to reproduce

• create an environment, and make a local repo for it with some namespace (eg dev), and in that repo put (and presumably edit/update) a spack package that you plan to build (in my case, I applied the patch in #34580 to a local copy of the ncview package)

• install the environment (wherever you like), and push the built package to your buildcache.

• create a new environment to build something entirely unrelated to the first environment (in my example, elfutils), but allow that environment to use your buildcache (via mirrors.yaml)

• run spack concretize

At this point you will get an error like: ==> Error: Unknown namespace: dev

If you make a local repo in the new environment, with the namespace dev (but nothing in it), then you get past that error and arrive at a new one: ==> Error: Package 'ncview' not found.

It looks like Spack copies the buildcache index to your misc_cache, and then the concretizer baulks when it can't find the repo or package that was used to build some buildcache entry, even if that entry has no relevance to the current environment.

A better action in the event of "unknown thing in cache" would be to ignore the unknown thing.

More info

The tail of the stack trace (with spack -dv concretize looks like:

  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/solver/asp.py", line 627, in solve
    setup.setup(self, specs, reuse=reuse)
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/solver/asp.py", line 1987, in setup
    self._facts_from_concrete_spec(reusable_spec, possible)
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/solver/asp.py", line 1895, in _facts_from_concrete_spec
    self.impose(h, spec, body=True)
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/solver/asp.py", line 1088, in impose
    imposed_constraints = self.spec_clauses(imposed_spec, body=body, required_from=name)
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/solver/asp.py", line 1311, in spec_clauses
    clauses = self._spec_clauses(*args, **kwargs)
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/solver/asp.py", line 1476, in _spec_clauses
    for virtual in dep.package.virtuals_provided:
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/spec.py", line 1639, in package
    self._package = spack.repo.path.get(self)
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/repo.py", line 977, in get
    return self.repo_for_pkg(spec).get(spec)
  File "/global/common/software/nersc/pm-2022q4/sw/spack-0.19.0/lib/spack/spack/repo.py", line 960, in repo_for_pkg
    raise UnknownNamespaceError(namespace, name=name)
spack.repo.UnknownNamespaceError: Unknown namespace: dev

but I think the root cause is that this call in SpackSolverSetup.setup (in spack/spack/solver/asp.py)

        # get list of all possible dependencies
        self.possible_virtuals = set(x.name for x in specs if x.virtual)
        possible = spack.package_base.possible_dependencies(
            *specs, virtuals=self.possible_virtuals, deptype=spack.dependency.all_deptypes
        )

the list of specs to search for possible dependencies includes things that are not relevant. A possible clue is that when I run spack dependencies -t elfutils I get the following output:

$ spack dependencies -t elfutils
alsa-lib                 fontconfig             jemalloc          libx11         nettle                      poppler                           py-py                             sed
amdblis                  fontsproto             json-c            libxau         nghttp2                     poppler-data                      py-pycparser                      shadow
amdlibflame              freetype               json-glib         libxaw         ninja                       popt                              py-pygments                       shared-mime-info
apple-gl                 fribidi                kbproto           libxcb         nspr                        postgresql                        py-pyopenssl                      singularity
apple-libunwind          fujitsu-mpi            knem              libxcomposite  nss                         psm                               py-pyparsing                      slurm
apple-libuuid            fujitsu-ssl2           krb5              libxcursor     numactl                     py-alabaster                      py-pysocks                        source-highlight
apr                      gawk                   lcms              libxdamage     nvhpc                       py-anytree                        py-pytest                         spectrum-mpi
apr-util                 gdb                    lerc              libxdmcp       ocl-icd                     py-asgiref                        py-pytest-runner                  sqlite
argobots                 gdbm                   libaec            libxext        oneapi-level-zero           py-asn1crypto                     py-python-dateutil                squashfs
armpl-gcc                gdk-pixbuf             libaio            libxfixes      opa-psm2                    py-atomicwrites                   py-pytoml                         ssmtp
asciidoc-py3             gdrcopy                libarchive        libxfont       openblas                    py-attrs                          py-pytz                           swig
assimp                   gettext                libatomic-ops     libxft         opencl-c-headers            py-automat                        py-requests                       tar
at-spi2-atk              ghostscript            libbsd            libxi          opencl-clhpp                py-babel                          py-scandir                        tcl
at-spi2-core             giflib                 libc              libxinerama    opencl-headers              py-backports-functools-lru-cache  py-semantic-version               teckit
atk                      git                    libcroco          libxkbcommon   opengl                      py-beautifulsoup4                 py-setuptools                     texinfo
atlas                    glib                   libdrm            libxkbfile     openjdk                     py-bottleneck                     py-setuptools-rust                texlive
autoconf                 glproto                libedit           libxml2        openjpeg                    py-brotlipy                       py-setuptools-scm                 tix
autoconf-archive         glx                    libelf            libxmu         openmpi                     py-certifi                        py-six                            tk
automake                 gmake                  libepoxy          libxpm         openpbs                     py-cffi                           py-snowballstemmer                torque
bash                     gmp                    libevent          libxrandr      openssh                     py-chardet                        py-sortedcontainers               uchardet
bdftopcf                 gnuconfig              libfabric         libxrender     openssl                     py-charset-normalizer             py-soupsieve                      ucx
bdw-gc                   gnutls                 libffi            libxscrnsaver  osmesa                      py-colorama                       py-sphinx                         unixodbc
berkeley-db              go                     libflame          libxslt        ossp-uuid                   py-configparser                   py-sphinx-rtd-theme               unzip
binutils                 go-bootstrap           libfontenc        libxt          pandoc                      py-constantly                     py-sphinxcontrib-applehelp        util-linux
bison                    gobject-introspection  libfuse           libxtst        pango                       py-contextlib2                    py-sphinxcontrib-devhelp          util-linux-uuid
blis                     googletest             libgcrypt         libyaml        patchelf                    py-cryptography                   py-sphinxcontrib-htmlhelp         util-macros
boost                    gperf                  libgd             libzmq         pcre                        py-cssselect                      py-sphinxcontrib-jsmath           valgrind
byacc                    graphite2              libgit2           llvm           pcre2                       py-cython                         py-sphinxcontrib-qthelp           veclibfort
bzip2                    graphviz               libgpg-error      llvm-amdgpu    perl                        py-django                         py-sphinxcontrib-serializinghtml  vim
ca-certificates-mozilla  groff                  libical           lsf            perl-alien-svn              py-docutils                       py-sphinxcontrib-websupport       wayland
cairo                    gtk-doc                libice            lua            perl-data-dumper            py-editables                      py-sqlparse                       wayland-protocols
chrpath                  gtkplus                libiconv          lustre         perl-encode-locale          py-enum34                         py-toml                           wget
clblast                  gts                    libidn2           lvm2           perl-extutils-config        py-flit-core                      py-tomli                          xauth
cmake                    guile                  libjpeg           lz4            perl-extutils-helpers       py-funcsigs                       py-twisted                        xcb-proto
comgr                    gzip                   libjpeg-turbo     lzo            perl-extutils-installpaths  py-hatchling                      py-typing                         xcb-util
compositeproto           harfbuzz               libjwt            m4             perl-file-listing           py-html5lib                       py-typing-extensions              xcb-util-image
coreutils                hcoll                  libmd             mariadb        perl-file-which             py-hyperlink                      py-urllib3                        xcb-util-keysyms
cpio                     hdf5                   libmicrohttpd     maven          perl-html-parser            py-hypothesis                     py-urllib3-secure-extra           xcb-util-renderutil
cray-libsci              help2man               libmng            mawk           perl-html-tagset            py-idna                           py-versioneer                     xcb-util-wm
cray-mpich               hip                    libnl             mbedtls        perl-http-cookies           py-imagesize                      py-wcwidth                        xextproto
cray-mvapich2            hip-rocclr             libnsl            mesa           perl-http-daemon            py-importlib-metadata             py-webencodings                   xineramaproto
cray-pmi                 hpcx-mpi               libpaper          mesa18         perl-http-date              py-incremental                    py-wheel                          xkbcomp
cryptsetup               hsa-rocr-dev           libpciaccess      meson          perl-http-message           py-iniconfig                      py-zipp                           xkbdata
cscope                   hsakmt-roct            libpng            mkfontdir      perl-http-negotiate         py-ipaddress                      py-zope-interface                 xmlto
cuda                     http-parser            libpsl            mkfontscale    perl-io-html                py-jinja2                         python                            xpmem
cups                     hwloc                  libpthread-stubs  mpfr           perl-libwww-perl            py-lxml                           qt                                xproto
curl                     ibm-java               librsvg           mpi-serial     perl-lwp-mediatypes         py-m2r                            randrproto                        xrandr
damageproto              icedtea                libseccomp        mpich          perl-module-build           py-mako                           rdma-core                         xtrans
dbus                     icu4c                  libsigsegv        mpilander      perl-module-build-tiny      py-markupsafe                     readline                          xxd-standalone
diffutils                inputproto             libsm             mpitrampoline  perl-net-http               py-mistune                        recordproto                       xz
docbook-xml              intel-mkl              libsodium         mpt            perl-test-needs             py-more-itertools                 renderproto                       yaksa
docbook-xsl              intel-mpi              libssh            mscgen         perl-try-tiny               py-nose                           rhash                             z3
double-conversion        intel-oneapi-mkl       libssh2           msgpack-c      perl-uri                    py-numexpr                        rocm-cmake                        zip
doxygen                  intel-oneapi-mpi       libszip           munge          perl-uri-encode             py-numpy                          rocm-device-libs                  zlib
essl                     intel-oneapi-tbb       libtiff           mvapich2       perl-www-robotrules         py-ordereddict                    rocm-opencl                       zstd
expat                    intel-parallel-studio  libtirpc          mvapich2-gdr   perl-xml-parser             py-packaging                      rocm-smi-lib                      zziplib
fca                      intltool               libtool           mvapich2x      pigz                        py-pandas                         rocminfo
findutils                isl                    libunistring      mxm            pixman                      py-parameterized                  roctracer-dev-api
fixesproto               itstool                libunwind         nasm           pkg-config                  py-pathlib2                       rpcsvc-proto
flex                     jansson                libuuid           ncurses        pkgconf                     py-pathspec                       ruby
flexiblas                jbigkit                libuv             netlib-lapack  pmix                        py-pip                            rust
font-util                jdk                    libwebp           netlib-xblas   pocl                        py-pluggy                         scrnsaverproto

I'm quite confident that elfutils does not depend on, eg, cray-mvapich2, even indirectly, I'm guessing that something in my misc_cache or buildcache has a possible dependency on cray-mvapich2, and that causes cray-mvapich2 to be reported asa dependies for elfutils.

(running spack clean -m does not change this behavior, as spack dependencies repopulates the misc_cache)

Error message

No response

Information on your system

$ spack debug report
* **Spack:** 0.19.0 (bb8b4f9979e87ae3fb63a3adcb768eda09e5b059)
* **Python:** 3.6.15
* **Platform:** linux-sles15-zen3
* **Concretizer:** clingo

General information

• [X] I have run spack debug report and reported the version of Spack/Python/Platform
• [X] I have searched the issues of this repo and believe this is not a duplicate
• [X] I have run the failing commands in debug mode and reported the output

[sleak-lbl]

update: the spack dependencies -t elfutils weirdness might be a red herring .. when I remove the buildcache from mirrors.yaml (and run spack clean -m), the environment concretizes, but spack dependencies -t elfutils still has the long list of nonsensical dependencies

[quellyn]

I may be seeing something similar to this when I try to build a Spack bootstrap cache:

quellyn@ro-rfe1:/usr/projects/shavano/spack-environments-dev/ATS3I/2.3.0> spack bootstrap mirror --binary-packages ../../spack.bootstrap
==> Adding "clingo-bootstrap@spack+python %gcc target=x86_64" and dependencies to the mirror at /usr/projects/shavano/spack-environments-dev/spack.bootstrap/bootstrap_cache
==> Error: Unknown namespace: repo_lapdev

My current workaround is to clone a fresh copy of Spack and checkout the same hash that I'm using in the other environment. Then from a fresh window on the same system, I source the setup script from my fresh Spack clone and run:

$ spack bootstrap mirror --binary-packages /usr/projects/shavano/spack-environments-dev/spack.bootstrap

That works without issue.