search

Alpine-DAV / ascent

A flyweight in situ visualization and analysis runtime for multi-physics HPC simulations
https://alpine-dav.github.io/ascent/
Other
189 stars 65 forks source link

2022-06 spack update #949

Closed cyrush closed 1 year ago

cyrush commented 2 years ago

Using this issue to track all steps for to update to latest Spack for our CI builds and deployment on ECP platforms.

Changes needed to existing dependent packages on spack develop:

Missing Public Packages:

Ascent Public (spack develop) Package Changes:

Ascent hot-copy (uberenv) Package Changes:

CI Container Config and Build Updates

ECP Public Install Config and Build Updates:

cyrush commented 2 years ago

Error: Spec version is not concrete: netlib-lapack

cyrush commented 2 years ago

our genten needs updates due to kokkos API changes - wait for new higher order moments branch to be merged into genten master.

cyrush commented 2 years ago

I am seeing repeated fetch errors for sphinx source from spack mirrors. 

#10 1341.6 ==> No binary for py-sphinx-4.4.0-v5mdrelwgt7tk2y5u5mbssukgodwx4tk found: installing from source
#10 1343.2 ==> Fetching https://mirror.spack.io/_source-cache/archive/6c/6caad9786055cb1fa22b4a365c1775816b876f91966481765d7d50e9f0dd35cc.tar.gz
#10 1343.3 ==> Error: UnicodeEncodeError: 'ascii' codec can't encode character '\xe4' in position 45: ordinal not in range(128)
cyrush commented 2 years ago

I also hit consistent errors with spack fetching python (both 3.8 and 3.9)

I disabled python to get a minimal container - but we need python to generate the unit test reports we rely on in CI. So CI tests fail with this container.

cyrush commented 2 years ago

Python 3.9 built with newer spack, however hit an activation error:

==> Error: Package merge blocked by file: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/python-3.9.13-m3tvtc2qdvgffcitl3isjr2u2jnww56i/lib/python3.9/site-packages/_distutils_hack/__pycache__/__init__.cpython-39.pyc

Same error in Python 3.8.

More details:

==> Activating extension py-numpy@1.22.4%gcc@7.5.0+blas+lapack patches=873745d arch=linux-ubuntu18.04-zen/sbojlbm for python@3.8.13%gcc@7.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu18.04-zen/trauecu
==> Activating extension py-setuptools@62.3.2%gcc@7.5.0 arch=linux-ubuntu18.04-zen/sjai4bd for python@3.8.13%gcc@7.5.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,4c24573,f2fd060 arch=linux-ubuntu18.04-zen/trauecu
==> Error: Package merge blocked by file: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/python-3.8.13-trauecunid25vkppz25nstapcbssl7am/lib/python3.8/site-packages/_distutils_hack/__pycache__/__init__.cpython-38.pyc

Both the main python install:

ls  /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/python-3.8.13-trauecunid25vkppz25nstapcbssl7am/lib/python3.8/site-
packages/_distutils_hack/

And the setup tools install:

ls /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/py-setuptools-62.3.2-sjai4bdyrc2vuthp2vuhyietb65imgv4/lib/python3.8/site-packages/
_distutils_hack  distutils-precedence.pth  pkg_resources  setuptools  setuptools-62.3.2.dist-info

contain this _distutils_hack/ files

spack activate fails b/c when spack activates setuptools, it tries to link it into the main python dir, it will notice a conflict and bail.

I tried two older versions of pysetup tools 59.4.0 49.6.0 and the folder was still there, but it could still be entangled with the python some how (I didn't do a clean python install for each try)

I am not sure if this is an issue with spack environment views.

@white238 saw this spack issue in another project.

I created a spack issue with a simple recipe to reproduce: https://github.com/spack/spack/issues/31166

cyrush commented 2 years ago

Hitting an issue bootstrapping clingo on Ubunutu 22.04 with gcc. 

10 244.9 ==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64" due to the follow
ing failures:
#10 244.9 github-actions-v0.2 raised AttributeError: 'NoneType' object has no attribute 'add_default_arg'
#10 244.9 github-actions-v0.1 raised ValueError: source is not trusted
#10 244.9 Run `spack --debug ...` for more detailed errors
#10 244.9 ==> Bootstrapping clingo from pre-built binaries

Here we are trying spack install zlib

root@6a1628339d8f:/# spack/bin/spack install zlib
==> Bootstrapping clingo from pre-built binaries
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.2/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-2l45t4kw3cqqwj6vbxhfwhzlo6b3q2p4.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.2/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-2l45t4kw3cqqwj6vbxhfwhzlo6b3q2p4.spack
==> Installing "clingo-bootstrap@spack%gcc@10.2.1~docs~ipo+python+static_libstdcpp build_type=Release arch=linux-centos7-x86_64" from a buildcache
==> Bootstrapping patchelf from pre-built binaries
==> Bootstrapping patchelf@0.13.1:0.13.99 %gcc target=x86_64 from sources
==> Installing patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd
==> No binary for patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/08/08c0237e89be74d61ddf8f6ff218439cdd62af572d568fb38913b53e222831de.tar.gz
==> No patches needed for patchelf
==> patchelf: Executing phase: 'autoreconf'
==> patchelf: Executing phase: 'configure'
==> patchelf: Executing phase: 'build'
==> patchelf: Executing phase: 'install'
==> patchelf: Successfully installed patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd
  Fetch: 0.40s.  Build: 4.63s.  Total: 5.03s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/patchelf-0.13.1-dsjzmt4wboogb6lrfycmit6tdxpni7pd
==> Bootstrapping clingo from sources
==> Installing libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc
==> No binary for libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/e6/e6a1b1b589654277ee790cce3734f07876ac4ccfaecbee8afa0b649cf529cc04.tar.gz
==> No patches needed for libiconv
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
==> libiconv: Successfully installed libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc
  Fetch: 0.93s.  Build: 11.63s.  Total: 12.56s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/libiconv-1.16-qgiqsj3me7n7a33idnkjhfrje6gtkzfc
==> Installing libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p
==> No binary for libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/be/be78ee4176b05f7c75ff03298d84874db90f4b6c9d5503f0da1226b3a3c48119.tar.gz
==> No patches needed for libsigsegv
==> libsigsegv: Executing phase: 'autoreconf'
==> libsigsegv: Executing phase: 'configure'
==> libsigsegv: Executing phase: 'build'
==> libsigsegv: Executing phase: 'install'
==> libsigsegv: Successfully installed libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p
  Fetch: 0.84s.  Build: 2.66s.  Total: 3.50s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/libsigsegv-2.13-dvx6jjg4rhvuwr6mzpbecfjt6hwiju2p
==> Installing berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6
==> No binary for berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0c/0cecb2ef0c67b166de93732769abdeba0555086d51de1090df325e18ee8da9c8.tar.gz
==> Applied patch /spack/var/spack/repos/builtin/packages/berkeley-db/drop-docs.patch
==> Ran patch() for berkeley-db
==> berkeley-db: Executing phase: 'autoreconf'
==> berkeley-db: Executing phase: 'configure'
==> berkeley-db: Executing phase: 'build'
==> berkeley-db: Executing phase: 'install'
==> berkeley-db: Successfully installed berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6
  Fetch: 3.41s.  Build: 11.90s.  Total: 15.31s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/berkeley-db-18.1.40-emdzfk4edy5a5kkudnchca5xvz6n6hs6
==> Installing pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
==> No binary for pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ef/ef9c7e61822b7cb8356e6e9e1dca58d9556f3200d78acab35e4347e9d4c2bbaf.tar.xz
==> No patches needed for pkgconf
==> pkgconf: Executing phase: 'autoreconf'
==> pkgconf: Executing phase: 'configure'
==> pkgconf: Executing phase: 'build'
==> pkgconf: Executing phase: 'install'
==> pkgconf: Successfully installed pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
  Fetch: 0.92s.  Build: 2.55s.  Total: 3.47s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/pkgconf-1.8.0-wsqf3pdlaknj3vt7bq7skjmztdd5flps
==> Installing zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun
==> No binary for zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
==> Applied patch /spack/var/spack/repos/builtin/packages/zlib/configure-cc.patch
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun
  Fetch: 0.31s.  Build: 0.64s.  Total: 0.95s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/zlib-1.2.12-edugrkv3cwwfvj7um4p4pcb77joh6bun
==> Installing ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj
==> No binary for ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/19/1979e7fe618c51ed1c9df43bba92f977a0d3fe7497ffa2a5e80dfc559a1e5a29
==> No patches needed for ca-certificates-mozilla
==> ca-certificates-mozilla: Executing phase: 'install'
==> ca-certificates-mozilla: Successfully installed ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj
  Fetch: 0.12s.  Build: 0.02s.  Total: 0.14s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/ca-certificates-mozilla-2022-03-29-n67mrapikyuvp7lnatgsizro6a6qdbzj
[+] /usr (external python-3.10-e5bbvi6anav5ecs3qlpuba5fdbhiux4z)
==> Installing re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45
==> No binary for re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45 found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0f/0fc45e4130a8a555d68e230d1795de0216dfe99096b61b28e67c86dfd7d86bda.tar.xz
==> No patches needed for re2c
==> re2c: Executing phase: 'autoreconf'
==> re2c: Executing phase: 'configure'
==> re2c: Executing phase: 'build'
==> re2c: Executing phase: 'install'
==> re2c: Successfully installed re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45
  Fetch: 0.32s.  Build: 8.41s.  Total: 8.73s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/re2c-2.2-oxzzls2ckny5zaui2bacjo4qqvoiee45
==> Installing diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z
==> No binary for diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/a6/a6bdd7d1b31266d11c4f4de6c1b748d4607ab0231af5188fc2533d0ae2438fec.tar.xz
==> No patches needed for diffutils
==> diffutils: Executing phase: 'autoreconf'
==> diffutils: Executing phase: 'configure'
==> diffutils: Executing phase: 'build'
==> diffutils: Executing phase: 'install'
==> diffutils: Successfully installed diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z
  Fetch: 0.35s.  Build: 14.91s.  Total: 15.26s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/diffutils-3.8-4zbnujfiapq5yeuc7xdv4bejn57w4d5z
==> Installing m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx
==> No binary for m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/3b/3be4a26d825ffdfda52a56fc43246456989a3630093cced3fbddf4771ee58a70.tar.gz
==> Applied patch /spack/var/spack/repos/builtin/packages/m4/checks-198.sysval.1.patch
==> Applied patch /spack/var/spack/repos/builtin/packages/m4/checks-198.sysval.2.patch
==> Ran patch() for m4
==> m4: Executing phase: 'autoreconf'
==> m4: Executing phase: 'configure'
==> m4: Executing phase: 'build'
==> m4: Executing phase: 'install'
==> m4: Successfully installed m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx
  Fetch: 0.48s.  Build: 19.60s.  Total: 20.08s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/m4-1.4.19-slscbtmpt5tj75r2xezhcr2cbcw3kzvx
==> Installing ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk
==> No binary for ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/30/30306e0c76e0f9f1f0de987cf1c82a5c21e1ce6568b9227f7da5b71cbea86c9d.tar.gz
==> No patches needed for ncurses
==> ncurses: Executing phase: 'autoreconf'
==> ncurses: Executing phase: 'configure'
==> ncurses: Executing phase: 'build'
==> ncurses: Executing phase: 'install'
==> ncurses: Successfully installed ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk
  Fetch: 0.43s.  Build: 28.46s.  Total: 28.89s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/ncurses-6.2-ylxqh4yqmlbrjiexg3ej3v7vibzvedbk
==> Installing bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw
==> No binary for bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/ab/ab5a03176ee106d3f0fa90e381da478ddae405918153cca248e682cd0c4a2269.tar.gz
==> Ran patch() for bzip2
==> bzip2: Executing phase: 'install'
==> bzip2: Successfully installed bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw
  Fetch: 0.24s.  Build: 1.37s.  Total: 1.61s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/bzip2-1.0.8-ufk3ruarnyb7d5vkutpwgozseljwzmlw
==> Installing readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu
==> No binary for readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/f8/f8ceb4ee131e3232226a17f51b164afc46cd0b9e6cef344be87c65962cb82b02.tar.gz
==> Ran patch() for readline
==> readline: Executing phase: 'autoreconf'
==> readline: Executing phase: 'configure'
==> readline: Executing phase: 'build'
==> readline: Executing phase: 'install'
==> readline: Successfully installed readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu
  Fetch: 0.46s.  Build: 3.62s.  Total: 4.08s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/readline-8.1-6op7wd26tpq2olfhj2fm5n33nq6yk7pu
==> Installing gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn
==> No binary for gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/37/37ed12214122b972e18a0d94995039e57748191939ef74115b1d41d8811364bc.tar.gz
==> No patches needed for gdbm
==> gdbm: Executing phase: 'autoreconf'
==> gdbm: Executing phase: 'configure'
==> gdbm: Executing phase: 'build'
==> gdbm: Executing phase: 'install'
==> gdbm: Successfully installed gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn
  Fetch: 0.20s.  Build: 2.95s.  Total: 3.16s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/gdbm-1.19-ujhnyjcyshsddopgbi4o7jjyfxnwbbwn
==> Installing perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue
==> No binary for perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/35/357951a491b0ba1ce3611263922feec78ccd581dddc24a446b033e25acf242a1.tar.gz
==> Fetching https://mirror.spack.io/_source-cache/archive/9d/9da50e155df72bce55cb69f51f1dbb4b62d23740fb99f6178bb27f22ebdf8a46.tar.gz
==> Moving resource stage
        source: /tmp/root/spack-stage/resource-cpanm-wbl2db56q2knu45sdbx4v7zn4ju6cuue/spack-src/
        destination: /tmp/root/spack-stage/spack-stage-perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue/spack-src/cpanm/cpanm
==> Ran patch() for perl
==> perl: Executing phase: 'configure'
==> perl: Executing phase: 'build'
==> perl: Executing phase: 'install'
==> perl: Successfully installed perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue
  Fetch: 2.38s.  Build: 51.42s.  Total: 53.80s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/perl-5.34.1-wbl2db56q2knu45sdbx4v7zn4ju6cuue
==> Installing bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri
==> No binary for bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/06/06c9e13bdf7eb24d4ceb6b59205a4f67c2c7e7213119644430fe82fbd14a0abb.tar.gz
==> No patches needed for bison
==> bison: Executing phase: 'autoreconf'
==> bison: Executing phase: 'configure'
==> bison: Executing phase: 'build'
==> bison: Executing phase: 'install'
==> bison: Successfully installed bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri
  Fetch: 0.63s.  Build: 15.03s.  Total: 15.66s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/bison-3.8.2-c62tseejkkakoffdrh3b7uitrlqcx5ri
==> Installing openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th
==> No binary for openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/93/9384a2b0570dd80358841464677115df785edb941c71211f75076d72fe6b438f.tar.gz
==> Ran patch() for openssl
==> openssl: Executing phase: 'install'
==> openssl: Successfully installed openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th
  Fetch: 1.01s.  Build: 11.01s.  Total: 12.02s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/openssl-1.1.1o-fbbkvo5kbkmxfkpgrjwwbuh6k3fil3th
==> Installing cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc
==> No binary for cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/f3/f316b40053466f9a416adf981efda41b160ca859e97f6a484b447ea299ff26aa.tar.gz
==> No patches needed for cmake
==> cmake: Executing phase: 'bootstrap'
==> cmake: Executing phase: 'build'
==> cmake: Executing phase: 'install'
==> cmake: Successfully installed cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc
  Fetch: 1.10s.  Build: 1m 35.84s.  Total: 1m 36.94s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/cmake-3.23.2-42fawrjzxxeqqtuayz4ghguomyv6bxzc
==> Installing clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc
==> No binary for clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc found: installing from source
==> Ran patch() for clingo-bootstrap
==> clingo-bootstrap: Executing phase: 'cmake'
==> clingo-bootstrap: Executing phase: 'build'
==> clingo-bootstrap: Executing phase: 'install'
==> clingo-bootstrap: Successfully installed clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc
  Fetch: 7.09s.  Build: 26.89s.  Total: 33.98s.
[+] /root/.spack/bootstrap/store/linux-ubuntu22.04-x86_64/gcc-11.2.0/clingo-bootstrap-spack-vz27bxzu76bwbefdb33nqsbw5osbirrc
==> Warning: Spec clingo-bootstrap@spack%gcc@11.2.0~docs~ipo+python~static_libstdcpp build_type=Release arch=linux-ubuntu22.04-x86_64 ^berkeley-db@18.1.40%gcc@11.2.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu22.04-x86_64 ^bison@3.8.2%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^bzip2@1.0.8%gcc@11.2.0~debug~pic+shared arch=linux-ubuntu22.04-x86_64 ^ca-certificates-mozilla@2022-03-29%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^cmake@3.23.2%gcc@11.2.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu22.04-x86_64 ^diffutils@3.8%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^gdbm@1.19%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^libiconv@1.16%gcc@11.2.0 libs=shared,static arch=linux-ubuntu22.04-x86_64 ^libsigsegv@2.13%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^m4@1.4.19%gcc@11.2.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64 ^ncurses@6.2%gcc@11.2.0~symlinks+termlib abi=none arch=linux-ubuntu22.04-x86_64 ^openssl@1.1.1o%gcc@11.2.0~docs~shared certs=mozilla arch=linux-ubuntu22.04-x86_64 ^perl@5.34.1%gcc@11.2.0+cpanm+shared+threads arch=linux-ubuntu22.04-x86_64 ^pkgconf@1.8.0%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^python@3.10%gcc@11.2.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,7d40923,f2fd060 arch=linux-ubuntu22.04-x86_64 ^re2c@2.2%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^readline@8.1%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu22.04-x86_64 did not provide module clingo
==> Warning: Spec clingo-bootstrap@spack%gcc@11.2.0~docs~ipo+python~static_libstdcpp build_type=Release arch=linux-ubuntu22.04-x86_64 ^berkeley-db@18.1.40%gcc@11.2.0+cxx~docs+stl patches=b231fcc arch=linux-ubuntu22.04-x86_64 ^bison@3.8.2%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^bzip2@1.0.8%gcc@11.2.0~debug~pic+shared arch=linux-ubuntu22.04-x86_64 ^ca-certificates-mozilla@2022-03-29%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^cmake@3.23.2%gcc@11.2.0~doc+ncurses+ownlibs~qt build_type=Release arch=linux-ubuntu22.04-x86_64 ^diffutils@3.8%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^gdbm@1.19%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^libiconv@1.16%gcc@11.2.0 libs=shared,static arch=linux-ubuntu22.04-x86_64 ^libsigsegv@2.13%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^m4@1.4.19%gcc@11.2.0+sigsegv patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-x86_64 ^ncurses@6.2%gcc@11.2.0~symlinks+termlib abi=none arch=linux-ubuntu22.04-x86_64 ^openssl@1.1.1o%gcc@11.2.0~docs~shared certs=mozilla arch=linux-ubuntu22.04-x86_64 ^perl@5.34.1%gcc@11.2.0+cpanm+shared+threads arch=linux-ubuntu22.04-x86_64 ^pkgconf@1.8.0%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^python@3.10%gcc@11.2.0+bz2+ctypes+dbm~debug+ensurepip+libxml2+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4+uuid+zlib patches=0d98e93,7d40923,f2fd060 arch=linux-ubuntu22.04-x86_64 ^re2c@2.2%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^readline@8.1%gcc@11.2.0 arch=linux-ubuntu22.04-x86_64 ^zlib@1.2.12%gcc@11.2.0+optimize+pic+shared patches=0d38234 arch=linux-ubuntu22.04-x86_64 did not provide module clingo
==> Error: cannot bootstrap the "clingo" Python module from spec "clingo-bootstrap@spack+python %gcc target=x86_64" due to the following failures:
github-actions-v0.2 raised AttributeError: 'NoneType' object has no attribute 'add_default_arg'
github-actions-v0.1 raised ValueError: source is not trusted
Run `spack --debug ...` for more detailed errors
cyrush commented 2 years ago

VTK-m Cuda build is failing in ubuntu 18.04 cuda 11.4 container builds:

+] /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/conduit-0.8.3-txnuvayyylh3ujihxfwzdxc4qzvr3hwa
==> Installing vtk-m-1.7.1-i6fdr5cmeooavmahuj75bbr637ktkuwm
==> No binary for vtk-m-1.7.1-i6fdr5cmeooavmahuj75bbr637ktkuwm found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/7e/7ea3e945110b837a8c2ba49b41e45e1a1d8d0029bb472b291f7674871dbbbb63.tar.gz
==> No patches needed for vtk-m
==> vtk-m: Executing phase: 'cmake'
==> vtk-m: Executing phase: 'build'
==> Error: ProcessError: Command exited with status 2:
    'make' '-j8'

The spack build process hangs before it cleans up, the hang is due to some lock in use by spack.

KeyboardInterrupt
  File "/usr/lib/python3.6/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/lib/python3.6/multiprocessing/pool.py", line 108, in worker
    task = get()
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 334, in get
    with self._rlock:
  File "/usr/lib/python3.6/multiprocessing/synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()

I hunted around and found the logs while the spack process was still hung.

Kokkos build errors. I need to explicitly disable kokkos in VTK-m (it was on for another package, not VTK-m)

/tmp/tmpxft_00005b45_00000000-6_CellLocatorBoundingIntervalHierarchy.cudafe1.stub.c:69:210: error: template argument 1 is invalid
 typedef Kokkos::Impl::ParallelFor< ::,  ::Kokkos::MDRangePolicy< ::Kokkos::Cuda,  ::Kokkos::Rank<(unsigned int)3u, ( ::Kokkos::Iterate)0, ( ::Kokkos::Iterate)0> ,  ::Kokkos::IndexType<int>  > ,  ::Kokkos::Cuda>  _ZN6Kokkos4Impl11ParallelForIZN4vtkm4cont22DeviceAdapterAlgorithmINS3_22DeviceAdapterTagKokkosEE12ScheduleTaskIKNS2_7worklet16spatialstructure19CellRangesExtracterEKNS2_8internal10InvocationINSC_17FunctionInterfaceIFvNS2_4exec22ConnectivityStructuredINS2_22TopologyElementTagCellENS2_23TopologyElementTagPointELi2EEENSF_24ExecutionWholeArrayConstINS2_3VecIdLi3EEENS3_21StorageTagMultiplexerIJNS3_15StorageTagBasicENS3_13StorageTagSOAENS3_23StorageTagUniformPointsENS3_26StorageTagCartesianProductISO_SO_SO_EENS3_14StorageTagCastINSL_IfLi3EEESO_EENST_ISU_SP_EENST_ISU_SS_EEEEEJEEENSC_21ArrayPortalBasicWriteINS2_5RangeEEES12_S12_NS10_IdEES13_S13_EEENSE_IFvNS8_18WorkletMapTopologyISH_SI_E9CellSetInENS8_8internal11WorkletBase12WholeArrayInENS17_8FieldOutES1C_S1C_S1C_S1C_S1C_EEENSE_IFvNS2_12placeholders3ArgILi1EEENS17_22IncidentElementIndicesENS1G_ILi2EEENS1G_ILi3EEENS1G_ILi4EEENS1G_ILi5EEENS1G_ILi6EEENS1G_ILi7EEENS1G_ILi8EEEEEELi1ENSC_19ArrayPortalImplicitINSC_12IndexFunctorEEENS1S_INS3_8internal15ConstantFunctorIiEEEES1U_S5_EEEEvRNSF_6kokkos8internal11TaskBasic3DIT_T0_EENSL_IiLi3EEEEUliiiE_NS_13MDRangePolicyIJNS_4CudaENS_4RankILj3ELNS_7IterateE0ELS2D_0EEENS_9IndexTypeIiEEEEES2B_EE;
                                                                                                                                                                                                                  ^
In file included from tmpxft_00005b45_00000000-6_CellLocatorBoundingIntervalHierarchy.cudafe1.stub.c:1:0:
/tmp/tmpxft_00005b45_00000000-6_CellLocatorBoundingIntervalHierarchy.cudafe1.stub.c:70:210: error: template argument 1 is invalid
 typedef Kokkos::Impl::ParallelFor< ::,  ::Kokkos::MDRangePolicy< ::Kokkos::Cuda,  ::Kokkos::Rank<(unsigned int)3u, ( ::Kokkos::Iterate)0, ( ::Kokkos::Iterate)0> ,  ::Kokkos::IndexType<int>  > ,  ::Kokkos::Cuda>  _ZN6Kokkos4Impl11ParallelForIZN4vtkm4cont22DeviceAdapterAlgorithmINS3_22DeviceAdapterTagKokkosEE12ScheduleTaskIKNS2_7worklet16spatialstructure19CellRangesExtracterEKNS2_8internal10InvocationINSC_17FunctionInterfaceIFvNS2_4exec22ConnectivityStructuredINS2_22TopologyElementTagCellENS2_23TopologyElementTagPointELi3EEENSF_24ExecutionWholeArrayConstINS2_3VecIdLi3EEENS3_21StorageTagMultiplexerIJNS3_15StorageTagBasicENS3_13StorageTagSOAENS3_23StorageTagUniformPointsENS3_26StorageTagCartesianProductISO_SO_SO_EENS3_14StorageTagCastINSL_IfLi3EEESO_EENST_ISU_SP_EENST_ISU_SS_EEEEEJEEENSC_21ArrayPortalBasicWriteINS2_5RangeEEES12_S12_NS10_IdEES13_S13_EEENSE_IFvNS8_18WorkletMapTopologyISH_SI_E9CellSetInENS8_8internal11WorkletBase12WholeArrayInENS17_8FieldOutES1C_S1C_S1C_S1C_S1C_EEENSE_IFvNS2_12placeholders3ArgILi1EEENS17_22IncidentElementIndicesENS1G_ILi2EEENS1G_ILi3EEENS1G_ILi4EEENS1G_ILi5EEENS1G_ILi6EEENS1G_ILi7EEENS1G_ILi8EEEEEELi1ENSC_19ArrayPortalImplicitINSC_12IndexFunctorEEENS1S_INS3_8internal15ConstantFunctorIiEEEES1U_S5_EEEEvRNSF_6kokkos8internal11TaskBasic3DIT_T0_EENSL_IiLi3EEEEUliiiE_NS_13MDRangePolicyIJNS_4CudaENS_4RankILj3ELNS_7IterateE0ELS2D_0EEENS_9IndexTypeIiEEEEES2B_EE;
cyrush commented 2 years ago

Device Link Errors with Dray + Umpire.

  >> 559     /tmp/tmpxft_00003d1e_00000000-6_ResourceManager.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_50_tmpxft_00003d1e_000000
             00_7_ResourceManager_cpp1_ii_42a9a1b2'
     560     /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-6.0.0-bf6cwaevo72rqisypouvy4j2i34cszcd/lib/libumpire.a(Replay.cpp.o): In f
             unction `__sti____cudaRegisterAll()':
  >> 561     /tmp/tmpxft_00003d1d_00000000-6_Replay.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_41_tmpxft_00003d1d_00000000_7_Repl
             ay_cpp1_ii_5eca6429'
     562     /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-6.0.0-bf6cwaevo72rqisypouvy4j2i34cszcd/lib/libumpire.a(Umpire.cpp.o): In f
             unction `__sti____cudaRegisterAll()':
  >> 563     /tmp/tmpxft_00003d1a_00000000-6_Umpire.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_41_tmpxft_00003d1a_00000000_7_Umpi
             re_cpp1_ii_4507af2f'
     564     /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-6.0.0-bf6cwaevo72rqisypouvy4j2i34cszcd/lib/libumpire.a(Allocator.cpp.o): I
             n function `__sti____cudaRegisterAll()':
  >> 565     /tmp/tmpxft_00003d1c_00000000-6_Allocator.cudafe1.stub.c:14: undefined reference to `__cudaRegisterLinkedBinary_44_tmpxft_00003d1c_00000000_7_A
             llocator_cpp1_ii_a17095a1'
  >> 566     collect2: error: ld returned 1 exit status
  >> 567     tests/thirdparty/CMakeFiles/t_umpire_smoke.dir/build.make:104: recipe for target 'tests/thirdparty/t_umpire_smoke' failed
  >> 568     make[2]: *** [tests/thirdparty/t_umpire_smoke] Error 1
cyrush commented 2 years ago

mfem failed to build in cuda container.

problem with petsc, which comes in directly from mfem and also sundials, new mfem spec for cuda container:

^mfem~cuda~petsc~sundials~slepc
cyrush commented 2 years ago

Umpire config issue in cuda container:

-- Looking for Umpire in: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-2022.03.1-6fjbta4536ffgtebjiznj5vczjv5ks7p
-- Configuring incomplete, errors occurred!
See also "/__w/1/s/build/CMakeFiles/CMakeOutput.log".
See also "/__w/1/s/build/CMakeFiles/CMakeError.log".
CMake Error at cmake/thirdparty/SetupUmpire.cmake:12 (find_package):
  Could not find a package configuration file provided by "umpire" with any
  of the following names:

    umpireConfig.cmake
    umpire-config.cmake
white238 commented 2 years ago

Umpire config issue in cuda container:

-- Looking for Umpire in: /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-2022.03.1-6fjbta4536ffgtebjiznj5vczjv5ks7p
-- Configuring incomplete, errors occurred!
See also "/__w/1/s/build/CMakeFiles/CMakeOutput.log".
See also "/__w/1/s/build/CMakeFiles/CMakeError.log".
CMake Error at cmake/thirdparty/SetupUmpire.cmake:12 (find_package):
  Could not find a package configuration file provided by "umpire" with any
  of the following names:

    umpireConfig.cmake
    umpire-config.cmake

They got moved from share/umpire/cmake to lib/cmake/umpire. Not sure why it is only broken in your CUDA build, the following line should be broken on all builds:

https://github.com/Alpine-DAV/ascent/blob/1641fc7d57d0350d961e95cf08341e0a7f4438a5/src/cmake/thirdparty/SetupUmpire.cmake#L12-L14

cyrush commented 2 years ago

@white238 thanks this also explains an issue I had building dray release with spack.

cyrush commented 2 years ago

now umpire can't find camp, I might have to explicitly set camp dir? Answer: Yes

CMake Error at /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/cmake-3.23.2-ljzqnmnctewf3yne2azj7f5u26ypdtoz/share/cmake-3.23/Modules/CMakeFindDependencyMacro.cmake:47 (find_package):
  Could not find a package configuration file provided by "camp" with any of
  the following names:

    campConfig.cmake
    camp-config.cmake

  Add the installation prefix of "camp" to CMAKE_PREFIX_PATH or set
  "camp_DIR" to a directory containing one of the above files.  If "camp"
  provides a separate development package or SDK, be sure it has been
  installed.
Call Stack (most recent call first):
  /uberenv_libs/spack/opt/spack/linux-ubuntu18.04-zen/gcc-7.5.0/umpire-2022.03.1-6fjbta4536ffgtebjiznj5vczjv5ks7p/lib/cmake/umpire/umpire-config.cmake:35 (find_dependency)
  cmake/thirdparty/SetupUmpire.cmake:21 (find_package)
cyrush commented 2 years ago

Current CUDA Spack TPL install size comparisons:

Old: 3.5GB New: 8.1GB

Issue: System cuda is not being used, spack is installing cuda 11.7 , config was not being passed to uberenv due to typo in script.

cyrush commented 2 years ago

Now we have RAJA BLT issue:

 ==> Installing raja-2022.03.0-nek4v4jgkkavtfvpprao4pfv5peky7hc
#10 2237.5 ==> No binary for raja-2022.03.0-nek4v4jgkkavtfvpprao4pfv5peky7hc found: installing from source
#10 2239.7 ==> No patches needed for raja
...
#10 2241.2   >> 123    /usr/bin/ld: cannot find -lcuda_runtime
#10 2241.2   >> 124    collect2: error: ld returned 1 exit status
#10 2241.2   >> 125    CMakeFiles/RAJA.dir/build.make:196: recipe for target 'lib/libRAJA.
#10 2241.2             so' failed
cyrush commented 2 years ago

problem using ubernev + spack on macos:

[exe: spack/bin/spack install uberenv-ascent@develop%apple-clang]
==> Error: No valid value for variant 'ownlibs' of package 'cmake'

concertize can't figure out what to use for ownlibs for cmake.

If I try to disable ownlibs, I get another problem:

236, in parse_specs
    raise spack.error.SpackError(msg)
spack.error.SpackError: Cannot find valid provider for virtual pkgconfig

uberenv-ascent inheriting from ascent seems to be completely confusing the clingo concretization process:

==> Error: No valid value for variant 'ownlibs' of package 'cmake' (so i add to the spec) ==> Error: No valid value for variant 'shared' of package 'uberenv-ascent' (so i add to spec) ==> Error: No valid value for variant 'cuda' of package 'uberenv-ascent' (so I add to sepc) ==> Error: No versions available for package 'umpire' umpire exists and has several versions there is lots of logic in these packages to make the right selections about this stuff, but it seem I have to set them all by hand ...

cyrush commented 2 years ago

the issue is related to disabling local configs:

env  SPACK_DISABLE_LOCAL_CONFIG=1 bin/spack spec uberenv-ascent 
==> Error: No valid value for variant 'ownlibs' of package 'cmake'

when we don't disable, it works. 

bin/spack spec uberenv-ascent 
Input spec
--------------------------------
uberenv-ascent

Concretized
--------------------------------
uberenv-ascent@0.8.0%apple-clang@11.0.0~adios~adios2~babelflow+blt_find_mpi~cuda~doc~dray~fides+fortran~genten~ipo~mfem+mpi~occa~openmp+python+raja+serial+shared+test+umpire+vtkh build_type=RelWithDebInfo arch=darwin-catalina-skylake

Still working on root cause, we are not using packages.yaml in this case only a compilers.yaml in the spack default place. So SPACK_DISABLE_LOCAL_CONFIG should only block ~/.spack/

cyrush commented 2 years ago

To update this thread: We hit issues with rocm package layouts vs what the spack packages expected. These are still unresolved for us, other folks may have been working on smoothing them out.

For hip builds, we are using a script.

cyrush commented 2 years ago

Since we are tracking spack develop (which was an accident on my part), I hit an issue with macOS not being able to build openmpi. Consistently fails to fetch hwloc source. https://github.com/spack/spack/issues/32503

Also raja tries to build with openmp support, but we don't have in default macOS compilers -- so we have to use ^raja~openmp there is solution to this already when we use the newest raja package -- https://github.com/spack/spack/pull/32284.

cyrush commented 1 year ago

Related PR #1012

cyrush commented 1 year ago

Spack was updated for CI part of the VTK-m 1.9 update - https://github.com/Alpine-DAV/ascent/pull/1015.

We still do not have a method for using spack for hip builds, instead we use build_ascent_hip.sh` and friends.