search

HDFGroup / hdf5-iotest

HDF5 Performance Analysis Checklist
Other
12 stars 9 forks source link

spack spec generated does not included needed g++ compiler dependency #25

Open Noerr opened 1 year ago

Noerr commented 1 year ago

Follow the spack install commands on a brand-new fresh Ubuntu 22.04 system and the spack install fails.

==> Installing berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu
==> No binary for berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0c/0cecb2ef0c67b166de93732769abdeba0555086d51de1090df325e18ee8da9c8.tar.gz
==> Applied patch /home/noah/spack/var/spack/repos/builtin/packages/berkeley-db/drop-docs.patch
==> Applied patch /home/noah/spack/var/spack/repos/builtin/packages/berkeley-db/tls.patch
==> Ran patch() for berkeley-db
==> berkeley-db: Executing phase: 'autoreconf'
==> berkeley-db: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    '/tmp/noah/spack-stage/spack-stage-berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu/spack-src/dist/configure' '--prefix=/home/noah/spack/opt/spack/linux-ubuntu22.04-icelake/gcc-12.1.0/berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu' '--disable-static' '--enable-dbm' '--enable-compat185' '--with-repmgr-ssl=no' '--enable-cxx' '--enable-stl'

1 error found in build log:
     131    checking if libtool supports shared libraries... yes
     132    checking whether to build shared libraries... yes
     133    checking whether to build static libraries... no
     134    checking SOSUFFIX from libtool... .so
     135    checking MODSUFFIX from libtool... .so
     136    checking JMODSUFFIX from libtool... .so
  >> 137    checking whether the C++ compiler supports templates for STL... configure: error: no

berkeley-db was the third dependency spack attempted to build. Fifteen dependencies are then skipped because berkeley-db failed. Naive googling around suggests to me that something (probably berkeley-db) is not properly declaring a dependency on g++ so it wasn't installed.

Noerr commented 1 year ago

Output spack spec:

$ spack spec hdf5iotest
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/patchelf-0.16.1/linux-centos7-x86_64-gcc-10.2.1-patchelf-0.16.1-p72zyan5wrzuabtmzq7isa5mzyh6ahdp.spack
==> Installing "patchelf@0.16.1%gcc@10.2.1 ldflags="-static-libstdc++ -static-libgcc"  build_system=autotools arch=linux-centos7-x86_64" from a buildcache
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-idkenmhnscjlu5gjqhpcqa4h7o2a7aow.spec.json
==> Fetching https://mirror.spack.io/bootstrap/github-actions/v0.4/build_cache/linux-centos7-x86_64/gcc-10.2.1/clingo-bootstrap-spack/linux-centos7-x86_64-gcc-10.2.1-clingo-bootstrap-spack-idkenmhnscjlu5gjqhpcqa4h7o2a7aow.spack
==> Installing "clingo-bootstrap@spack%gcc@10.2.1~docs~ipo+python+static_libstdcpp build_type=Release arch=linux-centos7-x86_64" from a buildcache
Input spec
--------------------------------
hdf5iotest

Concretized
--------------------------------
hdf5iotest@master%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
    ^autoconf@2.69%gcc@12.1.0 build_system=autotools patches=35c4492,7793209,a49dd5b arch=linux-ubuntu22.04-icelake
        ^perl@5.36.0%gcc@12.1.0+cpanm+open+shared+threads build_system=generic arch=linux-ubuntu22.04-icelake
            ^berkeley-db@18.1.40%gcc@12.1.0+cxx~docs+stl build_system=autotools patches=26090f4,b231fcc arch=linux-ubuntu22.04-icelake
            ^bzip2@1.0.8%gcc@12.1.0~debug~pic+shared build_system=generic arch=linux-ubuntu22.04-icelake
            ^gdbm@1.23%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
                ^readline@8.2%gcc@12.1.0 build_system=autotools patches=bbf97f1 arch=linux-ubuntu22.04-icelake
    ^automake@1.16.5%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
    ^hdf5@1.14.0%gcc@12.1.0~cxx~fortran~hl~ipo~java+mpi+shared~szip~threadsafe+tools api=default build_system=cmake build_type=RelWithDebInfo generator=make patches=0b5dd6f arch=linux-ubuntu22.04-icelake
        ^cmake@3.25.2%gcc@12.1.0~doc+ncurses+ownlibs~qt build_system=generic build_type=Release arch=linux-ubuntu22.04-icelake
            ^ncurses@6.4%gcc@12.1.0~symlinks+termlib abi=none build_system=autotools arch=linux-ubuntu22.04-icelake
            ^openssl@1.1.1t%gcc@12.1.0~docs~shared build_system=generic certs=mozilla arch=linux-ubuntu22.04-icelake
                ^ca-certificates-mozilla@2023-01-10%gcc@12.1.0 build_system=generic arch=linux-ubuntu22.04-icelake
        ^gmake@4.4.1%gcc@12.1.0~guile build_system=autotools arch=linux-ubuntu22.04-icelake
        ^pkgconf@1.8.0%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
        ^zlib@1.2.13%gcc@12.1.0+optimize+pic+shared build_system=makefile arch=linux-ubuntu22.04-icelake
    ^libtool@2.4.7%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
    ^m4@1.4.19%gcc@12.1.0+sigsegv build_system=autotools patches=9dc5fbd,bfdffa7 arch=linux-ubuntu22.04-icelake
        ^diffutils@3.9%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
            ^libiconv@1.17%gcc@12.1.0 build_system=autotools libs=shared,static arch=linux-ubuntu22.04-icelake
        ^libsigsegv@2.14%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
    ^openmpi@4.1.5%gcc@12.1.0~atomics~cuda~cxx~cxx_exceptions~gpfs~internal-hwloc~java~legacylaunchers~lustre~memchecker~orterunprefix+romio+rsh~singularity+static+vt+wrapper-rpath build_system=autotools fabrics=none schedulers=none arch=linux-ubuntu22.04-icelake
        ^hwloc@2.9.0%gcc@12.1.0~cairo~cuda~gl~libudev+libxml2~netloc~nvml~oneapi-level-zero~opencl+pci~rocm build_system=autotools libs=shared,static arch=linux-ubuntu22.04-icelake
            ^libpciaccess@0.17%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
                ^util-macros@1.19.3%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
            ^libxml2@2.10.3%gcc@12.1.0~python build_system=autotools arch=linux-ubuntu22.04-icelake
                ^xz@5.4.1%gcc@12.1.0~pic build_system=autotools libs=shared,static arch=linux-ubuntu22.04-icelake
        ^numactl@2.0.14%gcc@12.1.0 build_system=autotools patches=4e1d78c,62fc8a8,ff37630 arch=linux-ubuntu22.04-icelake
        ^openssh@9.3p1%gcc@12.1.0+gssapi build_system=autotools arch=linux-ubuntu22.04-icelake
            ^krb5@1.20.1%gcc@12.1.0+shared build_system=autotools arch=linux-ubuntu22.04-icelake
                ^bison@3.8.2%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
                ^gettext@0.21.1%gcc@12.1.0+bzip2+curses+git~libunistring+libxml2+tar+xz build_system=autotools arch=linux-ubuntu22.04-icelake
                    ^tar@1.34%gcc@12.1.0 build_system=autotools zip=pigz arch=linux-ubuntu22.04-icelake
                        ^pigz@2.7%gcc@12.1.0 build_system=makefile arch=linux-ubuntu22.04-icelake
                        ^zstd@1.5.4%gcc@12.1.0+programs build_system=makefile compression=none libs=shared,static arch=linux-ubuntu22.04-icelake
            ^libedit@3.1-20210216%gcc@12.1.0 build_system=autotools arch=linux-ubuntu22.04-icelake
            ^libxcrypt@4.4.33%gcc@12.1.0~obsolete_api build_system=autotools arch=linux-ubuntu22.04-icelake
        ^pmix@4.1.2%gcc@12.1.0~docs+pmi_backwards_compatibility~python~restful build_system=autotools arch=linux-ubuntu22.04-icelake
            ^libevent@2.1.12%gcc@12.1.0+openssl build_system=autotools arch=linux-ubuntu22.04-icelake
Noerr commented 1 year ago

A workaround is to delete the references to gcc version 12 in the user's spack compilers.yaml For whatever reason this fresh ubuntu 22.04 install provides a complete gcc,g++,gfortran install at version 11.3.0 at /usr/bin but just a gcc-12 without g++ or gfortran in the same place. The basic spack commands mentioned in readme lead to spack finding both and preferring to use gcc-12. Things break later that want a c++ compiler.

$ spack compilers
==> Available compilers
-- gcc ubuntu22.04-x86_64 ---------------------------------------
gcc@12.1.0  gcc@11.3.0  gcc@9.5.0

$ /usr/bin/gcc-12 --version
gcc-12 (Ubuntu 12.1.0-2ubuntu1~22.04) 12.1.0
Copyright (C) 2022 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

$/usr/bin/g++-12 --version
-bash: /usr/bin/g++-12: No such file or directory

$ vim .spack/linux/compilers.yaml    #deleting all of the gcc-12 section

$ spack compilers
==> Available compilers
-- gcc ubuntu22.04-x86_64 ---------------------------------------
gcc@11.3.0  gcc@9.5.0

Then try again:

spack spec hdf5iotest
spack install hdf5iotest

This is surely no fault of this hdf5-iotest project, but I don't know where to point. I comment here to help others find workaround.

brtnfld commented 1 year ago

Does it work if you set spack to use the system installed: autoconf, automake, libtoool? The packages do not need c++ or fortran.

Noerr commented 1 year ago

I don't know how to do that with spack even though I've been using spack on and off in several contexts for four years. A day with spack in my command line means a day down rabbit holes.

Noerr commented 1 year ago

I don't see indication that spack installed its own autoconf, automate, or libtool. 

$ git clone -c feature.manyFiles=true https://github.com/spack/spack.git

$ . spack/share/spack/setup-env.sh

$ spack create --name hdf5iotest
==> Using specified package name: 'hdf5iotest'
==> Warning: Unable to detect a build system. Using a generic package template.
==> Created template for hdf5iotest package
==> Created package file: /home/noah/spack/var/spack/repos/builtin/packages/hdf5iotest/package.py

$ spack spec hdf5iotest

$ spack install hdf5iotest
==> Installing libiconv-1.17-i5775tgaptmwnkwg24jduarm3zl5jxf3
==> No binary for libiconv-1.17-i5775tgaptmwnkwg24jduarm3zl5jxf3 found: installing from source
==> Fetching https://ftpmirror.gnu.org/libiconv/libiconv-1.17.tar.gz
==> No patches needed for libiconv
==> libiconv: Executing phase: 'autoreconf'
==> libiconv: Executing phase: 'configure'
==> libiconv: Executing phase: 'build'
==> libiconv: Executing phase: 'install'
==> libiconv: Successfully installed libiconv-1.17-i5775tgaptmwnkwg24jduarm3zl5jxf3
  Stage: 2.71s.  Autoreconf: 0.00s.  Configure: 29.03s.  Build: 11.55s.  Install: 2.83s.  Total: 46.44s
[+] /home/noah/spack/opt/spack/linux-ubuntu22.04-icelake/gcc-12.1.0/libiconv-1.17-i5775tgaptmwnkwg24jduarm3zl5jxf3
==> Installing libsigsegv-2.14-6baoqj5q6d5mh4wjunjotb5fzt4lmfjs
==> No binary for libsigsegv-2.14-6baoqj5q6d5mh4wjunjotb5fzt4lmfjs found: installing from source
==> Fetching https://ftpmirror.gnu.org/libsigsegv/libsigsegv-2.14.tar.gz
==> No patches needed for libsigsegv
==> libsigsegv: Executing phase: 'autoreconf'
==> libsigsegv: Executing phase: 'configure'
==> libsigsegv: Executing phase: 'build'
==> libsigsegv: Executing phase: 'install'
==> libsigsegv: Successfully installed libsigsegv-2.14-6baoqj5q6d5mh4wjunjotb5fzt4lmfjs
  Stage: 2.82s.  Autoreconf: 0.00s.  Configure: 10.71s.  Build: 3.44s.  Install: 0.52s.  Total: 17.84s
[+] /home/noah/spack/opt/spack/linux-ubuntu22.04-icelake/gcc-12.1.0/libsigsegv-2.14-6baoqj5q6d5mh4wjunjotb5fzt4lmfjs
==> Installing berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu
==> No binary for berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu found: installing from source
==> Fetching https://mirror.spack.io/_source-cache/archive/0c/0cecb2ef0c67b166de93732769abdeba0555086d51de1090df325e18ee8da9c8.tar.gz
==> Applied patch /home/noah/spack/var/spack/repos/builtin/packages/berkeley-db/drop-docs.patch
==> Applied patch /home/noah/spack/var/spack/repos/builtin/packages/berkeley-db/tls.patch
==> Ran patch() for berkeley-db
==> berkeley-db: Executing phase: 'autoreconf'
==> berkeley-db: Executing phase: 'configure'
==> Error: ProcessError: Command exited with status 1:
    '/tmp/noah/spack-stage/spack-stage-berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu/spack-src/dist/configure' '--prefix=/home/noah/spack/opt/spack/linux-ubuntu22.04-icelake/gcc-12.1.0/berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu' '--disable-static' '--enable-dbm' '--enable-compat185' '--with-repmgr-ssl=no' '--enable-cxx' '--enable-stl'

1 error found in build log:
     131    checking if libtool supports shared libraries... yes
     132    checking whether to build shared libraries... yes
     133    checking whether to build static libraries... no
     134    checking SOSUFFIX from libtool... .so
     135    checking MODSUFFIX from libtool... .so
     136    checking JMODSUFFIX from libtool... .so
  >> 137    checking whether the C++ compiler supports templates for STL... configure: error: no

See build log for details:
  /tmp/noah/spack-stage/spack-stage-berkeley-db-18.1.40-jw4pptioy6o3tcaiz46z7ze6bv3zhfvu/spack-build-out.txt
brtnfld commented 1 year ago

It is installing berkely-db as part of autoconf (i.e. it fails to install autoconf).

You can mark those packages as not buildable.

https://spack.readthedocs.io/en/latest/build_settings.html

I think spack was going to make this easier in a future/newer release if I remember correctly.