Container builds broken after moving spack extensions out of spack

[climbfuji]

Describe the bug Following yesterday's merge that moves the spack extensions out of the spack submodule, we are having a problem with the container builds: https://github.com/JCSDA/spack-stack/actions/runs/7637196138/job/20805589102

#15 [builder 4/6] RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y
#15 3.161 ==> Error: Package 'spack.pkg.builtin.base-env' not found.
#15 3.161 Use 'spack create' to create a new package.
#15 3.161 
#15 3.161 Did you mean one of the following packages?
#15 3.161   seq-gen
#15 3.161   aspell6-en
#15 3.161   r-base64enc
#15 ERROR: process "docker-shell cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y" did not complete successfully: exit code: 1

The problem is the way the spack container builds work: they clone spack (and spack alone) in the container for the install. Only the config is copied into the container from the machine that runs the build (i.e. runs docker).

@AlexanderRichert-NOAA Any ideas on how to fix this? Otherwise I fear we need to revert.

To Reproduce See above

Expected behavior Containers build

System: Any building containers with spack

Additional context n/a

[AlexanderRichert-NOAA]

Hmm... Is there a way to copy in the repos directory at the beginning of the recipe?

[climbfuji]

Not really. The way the spack.yaml finds its way into the container is via spack containerize > Dockerfile with Dockerfile being, for example (cont'd below).

FROM ubuntu:20.04 as bootstrap

ENV SPACK_ROOT=/opt/spack \
    CURRENTLY_BUILDING_DOCKER_IMAGE=1 \
    container=docker

ENV DEBIAN_FRONTEND=noninteractive   \
    LANGUAGE=en_US.UTF-8 \
    LANG=en_US.UTF-8 \
    LC_ALL=en_US.UTF-8

RUN apt-get -yqq update \
 && apt-get -yqq upgrade \
 && apt-get -yqq install --no-install-recommends \
        build-essential \
        ca-certificates \
        curl \
        file \
        g++ \
        gcc \
        gfortran \
        git \
        gnupg2 \
        iproute2 \
        locales \
        make \
        mercurial \
        subversion \
        python3 \
        python3-pip \
        python3-setuptools \
        unzip \
        zstd \
 && locale-gen en_US.UTF-8 \
 && pip3 install boto3 \
 && rm -rf /var/lib/apt/lists/*

RUN mkdir $SPACK_ROOT && cd $SPACK_ROOT && \
    git init --quiet && git remote add origin https://github.com/jcsda/spack && git fetch --depth=1 origin e7ef791045012b6ef989be97c6eee24ec571c4b6 && git checkout --detach FETCH_HEAD && \
    mkdir -p $SPACK_ROOT/opt/spack

RUN ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
          /usr/local/bin/docker-shell \
 && ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
          /usr/local/bin/interactive-shell \
 && ln -s $SPACK_ROOT/share/spack/docker/entrypoint.bash \
          /usr/local/bin/spack-env

RUN mkdir -p /root/.spack \
 && cp $SPACK_ROOT/share/spack/docker/modules.yaml \
        /root/.spack/modules.yaml \
 && rm -rf /root/*.* /run/nologin

# [WORKAROUND]
# https://superuser.com/questions/1241548/
#     xubuntu-16-04-ttyname-failed-inappropriate-ioctl-for-device#1253889
RUN [ -f ~/.profile ]                                               \
 && sed -i 's/mesg n/( tty -s \\&\\& mesg n || true )/g' ~/.profile \
 || true

WORKDIR /root
SHELL ["docker-shell"]

# Creates the package cache
RUN spack bootstrap now \
    && spack bootstrap status --optional \
    && spack spec hdf5+mpi

ENTRYPOINT ["/bin/bash", "/opt/spack/share/spack/docker/entrypoint.bash"]
CMD ["interactive-shell"]

# Build stage with Spack pre-installed and ready to be used
FROM bootstrap as builder

# Install OS packages needed to build the software
RUN apt-get -yqq update && apt-get -yqq upgrade \
 && apt-get -yqq install bc clang-10 libclang-10-dev libc++-10-dev cpp g++ gcc gfortran git git-lfs make mysql-server qt5-default libqt5svg5-dev qt5dxcb-plugin wget \
 && rm -rf /var/lib/apt/lists/*

#Create symbolic links for clang compilers
RUN cd /usr/bin && \
ln -svf clang-10 clang && \
ln -svf clang++-10 clang++ && \
ln -svf clang-cpp-10 clang-cpp && \
cd /usr/lib/llvm-10/lib && \
ln -svf libc++abi.so.1.0 libc++abi.so
# Build mpich outside of spack-stack
ENV MPICH_VERSION=4.1.1
ENV CC=clang
ENV CXX=clang++
ENV FC=gfortran
RUN mkdir -p /opt/mpich-${MPICH_VERSION}/src && \
cd /opt/mpich-${MPICH_VERSION}/src && \
wget http://www.mpich.org/static/downloads/${MPICH_VERSION}/mpich-${MPICH_VERSION}.tar.gz && \
tar -xzf mpich-${MPICH_VERSION}.tar.gz && \
cd mpich-${MPICH_VERSION} && \
CFLAGS="-fPIC" CXXFLAGS="-fPIC" FCFLAGS="-fPIC" LDFLAGS="-fPIC" \
./configure --prefix=/opt/mpich-${MPICH_VERSION} --enable-fortran --enable-cxx --with-device=ch4:ofi && \
make -j2 && \
make install && \
ls /opt/mpich-${MPICH_VERSION}
ENV PATH=/opt/mpich-${MPICH_VERSION}/bin:${PATH}
ENV CPATH=/opt/mpich-${MPICH_VERSION}/include:${CPATH}
ENV LD_LIBRARY_PATH=/opt/mpich-${MPICH_VERSION}/lib:${LD_LIBRARY_PATH}

# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo spack: \
&&   echo '  concretizer:' \
&&   echo '    unify: true' \
&&   echo '' \
&&   echo '  config:' \
&&   echo '    checksum: false' \
&&   echo '    build_jobs: 2' \
&&   echo '    connect_timeout: 60' \
&&   echo '' \
&&   echo '    install_tree: /opt/software' \
&&   echo '  compilers:' \
&&   echo '  - compiler:' \
&&   echo '      spec: clang@10.0.0' \
&&   echo '      paths:' \
&&   echo '        cc: /usr/bin/clang' \
&&   echo '        cxx: /usr/bin/clang++' \
&&   echo '        f77: /usr/bin/gfortran' \
&&   echo '        fc: /usr/bin/gfortran' \
&&   echo '      flags: {}' \
&&   echo '      operating_system: ubuntu20.04' \
&&   echo '      modules: []' \
&&   echo '      environment:' \
&&   echo '        prepend_path:' \
&&   echo '          LD_LIBRARY_PATH: /usr/lib/llvm-10/lib' \
&&   echo '      extra_rpaths: []' \
&&   echo '' \
&&   echo '  # Basic package config from configs/common/packages.yaml' \
&&   echo '  # Additional package config for container' \
&&   echo '  packages:' \
&&   echo '    all:' \
&&   echo '      providers:' \
&&   echo '        blas: [openblas]' \
&&   echo '        fftw-api: [fftw]' \
&&   echo '        gl: [opengl]' \
&&   echo '        glu: [openglu]' \
&&   echo '        jpeg: [libjpeg-turbo]' \
&&   echo '        lapack: [openblas]' \
&&   echo '        yacc: [bison]' \
&&   echo '        zlib-api: [zlib]' \
&&   echo '        mpi: [mpich@4.1.1]' \
&&   echo '      require: '"'"'%clang'"'"'' \
&&   echo '      target: [x86_64]' \
&&   echo '      compiler: [clang@10.0.0]' \
&&   echo '    awscli:' \
&&   echo '      version: [1.27.84]' \
&&   echo '    bacio:' \
&&   echo '      version: [2.4.1]' \
&&   echo '    bison:' \
&&   echo '      version: [3.8.2]' \
&&   echo '    boost:' \
&&   echo '      version: [1.83.0]' \
&&   echo '      variants: ~atomic +chrono +date_time +exception +filesystem ~graph ~iostreams' \
&&   echo '        ~locale ~log ~math ~mpi ~numpy +pic +program_options +python ~random +regex' \
&&   echo '        +serialization ~signals +system +test +thread +timer ~wave cxxstd=17 visibility=hidden' \
&&   echo '    bufr:' \
&&   echo '      version: [12.0.1]' \
&&   echo '      variants: +python' \
&&   echo '    cairo:' \
&&   echo '      variants: +pic' \
&&   echo '    cdo:' \
&&   echo '      version: [2.2.0]' \
&&   echo '      variants: ~openmp' \
&&   echo '    cmake:' \
&&   echo '      version: [3.23.1]' \
&&   echo '      variants: +ownlibs' \
&&   echo '    # Attention - when updating also check the various jcsda-emc-bundles env packages' \
&&   echo '    crtm:' \
&&   echo '      version: [2.4.0.1]' \
&&   echo '      variants: +fix' \
&&   echo '    ecbuild:' \
&&   echo '      version: [3.7.2]' \
&&   echo '    eccodes:' \
&&   echo '      version: [2.32.0]' \
&&   echo '      variants: +png' \
&&   echo '    ecflow:' \
&&   echo '      version: [5.11.4]' \
&&   echo '      variants: +ui' \
&&   echo '    eckit:' \
&&   echo '      version: [1.24.5]' \
&&   echo '      variants: linalg=eigen,lapack compression=lz4,bzip2' \
&&   echo '    ecmwf-atlas:' \
&&   echo '      version: [0.35.1]' \
&&   echo '      variants: +fckit +trans +tesselation +fftw' \
&&   echo '    ectrans:' \
&&   echo '      version: [1.2.0]' \
&&   echo '      variants: ~mkl +fftw' \
&&   echo '    eigen:' \
&&   echo '      version: [3.4.0]' \
&&   echo '    esmf:' \
&&   echo '      version: [8.6.0]' \
&&   echo '      variants: ~xerces ~pnetcdf snapshot=none +shared +external-parallelio' \
&&   echo '      require:' \
&&   echo '      - any_of: [fflags="-fp-model precise" cxxflags="-fp-model precise"]' \
&&   echo '        when: '"'"'%intel'"'"'' \
&&   echo '        message: Extra ESMF compile options for Intel' \
&&   echo '      - any_of: ['"'"''"'"']' \
&&   echo '        when: '"'"'%gcc'"'"'' \
&&   echo '        message: Extra ESMF compile options for GCC' \
&&   echo '    fckit:' \
&&   echo '      version: [0.11.0]' \
&&   echo '      variants: +eckit' \
&&   echo '    fftw:' \
&&   echo '      version: [3.3.10]' \
&&   echo '    fiat:' \
&&   echo '      version: [1.2.0]' \
&&   echo '    fms:' \
&&   echo '      version: ['"'"'2023.04'"'"']' \
&&   echo '      variants: precision=32,64 +quad_precision +gfs_phys +openmp +pic constants=GFS' \
&&   echo '        build_type=Release +deprecated_io' \
&&   echo '    fontconfig:' \
&&   echo '      variants: +pic' \
&&   echo '    freetype:' \
&&   echo '      variants: +pic' \
&&   echo '    g2:' \
&&   echo '      version: [3.4.5]' \
&&   echo '    g2c:' \
&&   echo '      version: [1.6.4]' \
&&   echo '    g2tmpl:' \
&&   echo '      # https://github.com/JCSDA/spack-stack/issues/525' \
&&   echo '      version: [1.10.2]' \
&&   echo '      require:' \
&&   echo '      - any_of: [fflags="-no-pie"]' \
&&   echo '        when: '"'"'@1.10.2: %clang'"'"'' \
&&   echo '        message: Extra ESMF compile options for version 1.10.2+ with clang' \
&&   echo '    gettext:' \
&&   echo '      version: [0.21.1]' \
&&   echo '    gfsio:' \
&&   echo '      version: [1.4.1]' \
&&   echo '    gftl-shared:' \
&&   echo '      version: [1.6.1]' \
&&   echo '    grib-util:' \
&&   echo '      version: [1.3.0]' \
&&   echo '    gsibec:' \
&&   echo '      version: [1.1.3]' \
&&   echo '    gsi-ncdiag:' \
&&   echo '      version: [1.1.2]' \
&&   echo '    gsl-lite:' \
&&   echo '      version: [0.37.0]' \
&&   echo '    hdf:' \
&&   echo '      version: [4.2.15]' \
&&   echo '      variants: +external-xdr ~fortran ~netcdf' \
&&   echo '    hdf5:' \
&&   echo '      version: [1.14.0]' \
&&   echo '      variants: +hl +fortran +mpi ~threadsafe +szip' \
&&   echo '    ip:' \
&&   echo '      version: [4.3.0]' \
&&   echo '      variants: precision=4,d,8' \
&&   echo '    ip2:' \
&&   echo '      version: [1.1.2]' \
&&   echo '    jasper:' \
&&   echo '      version: [2.0.32]' \
&&   echo '    jedi-cmake:' \
&&   echo '      version: [1.4.0]' \
&&   echo '    jpeg:' \
&&   echo '      version: [9.1.0]' \
&&   echo '    landsfcutil:' \
&&   echo '      version: [2.4.1]' \
&&   echo '    libjpeg-turbo:' \
&&   echo '      version: [2.1.0]' \
&&   echo '    libpng:' \
&&   echo '      version: [1.6.37]' \
&&   echo '      variants: +pic' \
&&   echo '    libyaml:' \
&&   echo '      version: [0.2.5]' \
&&   echo '    mapl:' \
&&   echo '      # 2.35.2 goes with esmf@8.4.2, 2.40.3 goes with esmf@8.5.0' \
&&   echo '      # turn off ~pflogger and extdata2g to avoid compilation' \
&&   echo '      # errors with intel@2021.7.0+, see' \
&&   echo '      # https://github.com/JCSDA/spack-stack/issues/769' \
&&   echo '      # also: ... extdata2g segfault UFS?' \
&&   echo '      version: [2.40.3]' \
&&   echo '      variants: +shared +pflogger ~f2py' \
&&   echo '    # If making changes here, also check the Discover site config and the CI workflows' \
&&   echo '    met:' \
&&   echo '      version: [11.1.0]' \
&&   echo '      variants: +python +grib2' \
&&   echo '    metplus:' \
&&   echo '      version: [5.1.0]' \
&&   echo '    metis:' \
&&   echo '      require: +int64 +real64' \
&&   echo '    mpich:' \
&&   echo '      variants: ~hwloc +two_level_namespace' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: mpich@4.1.1' \
&&   echo '        prefix: /opt/mpich-4.1.1' \
&&   echo '      version: [4.1.1]' \
&&   echo '    mysql:' \
&&   echo '      variants: +download_boost' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: mysql@8.0.32' \
&&   echo '        prefix: /usr' \
&&   echo '    nco:' \
&&   echo '      version: [5.0.6]' \
&&   echo '      variants: ~doc' \
&&   echo '    # ncview - when adding information here, also check Orion' \
&&   echo '    # and Discover site configs' \
&&   echo '    nemsio:' \
&&   echo '      version: [2.5.4]' \
&&   echo '    nemsiogfs:' \
&&   echo '      version: [2.5.3]' \
&&   echo '    nccmp:' \
&&   echo '      version: [1.9.0.1]' \
&&   echo '    ncio:' \
&&   echo '      version: [1.1.2]' \
&&   echo '    netcdf-c:' \
&&   echo '      version: [4.9.2]' \
&&   echo '      # If using 4.9.1, turn off byterange variant to fix compile error: ~byterange' \
&&   echo '      variants: +dap +mpi ~parallel-netcdf' \
&&   echo '    netcdf-cxx4:' \
&&   echo '      version: [4.3.1]' \
&&   echo '    netcdf-fortran:' \
&&   echo '      version: [4.6.1]' \
&&   echo '    nlohmann-json:' \
&&   echo '      version: [3.10.5]' \
&&   echo '    nlohmann-json-schema-validator:' \
&&   echo '      version: [2.1.0]' \
&&   echo '    odc:' \
&&   echo '      version: [1.4.6]' \
&&   echo '      variants: ~fortran' \
&&   echo '    openblas:' \
&&   echo '      version: [0.3.24]' \
&&   echo '      variants: +noavx512' \
&&   echo '    openmpi:' \
&&   echo '      variants: +internal-hwloc +two_level_namespace' \
&&   echo '    # Pin openssl to avoid duplicate packages being built' \
&&   echo '    openssl:' \
&&   echo '      variants: +shared' \
&&   echo '    p4est:' \
&&   echo '      version: ['"'"'2.8'"'"']' \
&&   echo '    parallelio:' \
&&   echo '      version: [2.5.10]' \
&&   echo '      variants: +pnetcdf' \
&&   echo '    parallel-netcdf:' \
&&   echo '      version: [1.12.2]' \
&&   echo '    pixman:' \
&&   echo '      variants: +pic' \
&&   echo '    # Do not build pkgconf - https://github.com/jcsda/spack-stack/issues/123' \
&&   echo '    pkgconf:' \
&&   echo '      buildable: false' \
&&   echo '    prod-util:' \
&&   echo '      version: [2.1.1]' \
&&   echo '    proj:' \
&&   echo '      version: [8.1.0]' \
&&   echo '      variants: ~tiff' \
&&   echo '    python:' \
&&   echo '      require: '"'"'@3.10.13'"'"'' \
&&   echo '    py-attrs:' \
&&   echo '      # https://github.com/JCSDA/spack-stack/issues/740' \
&&   echo '      version: [21.4.0]' \
&&   echo '    py-cartopy:' \
&&   echo '      variants: +plotting' \
&&   echo '      require: '"'"'@0.21.1'"'"'' \
&&   echo '    py-cryptography:' \
&&   echo '      variants: +rust_bootstrap' \
&&   echo '    # Introduced in https://github.com/JCSDA/spack-stack/pull/894, pin py-cython' \
&&   echo '    # to avoid duplicate packages being built (cylc dependencies soft-want @3:)' \
&&   echo '    py-cython:' \
&&   echo '      require: '"'"'@0.29.36'"'"'' \
&&   echo '    py-h5py:' \
&&   echo '      version: [3.7.0]' \
&&   echo '      variants: ~mpi' \
&&   echo '    # Comment out for now until build problems are solved' \
&&   echo '    # https://github.com/jcsda/spack-stack/issues/522' \
&&   echo '    # see also ewok-env virtual package and container' \
&&   echo '    # README.md' \
&&   echo '    #py-mysql-connector-python:' \
&&   echo '    #  version: ['"'"'8.0.32'"'"']' \
&&   echo '    py-netcdf4:' \
&&   echo '      version: [1.5.8]' \
&&   echo '      variants: ~mpi' \
&&   echo '    py-numpy:' \
&&   echo '      require: ['"'"'@1.22.3'"'"']' \
&&   echo '    py-pandas:' \
&&   echo '      variants: +excel' \
&&   echo '    # To avoid pip._vendor.pep517.wrappers.BackendInvalid errors with newer' \
&&   echo '    # versions of py-poetry-core when using external/homebrew Python as' \
&&   echo '    # we do at the moment in spack-stack.' \
&&   echo '    # Pin the py-setuptools version to avoid duplicate Python packages' \
&&   echo '    py-setuptools:' \
&&   echo '      require: ['"'"'@63.4.3'"'"']' \
&&   echo '    py-setuptools-rust:' \
&&   echo '      variants: +rust_bootstrap' \
&&   echo '    py-shapely:' \
&&   echo '      require: ['"'"'@1.8.0'"'"']' \
&&   echo '    qt:' \
&&   echo '      version: [5.15.3]' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: qt@5.12.8' \
&&   echo '        prefix: /usr' \
&&   echo '    scotch:' \
&&   echo '      version: [7.0.4]' \
&&   echo '      variants: +mpi+metis~shared~threads~mpi_thread+noarch' \
&&   echo '    sfcio:' \
&&   echo '      version: [1.4.1]' \
&&   echo '    shumlib:' \
&&   echo '      version: [macos_clang_linux_intel_port]' \
&&   echo '    sigio:' \
&&   echo '      version: [2.3.2]' \
&&   echo '    sp:' \
&&   echo '      version: [2.5.0]' \
&&   echo '      variants: precision=4,d,8' \
&&   echo '    udunits:' \
&&   echo '      version: [2.2.28]' \
&&   echo '    upp:' \
&&   echo '      version: [10.0.10]' \
&&   echo '    w3emc:' \
&&   echo '      version: [2.10.0]' \
&&   echo '      variants: precision=4,d,8' \
&&   echo '    w3nco:' \
&&   echo '      version: [2.4.1]' \
&&   echo '    wget:' \
&&   echo '      version: [1.21.2]' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: wget@1.20.3' \
&&   echo '        prefix: /usr' \
&&   echo '    wgrib2:' \
&&   echo '      version: [2.0.8]' \
&&   echo '    wrf-io:' \
&&   echo '      version: [1.2.0]' \
&&   echo '    yafyaml:' \
&&   echo '      version: [0.5.1]' \
&&   echo '    zlib:' \
&&   echo '      version: [1.2.13]' \
&&   echo '    zstd:' \
&&   echo '      version: [1.5.2]' \
&&   echo '      variants: +programs' \
&&   echo '    gcc:' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: gcc@9.4.0' \
&&   echo '        prefix: /usr' \
&&   echo '    gmake:' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: gmake@4.2.1' \
&&   echo '        prefix: /usr' \
&&   echo '    diffutils:' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: diffutils@3.7' \
&&   echo '        prefix: /usr' \
&&   echo '    git:' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: git@2.25.1~tcltk' \
&&   echo '        prefix: /usr' \
&&   echo '    git-lfs:' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: git-lfs@2.9.2' \
&&   echo '        prefix: /usr' \
&&   echo '    llvm:' \
&&   echo '      buildable: false' \
&&   echo '      externals:' \
&&   echo '      - spec: llvm@10.0.0 +clang' \
&&   echo '        prefix: /usr' \
&&   echo '' \
&&   echo '  specs:' \
&&   echo '  - base-env@1.0.0' \
&&   echo '  - jedi-base-env@1.0.0' \
&&   echo '  - ewok-env@1.0.0' \
&&   echo '  - jedi-fv3-env@1.0.0' \
&&   echo '  - jedi-mpas-env@1.0.0' \
&&   echo '  - bacio@2.4.1' \
&&   echo '  - bison@3.8.2' \
&&   echo '  - bufr@12.0.1' \
&&   echo '  - ecbuild@3.7.2' \
&&   echo '  - eccodes@2.32.0' \
&&   echo '  - ecflow@5' \
&&   echo '  - eckit@1.24.5' \
&&   echo '  - ecmwf-atlas@0.35.1 +fckit +trans +tesselation +fftw' \
&&   echo '  - fiat@1.2.0' \
&&   echo '  - ectrans@1.2.0 +fftw' \
&&   echo '  - eigen@3.4.0' \
&&   echo '  - fckit@0.11.0' \
&&   echo '  - fms@release-jcsda' \
&&   echo '  - g2@3.4.5' \
&&   echo '  - g2tmpl@1.10.2' \
&&   echo '  - gftl-shared@1.6.1' \
&&   echo '  - gsibec@1.1.3' \
&&   echo '  - hdf@4.2.15' \
&&   echo '  - hdf5@1.14.0' \
&&   echo '  - ip@4.3.0' \
&&   echo '  - jasper@2.0.32' \
&&   echo '  - jedi-cmake@1.4.0' \
&&   echo '  - libpng@1.6.37' \
&&   echo '  - nccmp@1.9.0.1' \
&&   echo '  - netcdf-c@4.9.2' \
&&   echo '  - netcdf-cxx4@4.3.1' \
&&   echo '  - netcdf-fortran@4.6.1' \
&&   echo '  - nlohmann-json@3.10.5' \
&&   echo '  - nlohmann-json-schema-validator@2.1.0' \
&&   echo '  - parallelio@2.5.10' \
&&   echo '  - parallel-netcdf@1.12.2' \
&&   echo '  - py-eccodes@1.5.0' \
&&   echo '  - py-f90nml@1.4.3' \
&&   echo '  - py-gitpython@3.1.27' \
&&   echo '  - py-h5py@3.7.0' \
&&   echo '  - py-numpy@1.22.3' \
&&   echo '  - py-pandas@1.5.3' \
&&   echo '  - py-pip' \
&&   echo '  - py-pyyaml@6.0' \
&&   echo '  - py-scipy@1.9.3' \
&&   echo '  - py-shapely@1.8.0' \
&&   echo '  - py-xarray@2023.7.0' \
&&   echo '  - sp@2.5.0' \
&&   echo '  - udunits@2.2.28' \
&&   echo '  - w3nco@2.4.1' \
&&   echo '  - w3emc@2.10.0' \
&&   echo '  - nco@5.0.6' \
&&   echo '  - esmf@8.5.0' \
&&   echo '  - mapl@2.40.3' \
&&   echo '  - zlib@1.2.13' \
&&   echo '  - zstd@1.5.2' \
&&   echo '  - odc@1.4.6' \
&&   echo '  - shumlib@macos_clang_linux_intel_port' \
&&   echo '  - awscli-v2@2.13.22' \
&&   echo '  - py-globus-cli@3.16.0' \
&&   echo '' \
&&   echo '  view: /opt/views/view') > /opt/spack-environment/spack.yaml

# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && spack env activate . && spack install --fail-fast && spack gc -y

# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . > activate.sh

# Put output of spack find into a file
RUN cd /opt/spack-environment && \
spack env activate -d . && \
spack find 2>&1 | tee /root/spack_find.out

# Bare OS image to run the installed executables
FROM ubuntu:20.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software

# paths.view is a symlink, so copy the parent to avoid dereferencing and duplicating it
COPY --from=builder /opt/views /opt/views

RUN { \
      echo '#!/bin/sh' \
      && echo '.' /opt/spack-environment/activate.sh \
      && echo 'exec "$@"'; \
    } > /entrypoint.sh \
&& chmod a+x /entrypoint.sh \
&& ln -s /opt/views/view /opt/view

#Set environment variables for installing tzdata
ENV DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC
ENV MPICH_VERSION=4.1.1
ENV CC=clang
ENV CXX=clang++
ENV FC=gfortran
ENV PATH=/opt/mpich-${MPICH_VERSION}/bin:${PATH}
ENV CPATH=/opt/mpich-${MPICH_VERSION}/include:${CPATH}
ENV LD_LIBRARY_PATH=/opt/mpich-${MPICH_VERSION}/lib:${LD_LIBRARY_PATH}

RUN apt-get -yqq update && apt-get -yqq upgrade \
 && apt-get -yqq install bc clang-10 libclang-10-dev libc++-10-dev cpp g++ gcc gfortran git git-lfs make mysql-server qt5-default libqt5svg5-dev qt5dxcb-plugin wget build-essential ca-certificates curl file gnupg2 iproute2 locales python3 python3-pip python3-setuptools unzip vim \
 && rm -rf /var/lib/apt/lists/*

# Create symbolic links for clang compilers
RUN cd /usr/bin && \
ln -svf clang-10 clang && \
ln -svf clang++-10 clang++ && \
ln -svf clang-cpp-10 clang-cpp && \
cd /usr/lib/llvm-10/lib && \
ln -svf libc++abi.so.1.0 libc++abi.so
# Copy spack find output from builder
ENV MPICH_VERSION=4.1.1
COPY --from=builder /root/spack_find.out /root/spack_find.out
# Copy mpich-${MPICH_VERSION} installation from builder
COPY --from=builder /opt/mpich-${MPICH_VERSION} /opt/mpich-${MPICH_VERSION}
# Make a non-root user:nonroot / group:nonroot for running MPI
RUN useradd -U -k /etc/skel -s /bin/bash -d /home/nonroot -m nonroot --uid 43891 && \
echo "ulimit -s unlimited" >> /home/nonroot/.bashrc && \
echo "ulimit -v unlimited" >> /home/nonroot/.bashrc && \
echo "export CC=clang" >> /home/nonroot/.bashrc && \
echo "export CXX=clang++" >> /home/nonroot/.bashrc && \
echo "export FC=gfortran" >> /home/nonroot/.bashrc && \
printf "[credential]\n    helper = cache --timeout=7200\n" >> /home/nonroot/.gitconfig && \
chown -R nonroot:nonroot /home/nonroot/.gitconfig
# Replicate settings for root user
RUN echo "ulimit -s unlimited" >> /root/.bashrc && \
echo "ulimit -v unlimited" >> /root/.bashrc && \
echo "export CC=clang" >> /root/.bashrc && \
echo "export CXX=clang++" >> /root/.bashrc && \
echo "export FC=gfortran" >> /root/.bashrc && \
printf "[credential]\n    helper = cache --timeout=7200\n" >> /root/.gitconfig

LABEL "app"="jedi-ci"
LABEL "mpi"="mpich"
ENTRYPOINT [ "/entrypoint.sh" ]
CMD [ "/bin/bash" ]

Thus, information from the host (spack-stack) gets into the spack container build via "copy & paste" (this is the spack way, not something spack-stack added on top of the spack container methodology).

[climbfuji]

What we could try to do is butcher the container template even more than we do already and force it to also clone spack-stack (but how do we get the current hash in there ... tbd) and run the setup.sh command.

[climbfuji]

This is not going to be easy as far as I can see ...

[AlexanderRichert-NOAA]

Can you check with the spack devs if you haven't already, and if we can't find a solution then revert?

I'm only just familiarizing myself with how the container infrastructure works, but could you add an argument to the docker build call that would pass in $SPACK_STACK_DIR and use a COPY directive to pull in the repos?

[climbfuji]

I can do both. I also looked under the hood, one avenue could be to overwrite spack/lib/spack/spack/container/images.py -> checkout_command and update various other places in the container logic to use spack-stack instead of just spack. But that's so many modifications that the benefits of pulling out the extension are outweighed.

[AlexanderRichert-NOAA]

Yeah I agree any changes to the Spack machinery would need to be part of main Spack otherwise it kind of defeats the purpose... :)

[climbfuji]

@AlexanderRichert-NOAA Any objections merging repos "jcsda-emc" and "jcsda-emc-bundles" into "spack-stack"?

[AlexanderRichert-NOAA]

No I'm good with that, since the metapackages are distinguished by '-env'.

[climbfuji]

No I'm good with that, since the metapackages are distinguished by '-env'.

Great, thanks. I think I have a solution for the containers without any changes in spack itself.