JCSDA / spack-stack

Creative Commons Zero v1.0 Universal
27 stars 44 forks source link

Improve workflow and/or documentation when using own machine #144

Closed climbfuji closed 2 years ago

climbfuji commented 2 years ago

I am still not sure how to tell users (they were all macOS users) to adapt their default site configs to work.

I had to walk all testers through this:

  1. Look in macOS reference site config README.md and install brew packages, configure Python
  2. Use top-level README.md: a. Create default site config b. Run spack external find ... and spack compiler find ...
  3. Stop cruising through the top-level README.md. Use a graphical diff tool to compare to the macos reference site config to figure out which packages to remove (for example, macOS native sqlite which doesn't have the dev. headers, but there are more) and to add (e.g. qt5 - this could be improved by updating the qt package.py), set packages:all::compilers and packages:all::mpi, ...
    • Essentially the same as what we do in the CI scripts
  4. Back to top-level README.md: concretize, install, create modules, create meta modules

Documentation is still missing:

kgerheiser commented 2 years ago

On a local machine, I think the steps should be spack compiler find, spack external find. Leave it at the default of only finding build tools and maybe some specific externals. Several of the workarounds in the CI are external packages on macOS (sqlite, m4, and some more). Then, set compiler/mpi. Should also have the ability to build MPI with Spack. Meta-module generation is disabled for internal MPI. I'm n

Adding packages: spack add <spec> or manually edit spack.yaml to include a spec. Talk about spec syntax and default variants/versions.

Chaining environments by setting upstreams. Provide pre-configured environments HPC systems that have upstreams configured?

Perhaps related to putting the documentation into Read the Docs instead of spread across several READMEs?

climbfuji commented 2 years ago

On a local machine, I think the steps should be spack compiler find, spack external find. Leave it at the default of only finding build tools and maybe some specific externals. Several of the workarounds in the CI are external packages on macOS (sqlite, m4, and some more). Then, set compiler/mpi. Should also have the ability to build MPI with Spack. Meta-module generation is disabled for internal MPI. I'm n

Meta modules are now generated in the macOS reference config, because I configured both mpich and openmpi to build correctly with two-level namespaces. Solves several problems (not only the namespaces) with the brew-built versions. Therefore the lmod whitelist in the macos reference site config contains openmpi and mpich.

External packages are tricky sometimes, because building some of them is either super complicated (qt) or doesn't work, others need to be built because the native versions are incomplete.

Adding packages: spack add <spec> or manually edit spack.yaml to include a spec. Talk about spec syntax and default variants/versions.

Chaining environments by setting upstreams. Provide pre-configured environments HPC systems that have upstreams configured?

Perhaps related to putting the documentation into Read the Docs instead of spread across several READMEs?

Making progress on this end.

climbfuji commented 2 years ago

I am working on this. Regarding upstreams, it looks like from the spack documentation (and my initial testing) that this only works if the installation is in the default location (see https://spack.readthedocs.io/en/latest/chain.html):

install_tree must point to the opt/spack directory inside of the Spack base directory

If so, that would be a bummer. I have:

$ spack config get upstreams
upstreams:
  spack-v0-0-1:
    install_tree: /work2/06146/tg854455/stampede2/spack-stack/spack-stack-0.0.1/envs/jedi-ufs-intel-2022.1.0/install
$ spack find
==> In environment /work2/06146/tg854455/stampede2/spack-stack/spack-stack-0.0.1/envs/jedi-ufs-intel-2022.1.0-test-esmf830b09
==> Root specs
ufs-weather-model-debug-env  ufs-weather-model-env

==> 0 installed packages

but the original environment has a ton of packages installed:

$ ls -l /work2/06146/tg854455/stampede2/spack-stack/spack-stack-0.0.1/envs/jedi-ufs-intel-2022.1.0/install/intel/2022.1.0/
total 636
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:36 bacio-2.4.1-2dbr5sy
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 09:45 base-env-main-o7rcuxr
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 16:47 base-env-main-xpuxzbm
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 11:01 boost-1.72.0-ynsvjdj
drwxr-sr-x  8 tg854455 G-81589 4096 May 17 18:12 bufr-11.6.0-57mww6w
drwxr-sr-x  6 tg854455 G-81589 4096 May 16 17:44 cmake-3.22.1-isrzon4
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:07 cmake-3.22.1-q64ygie
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:37 crtm-2.3.0-3wbyloq
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 12:36 curl-7.82.0-6tkrek7
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:36 ecbuild-3.6.1-w2puo5f
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:47 eccodes-2.21.0-xysgjeu
drwxr-sr-x  8 tg854455 G-81589 4096 May 18 12:09 ecflow-5.8.3-ynxpw4u
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:52 eckit-1.19.0-7vyfbcr
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 10:23 ecmwf-atlas-0.29.0-ifeu3w6
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 09:40 ectrans-develop-ppbvwap
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:28 eigen-3.4.0-veb3etp
drwxr-sr-x  7 tg854455 G-81589 4096 May 19 08:41 esmf-8.2.0-juzu4k5
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 18:05 esmf-8.2.0-khvgj6n
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 13:15 esmf-8.2.0-yvuex64
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:25 expat-2.4.6-7lit3mp
drwxr-sr-x  8 tg854455 G-81589 4096 May 17 13:03 fckit-0.9.5-rtcoguz
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 09:49 fftw-3.3.10-6hyfhvc
drwxr-sr-x  8 tg854455 G-81589 4096 May 17 17:35 fiat-develop-bnad33v
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 17:40 fms-2022.01-b7iitlv
drwxr-sr-x  6 tg854455 G-81589 4096 May 18 13:19 fms-2022.01-ki2yoad
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 18:05 fms-jcsda-release-stable-z2lsqyr
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 11:48 freetype-2.11.1-wkyckna
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:58 g2-3.4.5-y2f5ugn
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:28 g2tmpl-1.10.0-cyd2hr7
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 16:45 gdal-3.4.2-7qf4lak
drwxr-sr-x  6 tg854455 G-81589 4096 May 18 11:54 geos-3.9.1-yfmejt5
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:30 gftl-shared-1.3.0-z2rqu4c
drwxr-sr-x  6 tg854455 G-81589 4096 May 18 12:44 git-2.35.1-krvctc4
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:39 git-2.35.1-r572cyf
drwxr-sr-x 12 tg854455 G-81589 4096 May 17 12:54 go-bootstrap-1.4-bootstrap-20171003-kaomoby
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 11:52 googletest-1.10.0-naumjll
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:35 gsl-lite-0.37.0-6gkaybx
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:42 hdf-4.2.15-pjxcfme
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:35 hdf5-1.12.1-x54sfwv
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:42 ip-3.3.3-badgq35
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:42 jasper-2.0.32-svevup7
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 16:53 jedi-base-env-main-65z2oh2
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 10:24 jedi-base-env-main-ih7mk5k
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:36 jedi-cmake-1.3.0-jql2wou
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 16:53 jedi-ewok-env-main-cedbutq
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 18:05 jedi-fv3-bundle-env-main-krgmetl
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 16:53 jedi-ufs-bundle-env-main-wrbj2kl
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 10:24 jedi-ufs-bundle-env-main-yqxyvsq
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 18:05 jedi-um-bundle-env-main-e4i2rxf
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 11:51 json-c-0.15-se5pcqm
drwxr-sr-x  8 tg854455 G-81589 4096 May 17 12:36 libaec-1.0.5-5ycxngi
drwxr-sr-x  6 tg854455 G-81589 4096 May 16 17:46 libbsd-0.11.5-olt5n4z
drwxr-sr-x  6 tg854455 G-81589 4096 May 16 17:21 libffi-3.4.2-iiygodo
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 12:49 libgeotiff-1.6.0-lgqbv4d
drwxr-sr-x  7 tg854455 G-81589 4096 May 16 17:18 libiconv-1.16-fsc26nl
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:26 libidn2-2.3.0-eo3fgtu
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:29 libjpeg-turbo-2.1.0-cmhbxlc
drwxr-sr-x  6 tg854455 G-81589 4096 May 16 17:16 libmd-1.0.4-zycaukh
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 09:31 libpng-1.6.37-xbbldy4
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 11:44 libtiff-4.3.0-fruuyc5
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 11:05 libtirpc-1.2.6-eqmoazt
drwxr-sr-x  6 tg854455 G-81589 4096 May 16 17:50 libunistring-0.9.10-yvjzu6c
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 11:06 libxml2-2.9.12-oqfu5hu
drwxr-sr-x  5 tg854455 G-81589 4096 May 16 17:21 libyaml-0.2.5-6lkjojs
drwxr-sr-x  7 tg854455 G-81589 4096 May 16 17:19 lz4-1.9.3-sfebfcs
drwxr-sr-x  9 tg854455 G-81589 4096 May 19 08:48 mapl-2.8.1-42mszt7
drwxr-sr-x  9 tg854455 G-81589 4096 May 18 09:45 mapl-2.8.1-4uvbbxs
drwxr-sr-x  9 tg854455 G-81589 4096 May 18 16:52 mapl-2.8.1-o5tk3s4
drwxr-sr-x  5 tg854455 G-81589 4096 May 16 17:20 nasm-2.15.05-tsh6ukv
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 12:49 nccmp-1.9.0.1-hxzq2zn
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 13:01 nccmp-1.9.0.1-vp34uhb
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:45 netcdf-c-4.8.1-lxb3ujc
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 12:39 netcdf-c-4.8.1-rchd5ia
drwxr-sr-x  6 tg854455 G-81589 4096 May 18 12:45 netcdf-cxx4-4.3.1-3huipcv
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 13:02 netcdf-cxx4-4.3.1-rj6qxwt
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:59 netcdf-fortran-4.5.4-shhfiog
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 12:47 netcdf-fortran-4.5.4-tk3tlm2
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 10:07 ninja-1.10.2-myeosqx
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:28 nlohmann-json-3.10.5-kzbjmmj
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:41 nlohmann-json-schema-validator-2.1.0-s4ybwwx
drwxr-sr-x  7 tg854455 G-81589 4096 May 16 17:51 numactl-2.0.14-jqxs7pa
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 09:30 openblas-0.3.19-l475jk5
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:35 openjpeg-2.3.1-6czppv2
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 10:07 openssl-1.1.1n-w5xn57z
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 17:36 parallelio-2.5.4-mii2ncv
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 12:51 parallelio-2.5.4-ub64r2p
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 09:39 parallel-netcdf-1.12.2-kec2xp2
drwxr-sr-x  7 tg854455 G-81589 4096 May 16 17:18 pcre2-10.39-ysgd2lk
drwxr-sr-x  7 tg854455 G-81589 4096 May 16 17:19 pkgconf-1.8.0-ylt32zw
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 12:43 proj-7.2.1-7izgvkr
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:54 py-attrs-21.4.0-o4ulypn
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 17:30 py-beniget-0.4.1-ry2miib
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 16:45 py-boto3-1.18.12-i42lqnk
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 12:09 py-botocore-1.21.12-jpvvlka
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 18:09 py-bottleneck-1.3.2-dszyzum
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 16:53 py-cartopy-0.18.0-vy7b4lv
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:48 py-certifi-2021.10.8-6xrn265
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 17:30 py-cffi-1.15.0-rzu5pt6
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 18:09 py-cftime-1.0.3.4-hcraoih
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:50 py-cppy-1.1.0-omv6ly7
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:50 py-cycler-0.11.0-engakeh
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:57 py-cython-0.29.24-g5l3idu
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 18:09 py-eccodes-1.3.2-vxebxxf
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:54 py-findlibs-0.0.2-xsmfaxp
drwxr-sr-x  6 tg854455 G-81589 4096 May 18 11:48 py-fonttools-4.29.1-qmsoqzu
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:54 py-gast-0.5.3-5uyzjjt
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 18:05 py-h5py-3.6.0-ihxsrqw
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 12:10 py-jinja2-3.0.3-uwbml4e
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 11:49 py-jmespath-0.10.0-fowke5g
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 12:10 py-kiwisolver-1.3.2-vsovtwe
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:49 py-markupsafe-2.0.1-ld7c3c7
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 16:46 py-matplotlib-3.5.1-tet7rhn
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:54 py-mpi4py-3.1.2-h2u6cb5
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 09:57 py-netcdf4-1.5.3-4dkp6u7
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 12:50 py-netcdf4-1.5.3-oopw5tk
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 18:05 py-numexpr-2.7.3-ii74dxl
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 17:34 py-numpy-1.22.3-rre6xtt
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 17:29 py-packaging-21.3-d7g4pij
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 10:11 py-pandas-1.4.0-5c2ykil
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:49 py-pillow-9.0.0-7u2tj2k
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 10:07 py-pip-21.3.1-ifxlsg2
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:57 py-pkgconfig-1.5.5-x6l2qhl
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:54 py-ply-3.11-olqhtev
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:40 py-poetry-core-1.0.7-rqv3suc
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:56 py-pybind11-2.8.1-g36uzvg
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:54 py-pycodestyle-2.8.0-7natrmo
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:56 py-pycparser-2.20-jqwhbtk
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 09:37 py-pyhdf-0.10.4-brnogdd
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:56 py-pyparsing-3.0.6-yjrs6ud
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:48 py-pyshp-2.1.0-foczifo
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 09:38 py-python-dateutil-2.8.2-q5kjp4d
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 18:09 py-pythran-0.11.0-ruvyo23
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:54 py-pytz-2021.3-hrxkaue
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 17:34 py-pyyaml-6.0-dni5bjd
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:49 py-ruamel-yaml-0.17.16-vrmddox
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:49 py-ruamel-yaml-clib-0.2.0-r4tsrhb
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 12:50 py-s3transfer-0.5.0-fvnvrwj
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 09:57 py-scipy-1.8.0-lxoz5ju
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:40 py-setuptools-57.4.0-hqajveo
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 17:34 py-setuptools-scm-6.3.2-lv7z3pp
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 12:49 py-setuptools-scm-git-archive-1.1-jbqycoq
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 12:44 py-shapely-1.8.0-bvehfxb
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:56 py-six-1.16.0-7wmed7n
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:40 py-tomli-1.2.2-yyxubj2
drwxr-sr-x  4 tg854455 G-81589 4096 May 18 11:48 py-urllib3-1.26.6-l6aztsd
drwxr-sr-x  5 tg854455 G-81589 4096 May 17 12:26 py-wheel-0.37.0-gayr3kh
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 11:51 qhull-2020.2-yscmyc3
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 11:45 readline-8.1-mv33gsw
drwxr-sr-x  5 tg854455 G-81589 4096 May 18 18:01 shumlib-macos_clang_port-lgq4i4e
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:29 sp-2.3.3-mnowe65
drwxr-sr-x  7 tg854455 G-81589 4096 May 18 11:47 sqlite-3.37.2-ep4yc7k
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:38 udunits-2.2.28-w2wy76m
drwxr-sr-x  3 tg854455 G-81589 4096 May 19 08:48 ufs-weather-model-debug-env-main-4cz6nbg
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 10:23 ufs-weather-model-env-main-52mnj4b
drwxr-sr-x  3 tg854455 G-81589 4096 May 18 16:53 ufs-weather-model-env-main-hrvtk4v
drwxr-sr-x  7 tg854455 G-81589 4096 May 17 12:36 w3nco-2.4.1-jt3mv7m
drwxr-sr-x  6 tg854455 G-81589 4096 May 17 12:24 wget-1.21.2-a4d2wwo
drwxr-sr-x  4 tg854455 G-81589 4096 May 17 12:42 yafyaml-0.5.1-xk2qslo
drwxr-sr-x  6 tg854455 G-81589 4096 May 16 17:19 zlib-1.2.12-xys4fru
kgerheiser commented 2 years ago

I asked the Slack channel and upstreams are allowed to be anywhere, it's just poor wording. I know I've used upstreams before.

climbfuji commented 2 years ago

I asked the Slack channel and upstreams are allowed to be anywhere, it's just poor wording. I know I've used upstreams before.

Wondering what went wrong in my case ...

kgerheiser commented 2 years ago

Did you attempt to install packages in the downstream env? I don't think it'll list them as installed when you run spack find. Anyway, I was going to give it a go and try it out myself.

climbfuji commented 2 years ago

Did you attempt to install packages in the downstream env? I don't think it'll list them as installed when you run spack find. Anyway, I was going to give it a go and try it out myself.

Hmm, this is from the spack documentation, maybe it's wrong, too. I didn't try to install anything yet.

Once the upstream Spack instance has been added, spack find will automatically check the upstream instance when querying installed packages, and new package installations for the local Spack install will use any dependencies that are installed in the upstream instance.

Will try this afternoon.