search

ACCESS-NRI / ACCESS-OM3

ACCESS-OM3 repository for spack configuration information
Apache License 2.0
0 stars 0 forks source link

`spack.yaml` Environment Configuration #5

Closed CodeGat closed 1 week ago

CodeGat commented 5 months ago

Initial changes to the spack.yaml to support the access-om3 SBD.

Closes #3

CodeGat commented 5 months ago

@harshula Would we be using a similar spack.modules section (here in the PR: https://github.com/ACCESS-NRI/ACCESS-OM3/pull/5/files#diff-e8582e74fa156f4e5729a850e52b24f2fde2d815c2c9c360f88c4cf90db851abR73-R78) to the ACCESS-NRI/ACCESS-OM2 one (here in the repo: https://github.com/ACCESS-NRI/ACCESS-OM2/blob/main/spack.yaml#L31-L60)

harshula commented 5 months ago

Hi @CodeGat , I think we will need to do the same for access-om3.

github-actions[bot] commented 5 months ago

The model version in the spack.yaml has not been updated. Either update it manually, or comment the following to have it updated and committed automatically:

github-actions[bot] commented 5 months ago

The model version in the spack.yaml has not been updated. Either update it manually, or comment the following to have it updated and committed automatically:

github-actions[bot] commented 5 months ago

This ACCESS-NRI/ACCESS-OM3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 5 months ago

This ACCESS-NRI/ACCESS-OM3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

CodeGat commented 5 months ago

Note for tomorrow: it is deploying to 0.20 prerelease - we want to deploy to our prerelease fork. We might want to have that be a switch. See https://github.com/ACCESS-NRI/build-cd/issues/21 - it's time has finally come!

github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

CodeGat commented 5 months ago

This should now work given the above merged PRs. Now just need to wait for Gadi to be back up so we can get it into Prerelease.

harshula commented 5 months ago

1) We should try to keep the spack.yaml and the Spack Bundle Definition in-sync W.R.T enabling/disabling variants, including compiler flags. Currently, they are out-of-sync. e.g. https://github.com/ACCESS-NRI/spack-packages/blob/main/packages/access-om3/package.py 2) We can achieve (1) by moving the enabling/disabling of variants to the Spack Package Definition and keep them out of the spack.yaml and the Spack Bundle Definition.

harshula commented 5 months ago

Hi @micaeljtoliveira , Can you please summarise here what you told us about the variants during our chat?

micaeljtoliveira commented 5 months ago
github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 5 months ago

This access-om3 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

CodeGat commented 4 months ago

Going to unassign myself from this PR for the moment, since there's nothing further for me to do on it - it is accessible as a prerelease on Gadi. @dougiesquire et. al., have at it!

github-actions[bot] commented 4 months ago

This access-om3 model will be deployed as:

This Prerelease is accessible on Gadi using module use /g/data/vk83/prerelease/modules/access-models/ && module load access-om3/pr5-20, where the binaries shall be on your $PATH. This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om3-pr5-20 environment.

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 2 months ago

:rocket: Deploying access-om3 2024.04.0 as prerelease pr5-22

Details and usage instructions This `access-om3` model will be deployed as: * `2024.04.0` as a Release (when merged). * `pr5-22` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om3/pr5-22 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om3-pr5-22` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 2 months ago

:rocket: Deploying access-om3 2024.04.0 as prerelease pr5-23

Details and usage instructions This `access-om3` model will be deployed as: * `2024.04.0` as a Release (when merged). * `pr5-23` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om3/pr5-23 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om3-pr5-23` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
harshula commented 2 months ago

Builds successfully in a docker container with Spack v0.21 and v0.22.

harshula commented 2 months ago

Found a good issue reproducer on Gadi using Spack v0.22:

$ spack install fms@2023.02 %intel@2021.6.0 target=x86_64

1 error found in build log:
     9     -- Detecting C compile features - done
     10    -- Detecting Fortran compiler ABI info
     11    -- Detecting Fortran compiler ABI info - done
     12    -- Check for working Fortran compiler: /[GDATA]/test-v0.2
           2-fms/spack/lib/spack/env/intel/ifort - skipped
     13    -- Found MPI_C: /apps/openmpi/4.1.5/lib/libmpi.so (found version "3.
           1")
     14    -- Could NOT find MPI_Fortran (missing: MPI_Fortran_F77_HEADER_DIR M
           PI_Fortran_MODULE_DIR) (found version "3.1")
  >> 15    CMake Error at /apps/cmake/3.24.2/share/cmake-3.24/Modules/FindPacka
           geHandleStandardArgs.cmake:230 (message):
     16      Could NOT find MPI (missing: MPI_Fortran_FOUND Fortran) (found ver
           sion
     17      "3.1")
     18    Call Stack (most recent call first):
     19      /apps/cmake/3.24.2/share/cmake-3.24/Modules/FindPackageHandleStand
           ardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
     20      /apps/cmake/3.24.2/share/cmake-3.24/Modules/FindMPI.cmake:1835 (fi
           nd_package_handle_standard_args)
     21      CMakeLists.txt:84 (find_package)
harshula commented 2 months ago

The issue is probably caused by https://github.com/spack/spack/pull/43726

harshula commented 2 months ago

This commit to our Spack fork fixes the issue: https://github.com/ACCESS-NRI/spack/commit/21da7d7e2b5e2680cd9d2e0a2fb4a7d13d8baa9d

CodeGat commented 2 months ago

Sorry about the spam. GitHub Actions is currently having an internal error when running workflows that require environments: https://www.githubstatus.com/incidents/m70hk23gx3nx Will rerun when it has resolved itself

CodeGat commented 2 months ago

Attempting again as they have applied a mitigation.

github-actions[bot] commented 2 months ago

:rocket: Deploying access-om3 2024.04.0 as prerelease pr5-24

Details and usage instructions This `access-om3` model will be deployed as: * `2024.04.0` as a Release (when merged). * `pr5-24` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om3/pr5-24 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om3-pr5-24` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
anton-seaice commented 1 month ago

When I run the build from this it can't find the linked libraries. i.e. the only module installed is 'access-om3' but we need modules for netcdf, parallelio, esmf ... etc (i.e. access-om3-MOM6-CICE6: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory)

It looks like we need to explicitly include modules in our install location (e.g. https://github.com/COSIMA/spack-config/blob/7e86415adc33331fffec86cea615e374c4fa9b29/config/system/modules.yaml#L22)

CodeGat commented 1 month ago

Feel free to make the above modifications as you see fit, by the way! :)

github-actions[bot] commented 1 month ago

:rocket: Deploying access-om3 2024.04.0 as prerelease pr5-25

Details and usage instructions This `access-om3` model will be deployed as: * `2024.04.0` as a Release (when merged). * `pr5-25` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om3/pr5-25 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om3-pr5-25` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
dougiesquire commented 1 month ago

When I run the build from this it can't find the linked libraries. i.e. the only module installed is 'access-om3' but we need modules for netcdf, parallelio, esmf ... etc (i.e. access-om3-MOM6-CICE6: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory)

It looks like we need to explicitly include modules in our install location (e.g. https://github.com/COSIMA/spack-config/blob/7e86415adc33331fffec86cea615e374c4fa9b29/config/system/modules.yaml#L22)

@anton-seaice said in a meeting today that this was actually an issue with the config, not the build. I've been using the environment in this PR to build ACCESS-OM3 for a while now and not had any issues.

anton-seaice commented 1 month ago

When I run the build from this it can't find the linked libraries. i.e. the only module installed is 'access-om3' but we need modules for netcdf, parallelio, esmf ... etc (i.e. access-om3-MOM6-CICE6: error while loading shared libraries: libnetcdff.so.7: cannot open shared object file: No such file or directory) It looks like we need to explicitly include modules in our install location (e.g. https://github.com/COSIMA/spack-config/blob/7e86415adc33331fffec86cea615e374c4fa9b29/config/system/modules.yaml#L22)

@anton-seaice said in a meeting today that this was actually an issue with the config, not the build. I've been using the environment in this PR to build ACCESS-OM3 for a while now and not had any issues.

Oh sorry thats a different issue.

I do think we want to be able to load these modules directly. Loading them directly allows use with say the OM3 build script or using thse library's with a CICE standalone build etc (for development), so I think this or a similar change is worth doing.

dougiesquire commented 1 month ago

I do think we want to be able to load these modules directly. Loading them directly allows use with say the OM3 build script or using thse library's with a CICE standalone build etc (for development), so I think this or a similar change is worth doing.

Okay, would this be a change here though?

anton-seaice commented 1 month ago

Ill take advise on @harshula on that. For example, in COSIMA, they are listed in the "system" modules.yaml

https://github.com/COSIMA/spack-config/blob/7e86415adc33331fffec86cea615e374c4fa9b29/config/system/modules.yaml#L22

dougiesquire commented 1 month ago

Oh sorry thats a different issue.

@anton-seaice, so are you still having issues with using the environment in this PR? Can you provide more details about what you're trying to do and what the issue is? I've been using this environment to build ACCESS-OM3 for a while and not had any issues.

That said, I do think something has gone wrong with the prerelease environment module @CodeGat:

$ module use /g/data/vk83/prerelease/modules/access-models/
$ module load access-om3/pr5-25
$ which access-om3-MOM6-CICE6
/usr/bin/which: no access-om3-MOM6-CICE6 in (/home/599/ds0092/.linuxbrew/bin:/home/599/ds0092/.linuxbrew/sbin:/g/data/tm70/ds0092/software/mambaforge/bin:/g/data/tm70/ds0092/software/mambaforge/condabin:~/.linuxbrew/bin:/home/599/ds0092/.local/bin:/home/599/ds0092/bin:/opt/pbs/default/bin:/opt/nci/bin:/opt/bin:/opt/Modules/v4.3.0/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/opt/pbs/default/bin)

This should return

/g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/access-om3-nuopc-git.0.3.0_0.3.0-hmfbxxbebqygbnq5jqdmrtstxhtta2ov/bin/access-om3-MOM6-CICE6

Perhaps a clue is that the chatbot comment refers to spack 0.21 when we're using 0.22

anton-seaice commented 1 month ago

Yes I used this successfully.

I want to be able to use the dependencies from the environment without using access-om3.

e.g. to build debug / minimal examples to test the dependency:

#!/bin/bash

module purge
module use /g/data/ik11/spack/0.20.1/modules/access-om3/0.2.0/linux-rocky8-x86_64/
module load intel-compiler/2021.6.0 openmpi/4.1.5 parallelio/2.5.10

echo $PIO_ROOT
# ls -l $PIO_ROOT/lib/libpiof.so

mpifort pio_open.f90 $SPACK_PARALLELIO_ROOT/lib/libpiof.so -o pio_open.exe -L $SPACK_PARALLELIO_ROOT/lib -lpioc -I $SPACK_PARALLELIO_ROOT/include -cpp -fpp

Or to use the dependencies to build cice standalone, for testing and development contributions to cice main. e.g.

https://forum.access-hive.org.au/t/running-cice6-standalone/1692?u=anton

I see with the set up of /g/data/vk83/prerelease/modules/access-models/ maybe this isn't possible ?

CodeGat commented 1 month ago

Perhaps a clue is that the https://github.com/ACCESS-NRI/ACCESS-OM3/pull/5#issuecomment-2305886797 refers to spack 0.21 when we're using 0.22

Sorry @dougiesquire this is just an issue with the comment, see https://github.com/ACCESS-NRI/build-cd/issues/110

I want to be able to use the dependencies from the environment without using access-om3.

@anton-seaice Would you be able to add entries of the dependencies to the spack.modules.default.tcl.include section? For each of those dependencies, you would then also need to add entries into the spack.modules.default.tcl.projections section that have the same versions than in the spack.packages.*.require section. Happy to help out with this if you want.

As an aside, I am deleting the old 0.20 Prerelease spack. This contained access-om3-pr5-9, but I see that we are well past that, now :D

dougiesquire commented 1 month ago

Sorry @dougiesquire this is just an issue with the comment, see https://github.com/ACCESS-NRI/build-cd/issues/110

Okay, so any idea why this fails:

$ module use /g/data/vk83/prerelease/modules
$ module load access-om3/pr5-25
$ which access-om3-MOM6-CICE6
/usr/bin/which: no access-om3-MOM6-CICE6 in (blah)
CodeGat commented 1 month ago

Possibly because access-om3-nuopc isn't explicitly included? I'm not 100% sure.

anton-seaice commented 2 weeks ago

@CodeGat

I added this

tcl:
        hash_length: 0
        include:
          - access-om3
          - access-om3-nuopc
          - netcdf-c
          - netcdf-fortran
          - parallelio
          - esmf
          - fms
          - fortranxml

and did a spack install. However I still can't load the access-om3-nuopc module.

$ module use  /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0
$ module avail
----------------------------------------------------------------------- /g/data/tm70/as2285/spack0.22_sep/release/modules/linux-rocky8-x86_64 ------------------------------------------------------------------------
access-om3-nuopc/git.0.3.0_0.3.0  access-om3/2024.04.0  esmf/8.5.0  fms/2023.02  netcdf-c/4.9.2  netcdf-fortran/4.6.1  parallelio/2.6.2  

...
$ module load access-om3-nuopc/git.0.3.0_0.3.0
Loading access-om3-nuopc/git.0.3.0_0.3.0
  ERROR: Unable to locate a modulefile for 'fortranxml/4.1.2'
  ERROR: Load of requirement fortranxml/4.1.2 failed
aidanheerdegen commented 2 weeks ago 
$ module use  /g/data/tm70/as2285/spack0.22/release/linux-rocky8-x86_64/intel-2021.10.0
$ module avail

The permissions on your spack directory don't allow group access (executable bit is not set correctly):

$ ls -ld /g/data/tm70/as2285/spack0.22
drwxrwSr-- 10 as2285 tm70 4096 Sep 16 15:03 /g/data/tm70/as2285/spack0.22

Having said which, why do you want to make the fortranxml library accessible through module load?

anton-seaice commented 2 weeks ago

a. At the moment, it doesn't allow module loading the model i.e. https://github.com/ACCESS-NRI/ACCESS-OM3/pull/5#issuecomment-2313855031 - hence adding access-om3-nuopc to the tcl: include

b. Having the dependencies avail through module load is useful for development work. i.e. https://github.com/ACCESS-NRI/ACCESS-OM3/pull/5#issuecomment-2306113554

c. So dropping fortranxml from the tcl: include might allow the module load of access-om3-nuopc ? EDIT: doing a spack conretise --fresh fixed the problem with module load access-om3-nuopc

anton-seaice commented 2 weeks ago

I added this

tcl:
        hash_length: 0
        include:
          - access-om3
          - access-om3-nuopc
          - netcdf-c
          - netcdf-fortran
          - parallelio
          - esmf
          - fms
          - fortranxml

I did a spack concretise --fresh and spack install and the new install allowed a

module load access-om3-nuopc/git...

i.e. it works now

So I think this change needs to be made. Can I push it? How do you want to name the versions of these modules ?

Most of the default names seem ok?

----------------------------------------- /g/data/tm70/as2285/spack0.22_sep/release/modules/linux-rocky8-x86_64 -----------------------------------------
access-om3-nuopc/git.0.3.0_0.3.0                                            esmf/8.5.0        netcdf-c/4.9.2        
access-om3-nuopc/git.96f91599f746b4866bbd3f05ee6eb192ba70d391_0.3.0-git.15  fms/2023.02       netcdf-fortran/4.6.1  
access-om3/2024.04.0                                                        fortranxml/4.1.2  parallelio/2.6.2
CodeGat commented 2 weeks ago

Hey @anton-seaice I've seen the above checks fail. For Check spack.yaml (https://github.com/ACCESS-NRI/ACCESS-OM3/actions/runs/10953784507/job/30414616058?pr=5) we require an entry of access-om3-nuopc: '0.3.0' in the projections section, near the access-om3 one. As for the other one CI / CI / Check Config Fields - that's a infrastructure bug and I'll get onto it now.

github-actions[bot] commented 2 weeks ago

:rocket: Deploying access-om3 2024.04.0 as prerelease pr5-27

Details and usage instructions This `access-om3` model will be deployed as: * `2024.04.0` as a Release (when merged). * `pr5-27` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules module load access-om3/pr5-27 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.22/spack` in the `access-om3-pr5-27` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 1 week ago

:rocket: Deploying access-om3 2024.04.0 as prerelease pr5-28

Details and usage instructions This `access-om3` model will be deployed as: * `2024.04.0` as a Release (when merged). * `pr5-28` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules module load access-om3/pr5-28 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.22/spack` in the `access-om3-pr5-28` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 1 week ago

:rocket: Deploying access-om3 2024.09.0 as prerelease pr5-29

Details and usage instructions This `access-om3` model will be deployed as: * `2024.09.0` as a Release (when merged). * `pr5-29` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules module load access-om3/pr5-29 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.22/spack` in the `access-om3-pr5-29` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.