search

ACCESS-NRI / ACCESS-OM2

ACCESS-OM2: ACCESS Ocean-Sea Ice Model
Apache License 2.0
5 stars 0 forks source link

`ACCESS-OM2` Dependencies Update #60

Closed CodeGat closed 2 weeks ago

CodeGat commented 2 months ago

Updated compiler to intel@2021.6.0, updated package versions

Related issue https://github.com/ACCESS-NRI/ACCESS-OM2/issues/61 (doesn't close until compiler is also updated to OneAPI version, ifx)

github-actions[bot] commented 2 months ago

This access-om2 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 2 months ago

This access-om2 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 2 months ago

This access-om2 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 2 months ago

This access-om2 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

harshula commented 2 months ago

Dependencies: https://github.com/ACCESS-NRI/spack-config/pull/28 https://github.com/ACCESS-NRI/spack-config/pull/29

harshula commented 2 months ago

Hi @CodeGat , Should we also use Spack v0.21 instead of v0.20?

CodeGat commented 2 months ago

It is using that version - if you have access, check the vars in the Gadi Prerelease environment setting - they're pointing to the 0.21 prerelease spack on Gadi

aidanheerdegen commented 2 months ago

Am I correct, updating all those dependencies didn't change bit reproducibility?

CodeGat commented 2 months ago

I don't know if we'd tested the bit repro yet?

aidanheerdegen commented 2 months ago

I don't know if we'd tested the bit repro yet?

Yeah that was a mistake on my part.

I've created an issue for that

https://github.com/ACCESS-NRI/access-om2-configs/issues/107

I can make a PR to test it, or if you want to use this as a test for an automated PR workflow that would also be cool @CodeGat

CodeGat commented 2 months ago

That workflow might take a bit of thonking to figure out, so I'm happy for this one to be manual!

github-actions[bot] commented 2 months ago

This access-om2 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

CodeGat commented 2 months ago

Gadi is currently feeling sluggish and is failing to deploy our things. Will wait until it is better, then try again

github-actions[bot] commented 2 months ago

This access-om2 model will be deployed with the following versions:

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

CodeGat commented 2 months ago

Going to unassign myself from this PR for the moment, since I believe there is a bit of testing to be done. Feel free to reassign as needed if we end up needing to merge/change things, etc.

github-actions[bot] commented 1 month ago

This access-om2 model will be deployed as:

This Prerelease is accessible on Gadi using module use /g/data/vk83/prerelease/modules/access-models/ && module load access-om2/pr60-9, where the binaries shall be on your $PATH. This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr60-9 environment.

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

dougiesquire commented 1 month ago

@aidanheerdegen, some very basic testing indicates that there is no impact to performance from these updates.

In the table below:

I ran three years of each. The experiments were run at the same time. Times in the table are from the PAYU_WALLTIME field in job.yaml.

  Exp 1 Exp 2
output000 824.89 s 824.03 s
output001 838.87 s 841.34 s
output002 824.80 s 829.25 s
average 829.52 s 831.54
aidanheerdegen commented 1 month ago

Some very basic testing indicates that there is no impact to performance from these updates

Thanks @dougiesquire

aidanheerdegen commented 1 month ago

I have manually run the repro tests with the release-1deg_jra55_ryf branch, using the binaries from this PR:

$ git diff HEAD^^ manifests/exe.yaml
diff --git a/manifests/exe.yaml b/manifests/exe.yaml
index 0aeb8c6..80ed3f5 100644
--- a/manifests/exe.yaml
+++ b/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ieiy3e7hidn4dzaqly3ly2yu45mecgq4/bin/yatm.exe
+  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
   hashes:
-    binhash: 61890159678b68a1699cb989d2c2864e
-    md5: 7bbd1014eb48b16406877d1b93ae1a47
+    binhash: 4e8b4ef76e971c4af3b26cfac632e160
+    md5: 5baa1d417fe6708fc30cbeaa57d82f96
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-rh3xfkrgajya3ghtliacuhlx3pgvrzqs/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 6bff005e04c23c579f37b7b2c0189793
-    md5: 5e7c7ba864da95cd1329d098f1e47776
+    binhash: 3a65f67d21152e77034da28c22a94c66
+    md5: 37866455b057c85c3ea50c0ef0ea840b
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-ewcdbrfukblyjxpkhd3mfkj4yxfolal4/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: c5c24bc2ab515b08e47d6d3ab2b54cc6
-    md5: 2432c23048525b384fd57e8cd2ed7e3b
+    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
+    md5: a3f10baeadb88e813b4a8121f61a6226
$ model-config-tests -m checksum
============================================================ test session starts =============================================================
platform linux -- Python 3.11.7, pytest-8.2.1, pluggy-1.5.0                            
rootdir: /g/data/tm70/aph502/envs/model-config-tests/lib/python3.11/site-packages/model_config_tests
collected 39 items / 37 deselected / 2 selected                                                                                              

../../../../../g/data/tm70/aph502/envs/model-config-tests/lib/python3.11/site-packages/model_config_tests/test_bit_reproducibility.py . [ 50%]
.                                                                                                                                      [100%]

================================================ 2 passed, 37 deselected in 685.56s (0:11:25) ================================================
Details of python `venv` used for testing

``` $ pip list Package Version ------------------------- ----------- attrs 23.2.0 certifi 2024.6.2 cftime 1.6.3 charset-normalizer 3.3.2 f90nml 1.4.4 gitdb 4.0.11 GitPython 3.1.43 idna 3.7 iniconfig 2.0.0 jsonschema 4.22.0 jsonschema-specifications 2023.12.1 model_config_tests 0.0.1 nchash 0.1.5 netCDF4 1.6.5 numpy 1.26.4 packaging 24.0 payu 1.1.3 pip 23.2.1 pluggy 1.5.0 pytest 8.2.1 python-dateutil 2.9.0.post0 PyYAML 6.0.1 referencing 0.35.1 requests 2.32.3 rpds-py 0.18.1 ruamel.yaml 0.18.6 ruamel.yaml.clib 0.2.8 setuptools 65.5.0 six 1.16.0 smmap 5.0.1 tenacity 8.3.0 urllib3 2.2.1 ```

Based on the reproducibility and performance testing, I think this should be merged.

github-actions[bot] commented 1 month ago

This access-om2 model will be deployed as:

This Prerelease is accessible on Gadi using module use /g/data/vk83/prerelease/modules/access-models/ && module load access-om2/pr60-10, where the binaries shall be on your $PATH. This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr60-10 environment.

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

aidanheerdegen commented 1 month ago

We probably want to update to 2024.06.0 if that is when we're merging the PR.

Or ... we could use 2023.03.1, as we're not updating any model components, and we've since established it is bit repro and doesn't affect performance.

Sorry! And I'm not particularly sold on one over the other.

CodeGat commented 1 month ago

@aidanheerdegen Good idea. I'll use 2024.03.1 instead.

github-actions[bot] commented 1 month ago

This access-om2 model will be deployed as:

This Prerelease is accessible on Gadi using module use /g/data/vk83/prerelease/modules/access-models/ && module load access-om2/pr60-11, where the binaries shall be on your $PATH. This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr60-11 environment.

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

CodeGat commented 1 month ago

The above commit is testing out https://github.com/ACCESS-NRI/ACCESS-OM2/pull/60#discussion_r1630569602 - it can be removed if it is not needed, and we can go back to https://github.com/ACCESS-NRI/ACCESS-OM2/pull/60/commits/4ed4c7bbd51931478851de6ff0f0ac5124f6b3d3

github-actions[bot] commented 1 month ago

This access-om2 model will be deployed as:

This Prerelease is accessible on Gadi using module use /g/data/vk83/prerelease/modules/access-models/ && module load access-om2/pr60-12, where the binaries shall be on your $PATH. This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr60-12 environment.

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

github-actions[bot] commented 1 month ago

This access-om2 model will be deployed as:

This Prerelease is accessible on Gadi using module use /g/data/vk83/prerelease/modules/access-models/ && module load access-om2/pr60-13, where the binaries shall be on your $PATH. This Prerelease is also accessible on Gadi via /g/data/vk83/prerelease/apps/spack/0.21/spack in the access-om2-pr60-13 environment.

It will be deployed using:

If this is not what was expected, commit changes to config/versions.json.

CodeGat commented 2 weeks ago

Cleaned up the environment for access-om2-pr60-* with https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/9673293454/job/26686980742