search

ACCESS-NRI / ACCESS-OM2

ACCESS-OM2: ACCESS Ocean-Sea Ice Model
Apache License 2.0
5 stars 0 forks source link

`hdf5` Bug Avoidance #75

Open CodeGat opened 3 months ago

CodeGat commented 3 months ago

This PR principally updates the spack-config used to deploy the model, so we avoid the buggy hdf5 1.13.3 that caused 2023.03.1 to be unreleased. The spack-packages repo was also updated to keep it relatively up to date, although the changes shouldn't have much of a bearing on the deployment.

References https://github.com/ACCESS-NRI/ACCESS-OM2/pull/73 Closes #74

github-actions[bot] commented 3 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-3

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-3` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-3 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-3` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.03, spack-config 2024.07.02

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.03`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.03) * `access-nri/spack-config` version [`2024.07.02`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.02) If this is not what was expected, commit changes to `config/versions.json`.
CodeGat commented 3 months ago

This current build is using hdf5 1.14.2 - https://github.com/ACCESS-NRI/ACCESS-OM2/actions/runs/9774408940/job/26982640737?pr=75#step:8:366

dougiesquire commented 3 months ago

Thanks @CodeGat!

Some very basic testing indicates that the prerelease runs successfully but there is an impact on performance from these updates 😞

In the table below:

I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

  Exp 1 Exp 2
output000 820.90 s 841.14 s
output001 830.14 s 850.29 s
output002 819.19 s 868.89 s
average 823.41 s 853.44

Below is the (selective) output of a diff between the two experiments, confirming that (this time) the executables were indeed changed in Exp 2 to the prerelease versions:

$ git diff -- om2_1deg_jra55_ryf_pr75-orig om2_1deg_jra55_ryf_pr75-3
diff --git a/om2_1deg_jra55_ryf_pr75-orig/config.yaml b/om2_1deg_jra55_ryf_pr75-3/config.yaml
index de774f2..3c38ae4 100644
--- a/om2_1deg_jra55_ryf_pr75-orig/config.yaml
+++ b/om2_1deg_jra55_ryf_pr75-3/config.yaml
@@ -30,7 +30,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
       input:
             - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
             - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -38,7 +38,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
@@ -57,7 +57,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
       input:
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.05.30/grid.nc
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.10.22/kmt.nc
@@ -74,7 +74,7 @@ collate:
   mem: 30GB
   ncpus: 4
   queue: normal
-  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/mppnccombine.spack
+  exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/mppnccombine.spack

 # Misc
 runlog: false
diff --git a/om2_1deg_jra55_ryf_pr75-orig/manifests/exe.yaml b/om2_1deg_jra55_ryf_pr75-3/manifests/exe.yaml
index 80ed3f5..0d9571d 100644
--- a/om2_1deg_jra55_ryf_pr75-orig/manifests/exe.yaml
+++ b/om2_1deg_jra55_ryf_pr75-3/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: b21a1ef267acae687c3c4cbc0ae51a8e
+    md5: f434bd5b517481ae27b4c8f47f95b94d
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 3a65f67d21152e77034da28c22a94c66
-    md5: 37866455b057c85c3ea50c0ef0ea840b
+    binhash: 6f048474c92f2b2cc4ce14d159d9c3ad
+    md5: f74eb0ac0ca781c62d45c3a5e3f2fde7
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
-    md5: a3f10baeadb88e813b4a8121f61a6226
+    binhash: f4eaa918e5e24afa8050d4a5c9ae2382
+    md5: df8bec1f690c24cdfbeced2481352c38
aidanheerdegen commented 3 months ago

This is roughly a 4% degradation in performance. My supposition is that it's the parallelio, as this showed a degradation in performance.

Options at this point:

  1. Live with the change
  2. Get more data (I'm not too concerned about 1 degree, but if it has a big impact on 0.1 that could be expensive)
  3. Try reverting parallelio and see if that fixes it
  4. Don't bother with this right now, and in the BGC case just add the bug fixes
dougiesquire commented 3 months ago

I also tested a 0.1 deg configuration. In the table below:

I ran three months of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

  Exp 1 Exp 2
output000 4946.68 s 5040.16 s
output001 4517.40 s 4630.98 s
output002 4991.54 s 5098.09 s
average 4818.54 s 4923.07 s

So Exp 2 looks ~2-3% slower...

Below is the (selective) output of a diff between the two experiments:

$ git diff -- om2_01deg_jra55_ryf_pr75-orig om2_01deg_jra55_ryf_pr75-3
diff --git a/om2_01deg_jra55_ryf_pr75-orig/config.yaml b/om2_01deg_jra55_ryf_pr75-3/config.yaml
index f8c3249..6da9ffd 100644
--- a/om2_01deg_jra55_ryf_pr75-orig/config.yaml
+++ b/om2_01deg_jra55_ryf_pr75-3/config.yaml
@@ -28,7 +28,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
       input:
             - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.01deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
             - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -36,7 +36,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.01deg/2020.05.30/grid_spec.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.01deg/2020.05.30/ocean_hgrid.nc
@@ -55,7 +55,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_3600x2700_90x90_722p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_3600x2700_90x90_722p.exe
       input:
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.01deg/2020.05.30/grid.nc
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.01deg/2020.05.30/kmt.nc
diff --git a/om2_01deg_jra55_ryf_pr75-orig/manifests/exe.yaml b/om2_01deg_jra55_ryf_pr75-3/manifests/exe.yaml
index fe1c2e5..3e3de46 100644
--- a/om2_01deg_jra55_ryf_pr75-orig/manifests/exe.yaml
+++ b/om2_01deg_jra55_ryf_pr75-3/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: b21a1ef267acae687c3c4cbc0ae51a8e
+    md5: f434bd5b517481ae27b4c8f47f95b94d
 work/ice/cice_auscom_3600x2700_90x90_722p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_3600x2700_90x90_722p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-itelnfmk22iwr6qszxjmgr4zwk7vmw3z/bin/cice_auscom_3600x2700_90x90_722p.exe
   hashes:
-    binhash: 67dd37413d389fc240c8ca24e3337fba
-    md5: f2787a34fcedaedc63a979c260e1c82d
+    binhash: 81832be2447772591f06152856921a42
+    md5: 7edfdd3d4d3c28109bb31321976389a8
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
-    md5: a3f10baeadb88e813b4a8121f61a6226
+    binhash: f4eaa918e5e24afa8050d4a5c9ae2382
+    md5: df8bec1f690c24cdfbeced2481352c38
github-actions[bot] commented 3 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-4

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-4` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-4 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-4` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.03, spack-config 2024.07.02

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.03`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.03) * `access-nri/spack-config` version [`2024.07.02`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.02) If this is not what was expected, commit changes to `config/versions.json`.
dougiesquire commented 3 months ago

Because it's easy with the prerelease workflow, I also tried reverting parallelio. In the table below:

I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

  Exp 1 Exp 2
output000 829.89 s 833.71 s
output001 828.51 s 842.85 s
output002 833.95 s 836.21 s
average 830.78 s 837.59 s

So better (Exp 2 < 1% slower).

Below is the (selective) output of a diff between the two experiments:

$ git diff -- om2_01deg_jra55_ryf_pr75-orig om2_01deg_jra55_ryf_pr75-4
diff --git a/om2_1deg_jra55_ryf_pr75-orig/config.yaml b/om2_1deg_jra55_ryf_pr75-4/config.yaml
index de774f2..8b38a30 100644
--- a/om2_1deg_jra55_ryf_pr75-orig/config.yaml
+++ b/om2_1deg_jra55_ryf_pr75-4/config.yaml
@@ -30,7 +30,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
       input:
             - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
             - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -38,7 +38,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
@@ -57,7 +57,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-237agoozrv273r2sgeqlkzao7yosudw4/bin/cice_auscom_360x300_24x1_24p.exe
       input:
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.05.30/grid.nc
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.10.22/kmt.nc
@@ -74,7 +74,7 @@ collate:
   mem: 30GB
   ncpus: 4
   queue: normal
-  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/mppnccombine.spack
+  exe: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/mppnccombine.spack

 # Misc
 runlog: false
diff --git a/om2_1deg_jra55_ryf_pr75-orig/manifests/exe.yaml b/om2_1deg_jra55_ryf_pr75-4/manifests/exe.yaml
index 80ed3f5..014d83b 100644
--- a/om2_1deg_jra55_ryf_pr75-orig/manifests/exe.yaml
+++ b/om2_1deg_jra55_ryf_pr75-4/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-efhn2rufqy3orlpahmykegw5wxuowx3c/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: b21a1ef267acae687c3c4cbc0ae51a8e
+    md5: f434bd5b517481ae27b4c8f47f95b94d
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-237agoozrv273r2sgeqlkzao7yosudw4/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 3a65f67d21152e77034da28c22a94c66
-    md5: 37866455b057c85c3ea50c0ef0ea840b
+    binhash: 6f82a0227edc13bb1caefd00e90ff6f5
+    md5: a1ac59cfd349d8bcd37b227aa9dd8ea5
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.21/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-npjowklrqzkoz7xosbhuifyazjyrt5lk/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
-    md5: a3f10baeadb88e813b4a8121f61a6226
+    binhash: f4eaa918e5e24afa8050d4a5c9ae2382
+    md5: df8bec1f690c24cdfbeced2481352c38
dougiesquire commented 3 months ago

So better (Exp 2 < 1% slower).

@aidanheerdegen are we okay with this?

aidanheerdegen commented 3 months ago

So better (Exp 2 < 1% slower).

@aidanheerdegen are we okay with this?

I think it's your call. Well mostly your call. Yeah I think that is in the noise. Sorry I didn't see this earlier, we'd kinda decided to shelve the updates to the dependencies until we could work out what the issue was. Looks like you found it.

As long as that version of parallelio is fit for purpose then yeah, I think we should plough on and update everything but parallelio. It would be good to report something to the devs about impaired performance, but that would require a lot more work and who has the time?

I recall @anton-seaice had some issue with a parallelio bug in the MOM6-CICE6 world, but I suppose it doesn't affect OM2?

anton-seaice commented 3 months ago

PIO 2.6.2 does a bit more validation of compatibility between dependencies and file formats than 2.5.2 did, this might cause some delay.

I haven't found any issues with 2.6.2, it did fix some issues we had with backward compatibility in 2.5.10 (https://github.com/NCAR/ParallelIO/issues/1985).

@micaeljtoliveira found a floating point exception with hdf5, which we think has been fixed in the hdf5 main branch but not released yet. It would stop compilation when using debug flags, but normal release build compiles ok.

github-actions[bot] commented 3 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-5

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-5` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-5 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-5` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.07, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.07`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.07) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 2 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-6

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-6` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-6 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-6` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.07, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.07`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.07) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 2 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-7

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-7` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-7 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-7` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.07, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.07`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.07) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 2 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-7

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-7` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-7 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-7` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.07, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.07`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.07) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 2 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-8

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-8` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-8 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-8` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.07, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.07`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.07) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
github-actions[bot] commented 2 months ago

:rocket: Deploying access-om2 2024.03.2 as prerelease pr75-9

Details and usage instructions This `access-om2` model will be deployed as: * `2024.03.2` as a Release (when merged). * `pr75-9` as a Prerelease (during this PR). This Prerelease is accessible on Gadi using: ```bash module use /g/data/vk83/prerelease/modules/access-models/ module load access-om2/pr75-9 ``` where the binaries shall be on your `$PATH`. This Prerelease is also accessible on Gadi via `/g/data/vk83/prerelease/apps/spack/0.21/spack` in the `access-om2-pr75-9` environment.

:hammer_and_wrench: Using: spack-packages 2024.07.08, spack-config 2024.07.05

Details It will be deployed using: * `access-nri/spack-packages` version [`2024.07.08`](https://github.com/ACCESS-NRI/spack-packages/releases/tag/2024.07.08) * `access-nri/spack-config` version [`2024.07.05`](https://github.com/ACCESS-NRI/spack-config/releases/tag/2024.07.05) If this is not what was expected, commit changes to `config/versions.json`.
dougiesquire commented 2 months ago

In the table below:

I ran three years of each. The experiments were run at the same time on Gadi. Times in the table are from the PAYU_WALLTIME field in job.yaml.

  Exp 1 Exp 2
output000 818.15 s 835.54 s
output001 827.26 s 833.76 s
output002 826.75 s 833.64 s
average 824.05 s 834.31 s

So the prerelease runs (~ 1% slower).

Below is the (selective) output of a diff between the two experiments:

$ git diff -- orig pr75-9
diff --git a/orig/config.yaml b/pr75-9/config.yaml
index de774f2..8e48816 100644
--- a/orig/config.yaml
+++ b/pr75-9/config.yaml
@@ -30,7 +30,7 @@ input:
 submodels:
     - name: atmosphere
       model: yatm
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
       input:
             - /g/data/vk83/experiments/inputs/access-om2/remapping_weights/JRA55/global.1deg/2020.05.30/rmp_jrar_to_cict_CONSERV.nc
             - /g/data/vk83/experiments/inputs/JRA-55/RYF/v1-4/data
@@ -38,7 +38,7 @@ submodels:

     - name: ocean
       model: mom
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/fms_ACCESS-OM.x
       input:
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc
           - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/ocean_hgrid.nc
@@ -57,7 +57,7 @@ submodels:

     - name: ice
       model: cice5
-      exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+      exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
       input:
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.05.30/grid.nc
         - /g/data/vk83/experiments/inputs/access-om2/ice/grids/global.1deg/2020.10.22/kmt.nc
@@ -74,7 +74,7 @@ collate:
   mem: 30GB
   ncpus: 4
   queue: normal
-  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/mppnccombine.spack
+  exe: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/mppnccombine.spack

 # Misc
 runlog: false
diff --git a/orig/manifests/exe.yaml b/pr75-9/manifests/exe.yaml
index 80ed3f5..241395f 100644
--- a/orig/manifests/exe.yaml
+++ b/pr75-9/manifests/exe.yaml
@@ -2,17 +2,17 @@ format: yamanifest
 version: 1.0
 ---
 work/atmosphere/yatm.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/libaccessom2-git.2023.10.26=2023.10.26-ltfg7jcn6t4cefotvj3kjnyu5nru26xo/bin/yatm.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/libaccessom2-git.2023.10.26_2023.10.26-qotmoon5sfw6zp2hnjpntyjjpel7qi45/bin/yatm.exe
   hashes:
-    binhash: 4e8b4ef76e971c4af3b26cfac632e160
-    md5: 5baa1d417fe6708fc30cbeaa57d82f96
+    binhash: d14806ad4b0b890d4c2f78a7e34258ab
+    md5: 651ea586afa441f3d94376f28f26b54d
 work/ice/cice_auscom_360x300_24x1_24p.exe:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/cice5-git.2023.10.19=2023.10.19-v3zncpqjj2gyseudbwiudolcjq3k3leo/bin/cice_auscom_360x300_24x1_24p.exe
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/cice5-git.2023.10.19_2023.10.19-3khrmuxma7p4wifa5av2biukx4iomk2l/bin/cice_auscom_360x300_24x1_24p.exe
   hashes:
-    binhash: 3a65f67d21152e77034da28c22a94c66
-    md5: 37866455b057c85c3ea50c0ef0ea840b
+    binhash: 000959a21e84ba4baa66c32e61c03d39
+    md5: 237c0c1838fd072e43f7b632315cc2bb
 work/ocean/fms_ACCESS-OM.x:
-  fullpath: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
+  fullpath: /g/data/vk83/prerelease/apps/spack/0.22/release/linux-rocky8-x86_64/intel-2021.10.0/mom5-git.2023.11.09_2023.11.09-ysj247r6owjfwcralshyy4yqpges7n6o/bin/fms_ACCESS-OM.x
   hashes:
-    binhash: 92ce1ff1a38f44f92ceafd67e8e7142c
-    md5: a3f10baeadb88e813b4a8121f61a6226
+    binhash: 66f01cf1c5e589ba8a253bc09bb365c7
+    md5: 04e7b7594760b7072ff828dd463acf5e
dougiesquire commented 2 months ago

We decided not to merge this yet, given that there still seems to be a performance hit and further investigation is needed to work out what is causing this.

CodeGat commented 1 week ago

Going to have to rebase this PR to the new main - done some history rewriting here.