search

ACCESS-NRI / spack-packages

Spack package repository maintained by ACCESS-NRI
Apache License 2.0
4 stars 4 forks source link

Reproduce pre-industrial output #121

Closed penguian closed 1 week ago

penguian commented 2 weeks ago

Closes #120 The executables cice4, mom5 and um7 executables in the following configuration reproduce the output of https://github.com/ACCESS-NRI/access-esm1.5-configs/tree/pre-industrial https://github.com/ACCESS-NRI/access-esm1.5-configs/blob/0f769ae4338005f8ed3c6ce6478e004842d4a598/config.yaml

 spack find
-- linux-rocky8-x86_64_v4 / gcc@8.5.0 ---------------------------
bzip2@1.0.8  c-blosc@1.21.5  cmake@3.24.2  diffutils@3.9  gmake@4.3  hdf5@1.10.11  libaec@1.0.6  libiconv@1.17  lz4@1.9.4  netcdf-c@4.7.1  openmpi@4.0.2  pkgconf@1.9.5  snappy@1.1.10  zlib-ng@2.1.4  zstd@1.5.5

-- linux-rocky8-x86_64_v4 / intel@19.0.3.199 --------------------
cice4@access-esm1.5  dummygrib@1.0  fcm@2021.05.0  gcom4@access-esm1.5  gmake@4.4.1  mom5@access-esm1.5  netcdf-fortran@4.5.2  oasis3-mct@access-esm1.5  um7@access-esm1.5
==> 24 installed packages

except for 3 to 6 point differences in one field in each of 12 output000/atmosphere/aiihca.paa1* UM output files, as per the following example:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* CUMF-II Comparison Report *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

File 1: access-esm1.5-configs.53/output000/atmosphere/aiihca.paa1dec
File 2: access-esm1.5-configs.70/output000/atmosphere/aiihca.paa1dec
Files DO NOT compare
  * 0 differences in fixed_length_header (with 7 ignored indices)
  * 1 field differences, of which 1 are in data

Compared 2113/2113 fields, with 2112 matches

Maximum RMS diff as % of data in file 1: 0.58832395008410732 (field 443)
Maximum RMS diff as % of data in file 2: 0.58662723084335489 (field 443)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* Field 443/2113 - CO2 LAND SURFACE FLUX     KG/M**2/S *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Lookup compares, data DOES NOT compare
Compared 64/64 lookup values.
File_1 lookup info:
  t1(0101/12/01 00:00:335)  lblev(0)/blev(0.0)  lbproc(128)
Data differences:
  Number of point differences  : 6/27840
  Maximum absolute difference  : 547025.73463143967
  RMS difference               : 3288.6607554994839
  RMS diff as % of file_1 data : 0.58832395008410732
  RMS diff as % of file_2 data : 0.58662723084335489

Is this difference significant?

penguian commented 2 weeks ago

@blimlim I conducted my testing from the /g/data/tm70/pcl851/src/ACCESS-NRI/access-esm1.5-configs directory. If you look at config.yaml in that directory you will see that the executables used are from three different sets of directories:

  1. The original executables: 
    $ grep "  exe:" config.yaml
  exe: /g/data/access/payu/access-esm/bin/coe/um7.3x
  exe: /g/data/access/payu/access-esm/bin/coe/mom5xx
  exe: /g/data/access/payu/access-esm/bin/coe/cicexx
exe: /g/data/access/payu/access-esm/bin/mppnccombine
  2. The executables created via https://github.com/penguian/access-esm-build-gadi : 
    $ grep access-esm-build-gadi config.yaml
  # exe: /g/data/tm70/pcl851/src/penguian/access-esm-build-gadi/bin/um_hg3.exe-20240620
  # exe: /g/data/tm70/pcl851/src/penguian/access-esm-build-gadi/bin/momxx
  # exe: /g/data/tm70/pcl851/src/penguian/access-esm-build-gadi/bin/cice4.1_access-mct-12p-20240620
# exe: /g/data/tm70/pcl851/src/penguian/access-esm-build-gadi/bin/mppnccombine
  3. The executables created from the 120-reproduce-pre-industrial-output branch: 
    $ grep linux-rocky8 config.yaml
  # exe: /g/data/tm70/pcl851/src/ACCESS-NRI/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/um7-access-esm1.5-tysx4miwgxa6nzyx7v5sxnbosfok4z7o/bin/um_hg3.exe
  # exe: /g/data/tm70/pcl851/src/ACCESS-NRI/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/mom5-access-esm1.5-ogwxk6dpwoah5dkrsppuxuevqgtjt3fe/bin/fms_ACCESS-CM.x
  # exe: /g/data/tm70/pcl851/src/ACCESS-NRI/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/cice4-access-esm1.5-uwzdgjdt7hc3wgvceuby52sp32tuxb4k/bin/cice_access_360x300_12x1_12p.exe
# exe: /g/data/tm70/pcl851/src/ACCESS-NRI/release/linux-rocky8-x86_64_v4/intel-19.0.3.199/mom5-access-esm1.5-ogwxk6dpwoah5dkrsppuxuevqgtjt3fe/bin/mppnccombine.spack
penguian commented 2 weeks ago

Re-running with /g/data/tm70/pcl851/src/ACCESS-NRI/access-esm1.5-configs/config.yaml set to use the executables from /g/data/access/payu/access-esm/bin produces essentially the same differences in output. See /scratch/tm70/pcl851/access-esm/archive/output000.53-70.diff versus /scratch/tm70/pcl851/access-esm/archive/output000.53-71.diff.

penguian commented 2 weeks ago

@harshula The Ruff errors in packages/oasis3-mct/package.py at https://github.com/ACCESS-NRI/spack-packages/actions/runs/9638336022 etc. are intentional, and result from leaving the strings as f strings.

harshula commented 2 weeks ago

Hi @penguian , If you allow a netcdf-c range of 4.7.1 to 4.7.4. Don't you need to allow a netcdf-fortran range of 4.5.1 to 4.5.2?

$ cat /apps/netcdf/4.7.1/lib/Intel/pkgconfig/netcdf-fortran.pc 
prefix=/apps/netcdf/4.7.1
exec_prefix=${prefix}
libdir=/apps/netcdf/4.7.1/lib/Intel
includedir=/apps/netcdf/4.7.1/include/Intel
ccompiler=gcc
fcompiler=ifort

Name: netcdf-fortran
Description: NetCDF Client Library for Fortran
URL: http://www.unidata.ucar.edu/netcdf
Version: 4.5.1
Requires.private: netcdf > 4.1.1
Libs: -L${libdir} -lnetcdff
Libs.private: -L${libdir} -lnetcdff -lnetcdf
Cflags: -I${includedir}

$ cat /apps/netcdf/4.7.4/lib/Intel/pkgconfig/netcdf-fortran.pc 
prefix=/apps/netcdf/4.7.4
exec_prefix=${prefix}
libdir=/apps/netcdf/4.7.4/lib/Intel
includedir=/apps/netcdf/4.7.4/include/Intel
ccompiler=gcc
fcompiler=ifort

Name: netcdf-fortran
Description: NetCDF Client Library for Fortran
URL: http://www.unidata.ucar.edu/netcdf
Version: 4.5.2
Requires.private: netcdf > 4.1.1
Libs: -L${libdir} -lnetcdff
Libs.private: -L${libdir} -lnetcdff -lnetcdf
Cflags: -I${includedir}
harshula commented 2 weeks ago

Hi @penguian , Is -march=cascadelake overridden by the Spack compiler wrapper when granularity is set to generic?

penguian commented 1 week ago

With packages as follows:

$ spack find
-- linux-rocky8-x86_64_v4 / gcc@8.5.0 ---------------------------
bzip2@1.0.8  cmake@3.24.2  diffutils@3.9  gmake@4.3  libaec@1.0.6  libiconv@1.17  lz4@1.9.4  pkgconf@1.9.5  snappy@1.1.10  zstd@1.5.5

-- linux-rocky8-x86_64_v4 / intel@19.0.3.199 --------------------
cice4@access-esm1.5  dummygrib@1.0  gcom4@access-esm1.5  hdf5@1.10.11        netcdf-c@4.7.4        oasis3-mct@access-esm1.5  pkgconf@1.9.5      zlib-ng@2.1.4
cmake@3.24.2         fcm@2021.05.0  gmake@4.4.1          mom5@access-esm1.5  netcdf-fortran@4.5.2  openmpi@4.1.0             um7@access-esm1.5
==> 25 installed packages

the output of the pre-industrial experiment is still essentially bitwise identical to the output of https://github.com/ACCESS-NRI/access-esm1.5-configs/tree/pre-industrial https://github.com/ACCESS-NRI/access-esm1.5-configs/blob/0f769ae4338005f8ed3c6ce6478e004842d4a598/config.yaml which used the coe executables.

penguian commented 1 week ago

The um7 SPD still allows -xHost, and also has a strange structure with many explicit environment variable definitions. I will create a separate Issue to address this. Even so, I think that we are ready to try another iteration of https://github.com/ACCESS-NRI/ACCESS-ESM1.5/blob/2-spack-yaml/spack.yaml

harshula commented 1 week ago

Hi @penguian , The netcdf-fortran SPD contains:

    depends_on("netcdf-c")
    depends_on("netcdf-c@4.7.4:", when="@4.5.3:")  # nc_def_var_szip required
    depends_on("doxygen", when="+doc", type="build")

If the source package does not contain any C code and only contains Fortran code, just depend on netcdf-fortran.

penguian commented 1 week ago

I changed packages/cice4/package.py as cice4 does not use C in any meaningful way.

penguian commented 1 week ago

@harshula Thanks for the approval, but this PR was still a draft. I have now completed my intended changes.

penguian commented 1 week ago

Testing succeeded.