Open penguian opened 7 months ago
Looking deeper using mule-summary
, and accounting for field numbers starting from 0 rather than 1:
[pcl851@gadi-login-02 access-esm]$ mule-summary archive.coecms.1/access-esm/restart000/atmosphere/restart_dump.astart|grep -C2 "1115\|1116\|1117\|4100\|4102\|5485\|5524"
/g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/validators.py:198: UserWarning:
File: archive.coecms.1/access-esm/restart000/atmosphere/restart_dump.astart
Field validation failures:
Fields (1114,1115,1116)
Field grid latitudes inconsistent (STASH grid: 23)
File : 145 points from -90.0, spacing 1.25
Field (Expected): 180 points from -89.5, spacing 1.25
Field (Lookup) : 180 points from 89.5, spacing -1.0
Field validation failures:
Fields (4099,4101,5484,5523)
Skipping Field validation due to irregular lbcode:
Field lbcode: 31320
warnings.warn(msg)
1113 | W COMPNT OF WIND AFTER TIMESTEP | 8760 | 37 | 42 | 150 | 192 | 145 | 0102/01/01 00:00:01
1114 | W COMPNT OF WIND AFTER TIMESTEP | 8760 | 38 | 42 | 150 | 192 | 145 | 0102/01/01 00:00:01
1115 | RIVER SEQUENCE | 8760 | 9999 | 1905 | 151 | 360 | 180 | 0102/01/01 00:00:01
1116 | RIVER DIRECTION | 8760 | 9999 | 1906 | 152 | 360 | 180 | 0102/01/01 00:00:01
1117 | RIVER WATER STORAGE M2 | 8760 | 9999 | 1902 | 153 | 360 | 180 | 0102/01/01 00:00:01
1118 | ACCUMULATED SURFACE RUNOFF KG/M2 | 8760 | 9999 | 1907 | 155 | 192 | 145 | 0102/01/01 00:00:01
1119 | ACCUMULATED SUB-SURFACE RUNOFF KG/M2 | 8760 | 9999 | 1908 | 156 | 192 | 145 | 0102/01/01 00:00:01
--
4098 | SEA-ICE TEMPERATURE AFTER B. LAYER | 8760 | 0 | 209 | 3049 | 192 | 145 | 0101/12/01 00:00:335
4099 | FLUX OF TRACER 1 IN BL | 8760 | 0 | 1389 | 3100 | 192 | 145 | 0101/12/01 00:00:335
4100 | FLUX OF TRACER 1 IN BL | 8760 | 0 | 1389 | 3100 | 43 | 48 | 0101/12/31 00:00:365
# ----- | ------------------------------------ | ---- | ----- | ---- | ------- | ----- | ----- | --------------------
# index | stash_name | lbft | lblev | lbfc | lbuser4 | lbnpt | lbrow | t1
# ----- | ------------------------------------ | ---- | ----- | ---- | ------- | ----- | ----- | --------------------
4101 | FLUX OF TRACER 2 IN BL | 8760 | 0 | 1389 | 3101 | 192 | 145 | 0101/12/01 00:00:335
4102 | FLUX OF TRACER 2 IN BL | 8760 | 0 | 1389 | 3101 | 43 | 48 | 0101/12/31 00:00:365
4103 | UNKNOWN STASH CODE | 8760 | 9999 | 1510 | 3173 | 192 | 145 | 0101/12/01 00:00:335
4104 | UNKNOWN STASH CODE | 8760 | 9999 | 1510 | 3173 | 192 | 145 | 0101/12/01 00:00:335
--
5483 | ATM TRACER 1 AFTER TS | 8760 | 37 | 501 | 33001 | 192 | 145 | 0101/12/01 00:00:335
5484 | ATM TRACER 1 AFTER TS | 8760 | 38 | 501 | 33001 | 192 | 145 | 0101/12/01 00:00:335
5485 | ATM TRACER 1 AFTER TS | 8760 | 1 | 501 | 33001 | 57 | 48 | 0101/12/31 00:00:365
5486 | ATM TRACER 2 AFTER TS | 8760 | 1 | 502 | 33002 | 192 | 145 | 0101/12/01 00:00:335
5487 | ATM TRACER 2 AFTER TS | 8760 | 2 | 502 | 33002 | 192 | 145 | 0101/12/01 00:00:335
--
5522 | ATM TRACER 2 AFTER TS | 8760 | 37 | 502 | 33002 | 192 | 145 | 0101/12/01 00:00:335
5523 | ATM TRACER 2 AFTER TS | 8760 | 38 | 502 | 33002 | 192 | 145 | 0101/12/01 00:00:335
5524 | ATM TRACER 2 AFTER TS | 8760 | 1 | 502 | 33002 | 57 | 48 | 0101/12/31 00:00:365
Removing all fields with lbuser4 in [151,152,153,155,156,3100,3101,33001,33002]
eliminates the ValueError
and the warning messages from mule-cumf
. This results in the following:
[pcl851@gadi-login-02 access-esm]$ mule-cumf archive.coecms.?/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
mule : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* CUMF-II Comparison Report *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
File 1: archive.coecms.1/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart
File 2: archive.coecms.2/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart
Files DO NOT compare
* 0 differences in fixed_length_header (with 7 ignored indices)
* 1 field differences, of which 1 are in data
Compared 5359/5359 fields, with 5358 matches
Maximum RMS diff as % of data in file 1: 0.17261724377693055 (field 4235)
Maximum RMS diff as % of data in file 2: 0.172578822112673 (field 4235)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* Field 4235/5359 - CO2 LAND SURFACE FLUX KG/M**2/S *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
Lookup compares, data DOES NOT compare
Compared 64/64 lookup values.
File_1 lookup info:
t1(0101/12/01 00:00:335) lblev(0)/blev(0.0) lbproc(128)
Data differences:
Number of point differences : 6/27840
Maximum absolute difference : 157313.73429443128
RMS difference : 970.40634872502915
RMS diff as % of file_1 data : 0.17261724377693055
RMS diff as % of file_2 data : 0.172578822112673
Given
[pcl851@gadi-login-02 access-esm]$ mule-summary archive.coecms.1/access-esm/restart000/atmosphere/fixed_bis.restart_dump.astart|grep "CO2 LAND SURFACE FLUX"
4235 | CO2 LAND SURFACE FLUX KG/M**2/S | 8760 | 0 | 1562 | 3326 | 192 | 145 | 0101/12/01 00:00:335
[pcl851@gadi-login-02 access-esm]$ grep "CO2 LAND SURFACE FLUX" ~access/umdir/vn7.3/ctldata/STASHmaster/STASHmaster_A
1| 1 | 3 | 326 |CO2 LAND SURFACE FLUX KG/M**2/S |
I also eliminated fields with lbuser4==3326
. This finally results in a clean comparison:
[pcl851@gadi-login-02 access-esm]$ mule-cumf archive.coecms.?/access-esm/restart000/atmosphere/fixed.restart_dump.astart
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* (CUMF-II) Module Information *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
mule : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/mule/__init__.py (version 2022.07.1)
um_utils : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_utils/__init__.py (version 2022.07.1)
um_packing : /g/data/hh5/public/apps/miniconda3/envs/analysis3-23.07/lib/python3.10/site-packages/um_packing/__init__.py (version 2022.07.1) (packing lib from SHUMlib: 2023061)
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
* CUMF-II Comparison Report *
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
File 1: archive.coecms.1/access-esm/restart000/atmosphere/fixed.restart_dump.astart
File 2: archive.coecms.2/access-esm/restart000/atmosphere/fixed.restart_dump.astart
Files compare
* 0 differences in fixed_length_header (with 7 ignored indices)
* 0 field differences, of which 0 are in data
Compared 5358/5358 fields, with 5358 matches
See https://github.com/penguian/access-esm/blob/pre-industrial-fix-restart/bin/fix_esm15_restart.py for a Python script that will fix restart_dump.astart
by removing the fields listed above.
Usage:
$ module use /g/data/access/modules
$ module load um
$ module use /g/data/hh5/public/modules/
$ module load conda/analysis3-23.07
$ cd archive/access-esm/restart000/atmosphere/
$ python $PATHTO/bin/fix_esm15_restart.py restart_dump.astart fixed.restart_dump.astart
Thanks to Scott Wales for the original Python code as per https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/5
Because of the STASH time averaging the restart file contains partial time averages of diagnostic fields as well as the real prognostics that are strictly required. Some of these problems come from the diagnostics. E.g. fields with STASH codes 3,326 and 3,327 are CO2 fluxes which are not set in this concentration driven experiment.
The river model runs on its own 1x1 degree grid and mule thinks the river fields (stash codes 151,152,153) are invalid because this doesn't match the overall file grid.
Fields with stash codes 155 & 156 are accumulated runoff. According to the STASHmaster file these are supposed to be saved on land points only.
~access/umdir/vn7.3/ctldata/STASHmaster/STASHmaster_A
1| 1 | 0 | 155 |ACCUMULATED SURFACE RUNOFF KG/M2|
2| 3 | 0 | 1 | 21 | 5 | -1 | -1 | 0 | 0 | 0 | 0 |
The 21 in the 4th field of the second row means land points only.
However for some reason the field header in the restart file shows it as global
mule-pumf --components=lookup --headers-only --field-property lbuser4=155 restart file
%%%%%%%%%%%%%%%%%%%%
* Field 1118/3640 *
%%%%%%%%%%%%%
...
(15) lblrec : 10865
(21) lbpack : 0
where lbpack=10865 is the data size, here equal to the number of land points.
C.f. soil moisture (lbuser4=9) which shows
(21) lbpack : 120
The model itself gets the packing from the STASHmaster file so reads this field ok. Not clear why it's not written properly? It's ok in ACCESS-CM2 restart files.
An alternative way to get rid of the problem fields is
mule-select restart restart.subset --include lbproc=0 --exclude lbuser4=151,152,153,155,156
which includes only the true prognostic fields (lbproc=0) and excludes the problem ones.
I also created a ticket against mule-cumf
: https://code.metoffice.gov.uk/trac/um/ticket/7560#ticket
Running Martin Dix's fix as per https://github.com/coecms/access-esm/commit/0f769ae4338005f8ed3c6ce6478e004842d4a598 results in a corrupted
archive/access-esm/restart000/atmosphere/restart_dump.astart
file in the following sense.I ran the configuration twice, moving the
/scratch/tm70/pcl851/access-esm/archive
directory to/scratch/tm70/pcl851/access-esm/archive.coecms.1
and then/scratch/tm70/pcl851/access-esm/archive.coecms.2
, then ran the following commands with results as shown.See also https://forum.access-hive.org.au/t/how-to-use-cumf-or-mule-cumf-with-access-esm1-5-um-output/1794/6