dougiesquire commented 1 month ago

The set of diagnostics output by access-om3 configurations is still mostly just inherited from CESM. There are possibly many diagnostics that we want that aren't requested, and others that are requested that we don't want. It would be helpful to compile a set of "essential" diagnostics that should always be requested in configurations so that key metrics/analyses can be calculated.

I'm probably being naive in thinking that it's possible to come up with a single list of essential diagnsotics. In that case, we still need to know what diagnostics are needed to calculate particular metrics so that we can ensure everything we need is saved in upcoming test runs.

aekiss commented 1 month ago

We should start with the CICE and MOM diagnostics used in ACCESS-OM2.

dougiesquire commented 1 month ago

Is there a single list of diagnostics for ACCESS-OM2?

aekiss commented 1 month ago

The default configs are pretty consistent

aekiss commented 1 month ago

It would be good if we could use make_diag_table to generate diag_table, as this makes it easier to create consistent filenames, e.g. when creating one file per variable.

I'm looking into this - see https://github.com/COSIMA/make_diag_table/issues/5

dougiesquire commented 1 month ago

It would be good if we could use make_diag_table to generate diag_table, as this makes it easier to create consistent filenames, e.g. when creating one file per variable.

I'm looking into this - see COSIMA/make_diag_table#5

I was just doing the same thing. I'll leave it with you :)

minghangli-uni commented 1 month ago

Essential diagnostics are needed as discussed in TWG.

Zonal average temperature and salinity (i.e. depth/latitude maps) (Fig. 12 Kiss et al. 2020) [1993 - 2017]
Time series of global average temperature, salinity and sea surface temperature. (Fig. 3 in Kiss et al. 2020), and sea surface height.
Zonally integrated overturning in density / latitude space (Fig. 7 Kiss et al. 2020) [1993 - 2017]
Time series of Drake Passage zonal transport. (Fig. 4 in Kiss et al. 2020)

dougiesquire commented 1 month ago

@minghang is working through what MOM5 diagnostics output in the ACCESS-OM2 default configs are available in MOM6 and what they are called.

aekiss commented 1 month ago

I've confirmed that make_diag_table is compatible with MOM6 - see https://github.com/COSIMA/make_diag_table/issues/5

minghangli-uni commented 1 month ago

Great to know! Thanks @aekiss

aekiss commented 1 month ago

To handle many output files I used these settings in input.nml:

 &diag_manager_nml
    debug_diag_manager = .true.
    issue_oor_warnings = .true.
    flush_nc_files = .true.
    max_axes          = 400
    max_files         = 200
    max_num_axis_sets = 200
 /

This also issues warnings in access-om3.err for any unavailable diagnostics, so a test run with an OM2 diag_table will reveal which diagnostics need to be renamed.

aekiss commented 1 month ago

We'll also need to consider which ones to vertically remap onto a non-native grid - see https://mom6.readthedocs.io/en/main/api/generated/pages/Diagnostics.html#native-diagnostics

minghangli-uni commented 1 month ago

Ref https://github.com/COSIMA/access-om3/issues/178, we currently dont have a proper isopycnal coordinate at the moment.

However we dont have a proper 0.25deg density coordinate as was noted from https://github.com/COSIMA/MOM6-CICE6/issues/40

minghangli-uni commented 1 month ago

Below diagnostics are the conversion between MOM5 and MOM6. The table is still being updated.

The diagnostics of MOM5, including description, unit, method and packing are generated by matching information from the MOM_diags.txt and diag_table. For diagnostics with missing description, unit, method and packing, these details are not present in the MOM_diags.txt or in mom5. I am not sure how these diagnostiics are traced. Any thoughts? @aekiss

Static 2D Grid Data

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
area_t	area_t	Tracer cell area	m²	none	real*4
area_u	area_u	Surface area of x-direction flow (U) cells	m²	none	real*4
area_v	N/A	Surface area of y-direction flow (V) cells	m²	none	real*4
drag_coeff	drag_coeff	Dimensionless bottom drag coefficient	dimensionless	none	real*4
dxt	dxt	Ocean dxt on t-cells	m	none	real*4
dxu	dxu	Ocean dxu on u-cells	m	none	real*4
dyt	dyt	Ocean dyt on t-cells	m	none	real*4
dyu	dyu	Ocean dyu on u-cells	m	none	real*4
N/A	geolat_c	Ucell latitude	degrees_N	none	real*4
N/A	geolat_t	Tracer latitude	degrees_N	none	real*4
N/A	geolon_c	UV longitude	degrees_E	none	real*4
N/A	geolon_t	Tracer longitude	degrees_E	none	real*4
geolat_c	N/A	Latitude of corner (Bu) points	degrees_north	none	real*4
geolat	N/A	Latitude of tracer (T) points	degrees_north	none	real*4
geolat_v	N/A	Latitude of meridional velocity (Cv) points	degrees_north	none	real*4
geolon_c	N/A	Longitude of corner (Bu) points	degrees_east	none	real*4
geolon	N/A	Longitude of tracer (T) points	degrees_east	none	real*4
geolon_v	N/A	Longitude of meridional velocity (Cv) points	degrees_east	none	real*4
depth_ocean	ht	Ocean depth on t-cells	m	none	real*4
depth_ocean	hu	Ocean depth on u-cells	m	none	real*4
	kmt	Number of depth levels on t-grid	dimensionless	none	real*4
	kmu	Number of depth levels on u-grid	dimensionless	none	real*4

Monthly 3D Fields

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
agessc	age_global	Sea water age since surface contact	yr	average	real*4
	buoyfreq2_wt	Squared buoyancy frequency at T-cell bottom	1/s²	average	real*4
difvho	diff_cbt_t	Total vertical diffusion of temperature (w/o neutral)	m²/s	average	real*4
	dzt	T-cell thickness	m	average	real*4
rhopot0	pot_rho_0	Potential density referenced to 0 dbar	kg/m³	average	real*4
rhopot2	pot_rho_2	Potential density referenced to 2000 dbar	kg/m³	average	real*4
thetao	pot_temp	Potential temperature	°C	average	real*4
so	salt	Practical Salinity	psu	average	real*4
	temp_xflux_adv	Temperature x flux (advection)
	temp_yflux_adv	Temperature y flux (advection)
	temp	Conservative temperature	°C	average	real*4
umo	tx_trans	T-cell i-mass transport	kg/s	average	real*4
	ty_trans_gm	T-cell mass j-transport from GM	kg/s	average	real*4
	ty_trans_nrho_submeso	T-cell j-mass transport from submesoscale param on neutral rho	kg/s	average	real*4
	ty_trans_rho_gm	T-cell j-mass transport from GM on potential density	kg/s	average	real*4
	ty_trans_rho	T-cell j-mass transport on potential density	kg/s	average	real*4
	ty_trans_submeso	T-cell mass j-transport from submesoscale param	kg/s	average	real*4
vmo	ty_trans	T-cell j-mass transport	kg/s	average	real*4
uo	u	i-current	m/sec	average	real*4
vo	v	j-current	m/sec	average	real*4
	vert_pv	Vertical piece of Ertel PV: (f+zeta)*N²	1/sec³	average	real*4
wd	wt	Dia-surface velocity T-points	m/sec	average	real*4

Monthly 3D Squared Fields

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
uo	u	i-current	m/sec	pow02	real*4
vo	v	j-current	m/sec	pow02	real*4

monthly 2d fields

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
KHTH_t	agm	GM diffusivity at surface	m^2/sec	average	real*4
KHTR_h	aredi	neutral diffusivity at k=1	m^2/sec	average	real*4
bmf_u	bmf_u	Bottom u-stress via bottom drag	N/m^2	average	real*4
bmf_v	bmf_v	Bottom v-stress via bottom drag	N/m^2	average	real*4
	ekman_we	Ekman vertical velocity averaged to wt-point	m/s	average	real*4
	eta_nonbouss	surface height including steric contribution	meter	average	real*4
latent_evap	evap_heat	latent heat flux into ocean (<0 cools ocean)	W/m^2	average	real*4
evap	evap	mass flux from evaporation/condensation (>0 enters ocean)	(kg/m^3)*(m/sec)	average	real*4
heat_content_fprec	fprec_melt_heat	heat flux to melt frozen precip (<0 cools ocean)	W/m^2	average	real*4
fprec	fprec	snow falling onto ocean (>0 enters ocean)	(kg/m^3)*(m/sec)	average	real*4
	frazil_3d_int_z	Vertical sum of ocn frazil heat flux over time step	W/m^2	average	real*4
lprec	lprec	liquid precip (including ice melt/form) into ocean (>0 enters ocean)	(kg/m^3)*(m/sec)	average	real*4
LW	lw_heat	longwave flux into ocean (<0 cools ocean)	W/m^2	average	real*4
	melt	water flux transferred with sea ice form/melt (>0 enters ocean)	(kg/m^3)*(m/sec)	average	real*4
seaice_melt_heat	mh_flux	heat into ocean due to melting ice (>0 heats ocean)	(W/m^2)	average	real*4
MLD_003	mld	mixed layer depth determined by density criteria	m	average	real*4
	net_sfc_heating	surface ocean heat flux coming through coupler and mass transfer	Watts/m^2	average	real*4
pbo	pbot_t	bottom pressure on T cells' // trim ( model_type )	dbar	average	real*4
	pme_net	precip-evap into ocean (total w/ restore + normalize)	(kg/m^3)*(m/sec)	average	real*4
	pme_river	mass flux of precip-evap+river via sbc (liquid	frozen	average	real*4
N/A	river	mass flux of river (runoff + calving) entering ocean	(kg/m^3)*(m/sec)	average	real*4
ficeberg	N/A	Frozen runoff (calving) and iceberg melt into ocean	(kg/m^3)*(m/sec)	average	real*4
friver	runoff	Liquid runoff (rivers) into ocean	(kg/m^3)*(m/sec)	average	real*4
N/A	sea_level_sq	square of effective sea level (eta_t + patm/(rho0*g)) on T cells	m^2	average	real*4
e_D	sea_level	effective sea level (eta_t + patm/(rho0*g)) on T cells	meter	average	real*4
hfsso	sens_heat	sensible heat into ocean (<0 cools ocean)	W/m^2	average	real*4
net_heat_coupler	sfc_hflux_coupler	surface heat flux coming through coupler	Watts/m^2	average	real*4
heat_content_lrunoff	sfc_hflux_from_runoff	heat flux (relative to 0C) from liquid river runoff	Watts/m^2	average	real*4
Heat_PmE	sfc_hflux_pme	heat flux (relative to 0C) from pme transfer of water across ocean surface	Watts/m^2	average	real*4
salt_flux_in	sfc_salt_flux_coupler	sfc_salt_flux_coupler: flux from the coupler	kg m-2 s-1	average	real*4
	sfc_salt_flux_ice		kg m-2 s-1	average	real*4
	sfc_salt_flux_restore		kg m-2 s-1	average	real*4
sfdsi	N/A	Net salt flux into ocean at surface (restoring + sea-ice)	kg m-2 s-1	average	real*4
tos	surface_pot_temp	Sea Surface Temperature	degC	average	real*4
sos	surface_salt	Sea Surface Salinity	psu	average	real*4
SW_pen	swflx	shortwave flux into ocean (>0 heats ocean)	W/m^2	average	real*4
taux	tau_x	i-directed wind stress forcing u-velocity	N/m^2	average	real*4
tauy	tau_y	j-directed wind stress forcing v-velocity	N/m^2	average	real*4
	temp_int_rhodz	vertical sum of Conservative temperature * rho_dzt	deg_C(kg/m^3)m	average	real*4
	temp_xflux_adv_int_z	z-integral of cprhodytutemp	W	average	real*4
	temp_xflux_gm_int_z	z-integral cpgm_xfluxdytrho_dzttemp	W	average	real*4
	temp_xflux_ndiffuse_int_z	z-integral cpndiffuse_xfluxdytrho_dzttemp	W	average	real*4
	temp_xflux_submeso_int_z	z-integral cpsubmeso_xfluxdytrho_dzttemp	W	average	real*4
	temp_yflux_adv_int_z	z-integral of cprhodxtvtemp	W	average	real*4
	temp_yflux_gm_int_z	z-integral cpgm_yfluxdytrho_dzttemp	W	average	real*4
	temp_yflux_ndiffuse_int_z	z-integral cpndiffuse_yfluxdxtrho_dzttemp	W	average	real*4
	temp_yflux_submeso_int_z	z-integral cpsubmeso_yfluxdxtrho_dzttemp	W	average	real*4
umo_2d	tx_trans_int_z	T-cell i-mass transport vertically summed	trim ( transport_dims )	average	real*4
vmo_2d	ty_trans_int_z	T-cell j-mass transport vertically summed	trim ( transport_dims )	average	real*4
seaice_melt	wfiform	water out of ocean due to ice form (>0 enters ocean)	(kg/m^3)*(m/sec)	average	real*4
seaice_melt	wfimelt	water into ocean due to ice melt (>0 enters ocean)	(kg/m^3)*(m/sec)	average	real*4

monthly 2d fields with different reduction methods

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
MLD_003	mld	mixed layer depth determined by density criteria	m	max	real*4
tos	surface_pot_temp	Sea Surface Temperature	degC	max	real*4

daily 2d fields

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
	bottom_temp	Conservative temperature	degC	average	real*4
	frazil_3d_int_z	Vertical sum of ocn frazil heat flux over time step	W/m^2	average	real*4
MLD_003	mld	mixed layer depth determined by density criteria	m	average	real*4
	pme_river	mass flux of precip-evap+river via sbc (liquid	frozen	average	real*4
e_D	sea_level	effective sea level (eta_t + patm/(rho0*g)) on T cells	meter	average	real*4
net_heat_coupler	sfc_hflux_coupler	surface heat flux coming through coupler	Watts/m^2	average	real*4
heat_content_lrunoff	sfc_hflux_from_runoff	heat flux (relative to 0C) from liquid river runoff	Watts/m^2	average	real*4
Heat_PmE.	sfc_hflux_pme	heat flux (relative to 0C) from pme transfer of water across ocean surface	Watts/m^2	average	real*4
tos	surface_pot_temp
sos	surface_salt
SSU	usurf	i-surface current	m/sec	average	real*4
SSV	vsurf	j-surface current	m/sec	average	real*4

daily 2d maximum fields

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
	bottom_temp
e_D	sea_level	effective sea level (eta_t + patm/(rho0*g)) on T cells	meter	max	real*4
tos	surface_pot_temp	Sea Surface Temperature	deg_C	max	real*4

daily 2d minimum fields

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
tos	surface_pot_temp	Sea Surface Temperature	deg_C	min	real*4

daily scalar snapshots

Diagnostic (MOM6)	Diagnostic (MOM5)	Description	Unit	Method	Packing
	eta_global	global ave eta_t plus patm_t/(g*rho0)	meter	none	real*8
KE	ke_tot	Globally integrated ocean kinetic energy	10^15 Joules	none	real*8
	pe_tot	Globally integrated ocean potential energy	10^15 Joules	none	real*8
rhoinsitu	rhoave	global mean ocean in-situ density from ocean_density_mod	kg/m^3	none	real*8
soga	salt_global_ave	Global mean salt in liquid seawater	psu	none	real*8
sosga	salt_surface_ave	Global mass weighted mean surface salt in liquid seawater	psu	none	real*8
thetaoga	temp_global_ave	Global mean temp in liquid seawater	deg_C	none	real*8
tosga	temp_surface_ave	Global mass weighted mean surface temp in liquid seawater	deg_C	none	real*8
	total_net_sfc_heating	total ocean surface flux from coupler and mass transfer	Watts/1e15	none	real*8
	total_ocean_evap_heat	total latent heat flux into ocean (<0 cools ocean)	Watts/1e15	none	real*8
	total_ocean_evap	total evaporative ocean mass flux (>0 enters ocean)	(kg/sec)/1e15	none	real*8
	total_ocean_fprec_melt_heat	total heat flux to melt frozen precip (<0 cools ocean)	Watts/1e15	none	real*8
	total_ocean_fprec	total snow falling onto ocean (>0 enters ocean)	(kg/sec)/1e15	none	real*8
	total_ocean_heat	Total heat in the liquid ocean referenced to 0degC	Joule/1e25	none	real*8
total_net_heat_coupler	total_ocean_hflux_coupler	total surface heat flux passed through coupler	Watts/1e15	none	real*8
	total_ocean_hflux_evap	total ocean heat flux from evap transferring water across surface	Watts/1e15	none	real*8
	total_ocean_hflux_prec	total ocean heat flux from precip transferring water across surface	Watts/1e15	none	real*8
	total_ocean_lprec	total liquid precip into ocean (>0 enters ocean)	(kg/sec)/1e15	none	real*8
	total_ocean_lw_heat	total longwave flux into ocean (<0 cools ocean)	Watts/1e15	none	real*8
	total_ocean_melt	total liquid water melted from sea ice (>0 enters ocean)	(kg/sec)/1e15	none	real*8
	total_ocean_mh_flux	total heat flux into ocean from melting ice (>0 heats ocean)	Watts/1e15	none	real*8
	total_ocean_pme_river	total ocean precip-evap+river via sbc (liquid	frozen	none	real*8
	total_ocean_river_heat	total heat flux into ocean from liquid+solid runoff (<0 cools ocean)	Watts/1e15	none	real*8
	total_ocean_river	total liquid river water and calving ice entering ocean	kg/sec/1e15	none	real*8
	total_ocean_runoff_heat	total ocean heat flux from liquid river runoff	Watts/1e15	none	real*8
	total_ocean_runoff	total liquid river runoff (>0 water enters ocean)	(kg/sec)/1e15	none	real*8
	total_ocean_salt	total mass of salt in liquid seawater	kg/1e18	none	real*8
	total_ocean_sens_heat	total sensible heat into ocean (<0 cools ocean)	Watts/1e15	none	real*8
	total_ocean_sfc_salt_flux_coupler
	total_ocean_swflx_vis	total visible shortwave into ocean (>0 heats ocean)	Watts/1e15	none	real*8
	total_ocean_swflx	total shortwave flux into ocean (>0 heats ocean)	Watts/1e15	none	real*8

aekiss commented 1 month ago

Awesome, thanks @minghangli-uni this is super helpful!

aekiss commented 1 month ago

For diagnostics with missing description, unit, method and packing, these details are not present in the MOM_diags.txt or in mom5. I am not sure how these diagnostiics are traced. Any thoughts? @aekiss

Some diagnostic names are generated algorithmically, so they don't show up if you search for them in the code. The description and unit should be available from output .nc files, e.g. in /g/data/cj50/access-om2/raw-output/access-om2-01/01deg_jra55v140_iaf/output000/ocean, but it might be easier to use the cookbook tools (cc.querying.get_variables), since the cookbook database has indexed this metadata - see https://cosima-recipes.readthedocs.io/en/latest/Tutorials/Using_Explorer_tools.html#COSIMA-Cookbook-solution

minghangli-uni commented 1 month ago

Thank you @aekiss !

But I am still not sure about the exact process (algorithmically) of how these diagnostics were generated. While the descriptions and units can be tracked from the output nc files, these are essentially the final output files. Are we expected to have similar diagnostics in MOM6 as well? If so, we may end up tracking similar processes as MOM5.

minghangli-uni commented 1 month ago

Below is a sample of the diag_table generated by diag_table_source.yaml and make_diag_table.py with minor tweaks:

Static grids are included in a single netcdf file.
Scalar fields are included in a single netcdf file.
2D and 3D fields (daily or monthly) are saved in separate files.
We need to discuss which fields require remapped grids.

If every of us is happy with this format, I will generate the first draft.

ACCESS-OM3-025 1900 1 1 0 0 0

static 2d grid data

"access-om3.mom6.h.static", -1, "months", 1, "days", "time" "ocean_model", "area_t", "area_t", "access-om3.mom6.h.static", "all", "none", "none", 2 "ocean_model", "area_u", "area_u", "access-om3.mom6.h.static", "all", "none", "none", 2

monthly 3d fields

"access-om3.mom6.h.3d.agessc.1.monthly.mean.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "agessc", "agessc", "access-om3.mom6.h.3d.agessc.1.monthly.mean.ym%4yr%2mo", "all", "mean", "none", 2

"access-om3.mom6.h.3d.difvho.1.monthly.mean.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "difvho", "difvho", "access-om3.mom6.h.3d.difvho.1.monthly.mean.ym%4yr%2mo", "all", "mean", "none", 2

monthly 3d squared fields

"access-om3.mom6.h.3d.uo.1.monthly.pow02.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "uo", "uo", "access-om3.mom6.h.3d.uo.1.monthly.pow02.ym%4yr%2mo", "all", "pow02", "none", 2

"access-om3.mom6.h.3d.vo.1.monthly.pow02.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "vo", "vo", "access-om3.mom6.h.3d.vo.1.monthly.pow02.ym%4yr%2mo", "all", "pow02", "none", 2

monthly 2d fields

"access-om3.mom6.h.2d.KHTH_t.1.monthly.mean.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "KHTH_t", "KHTH_t", "access-om3.mom6.h.2d.KHTH_t.1.monthly.mean.ym%4yr%2mo", "all", "mean", "none", 2

"access-om3.mom6.h.2d.KHTR_h.1.monthly.mean.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "KHTR_h", "KHTR_h", "access-om3.mom6.h.2d.KHTR_h.1.monthly.mean.ym%4yr%2mo", "all", "mean", "none", 2

monthly 2d fields with different reduction methods

"access-om3.mom6.h.2d.mlotst.1.monthly.max.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "mlotst", "mlotst", "access-om3.mom6.h.2d.mlotst.1.monthly.max.ym%4yr%2mo", "all", "max", "none", 2

"access-om3.mom6.h.2d.tos.1.monthly.min.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years" "ocean_model", "tos", "tos", "access-om3.mom6.h.2d.tos.1.monthly.min.ym%4yr%2mo", "all", "min", "none", 2

daily 2d fields

"access-om3.mom6.h.2d.tob.1.daily.mean.ym%4yr%2mo", 1, "days", 1, "days", "time", 1, "years" "ocean_model", "tob", "tob", "access-om3.mom6.h.2d.tob.1.daily.mean.ym%4yr%2mo", "all", "mean", "none", 2

"access-om3.mom6.h.2d.mlotst.1.daily.mean.ym%4yr%2mo", 1, "days", 1, "days", "time", 1, "years" "ocean_model", "mlotst", "mlotst", "access-om3.mom6.h.2d.mlotst.1.daily.mean.ym%4yr%2mo", "all", "mean", "none", 2

daily 2d maximum fields

"access-om3.mom6.h.2d.tob.1.daily.max.ym%4yr%2mo", 1, "days", 1, "days", "time", 1, "years" "ocean_model", "tob", "tob", "access-om3.mom6.h.2d.tob.1.daily.max.ym%4yr%2mo", "all", "max", "none", 2

"access-om3.mom6.h.2d.e_D.1.daily.max.ym%4yr%2mo", 1, "days", 1, "days", "time", 1, "years" "ocean_model", "e_D", "e_D", "access-om3.mom6.h.2d.e_D.1.daily.max.ym%4yr%2mo", "all", "max", "none", 2

daily 2d minimum fields

"access-om3.mom6.h.2d.tos.1.daily.min.ym%4yr%2mo", 1, "days", 1, "days", "time", 1, "years" "ocean_model", "tos", "tos", "access-om3.mom6.h.2d.tos.1.daily.min.ym%4yr%2mo", "all", "min", "none", 2

daily scalar snapshots

"access-om3.mom6.h.scalar.1.daily.ym%4yr%2mo", 1, "days", 1, "days", "time", 1, "years" "ocean_model", "soga", "soga", "access-om3.mom6.h.scalar.1.daily.ym%4yr%2mo", "all", "none", "none", 1 "ocean_model", "sosga", "sosga", "access-om3.mom6.h.scalar.1.daily.ym%4yr%2mo", "all", "none", "none", 1

monthly 2d snapshots

monthly 3d snapshots

dougiesquire commented 1 month ago

A couple of comments questions using the following as an example:

"access-om3.mom6.h.3d.agessc.1.monthly.mean.ym%4yr%2mo", 1, "months", 1, "days", "time", 1, "years"

I think 1.monthly should be 1monthly (assuming that the 1 is the number of months)
Why do we need ym? Can this be dropped?
I think the date should reflect the time period of the data in the file. For example, this file includes 1 year of data so the time stamp should be %4yr only

Altogether, this would mean the above changes to:

"access-om3.mom6.h.3d.agessc.1monthly.mean.%4yr", 1, "months", 1, "days", "time", 1, "years"

or if we want to be more concise we could shorten the frequency to be consistent with CMIP6 vocab (though this only really saves a few characters)

"access-om3.mom6.h.3d.agessc.1mon.mean.%4yr", 1, "months", 1, "days", "time", 1, "years"

Thoughts @minghangli-uni, @aekiss?

minghangli-uni commented 1 month ago

@dougiesquire Thanks for the comments.

I am more inclined to this format, "access-om3.mom6.h.3d.agessc.1mon.mean.%4yr", 1, "months", 1, "days", "time", 1, "years" It looks concise and doesn’t seem to lose any information.

aekiss commented 1 month ago

ym is just an aesthetic workaround because FMS prepends _ prior to the date, and it looks ugly having that preceded by another separator - see https://github.com/COSIMA/access-om2/issues/185#issuecomment-632428116.

In our case access-om3.mom6.h.3d.agessc.1mon.mean.%4yr will produce filenames like access-om3.mom6.h.3d.agessc.1mon.mean._1900.nc and the ._ may offend those of a sensitive disposition.

aekiss commented 1 month ago

I'm not sure that .y_1900 looks a whole lot better than ._1900 though.

aekiss commented 1 month ago

Other than that, I'm happy with

"access-om3.mom6.h.3d.agessc.1mon.mean.%4yr", 1, "months", 1, "days", "time", 1, "years"

although it would be even better if the nuopc prefix was shortened somehow to omit h and maybe access- too (ditto for cice6 and ww3 outputs).

aekiss commented 1 month ago

@minghangli-uni I'm happy to look over your diag_table_source.yaml when you've finished.

minghangli-uni commented 1 month ago

I am thinking adding some scripts to make_diag_table.py to include descriptions for each of the diagnostics. This might make the table a bit messy, but it should clarify the diag_table. It’s not urgent, though.

aekiss commented 1 month ago

hm, that's an interesting idea. But isn't that information already in available_diags.000000?

aekiss commented 1 month ago

BTW, did you need to change make_diag_table.py to produce the new diag_table, or only diag_table_source.yaml?

minghangli-uni commented 1 month ago

But isn't that information already in available_diags.000000?

Oops, yes, I forgot that info is already existing in OM3.

BTW, did you need to change make_diag_table.py to produce the new diag_table, or only diag_table_source.yaml

Specifically for 1mon, I think I need to change make_diag_table.py. Other changes will only be needed in diag_table_source.yaml.

minghangli-uni commented 1 month ago

This line

        return indict['file_name_separator'].join(fn)

minghangli-uni commented 1 month ago

This is the snippet to update fn in set_filename to resolve this issue.

def update_fn(fn):
    tmpi = 0
    updated_fn = []

    while tmpi < len(fn):
        if tmpi + 1 < len(fn) and fn[tmpi].isdigit() and fn[tmpi + 1] in ['yr', 'mon', 'day', 'hr']:
            combined = f'{fn[tmpi]}{fn[tmpi + 1]}' # e.g., combine '1' and 'mon'
            updated_fn.append(combined)
            tmpi += 2
        else:
            updated_fn.append(fn[tmpi])
            tmpi += 1
    return updated_fn

@aekiss I've tested it and it worked. Would you like me to create a PR to the repo with this change? There must be a better way to achieve it.

aekiss commented 1 month ago

Let's move this discussion to https://github.com/COSIMA/make_diag_table/issues/7

aekiss commented 1 month ago

https://github.com/COSIMA/make_diag_table/ now supports multiple filename separators

dougiesquire commented 1 month ago

FMS prepends _ prior to the date

This annoys me so much. WHY?!

dougiesquire commented 1 month ago

THIS HAS BEEN UPDATED BELOW

We agreed in a meeting today to go forward with this format (note, this comment has been updated in response to comments below):

access-om3.<component>.<variable_info>.<frequency>_<time_cell_method>_<date_stamp>.nc

where

<variable_info> is only used for MOM output, where it is equal to
- <n_dimension>.<variable_name> for single-file-per-variable output (e.g. 3d.frazil_heat_tendency)
- <n_dimension>.<descriptor> for multi-variable output all of the same dimensionality (e.g. 1d.scalar)
- md.<descriptor> for multi-variable output of multiple dimensionalities
<date_stamp> has the format YYYY_MM_DD_SSSSS and indicates the period. E.g.:
- if the file contains a years' worth of data", <date_stamp> = YYYY
- if the file contains a months' worth of data", <date_stamp> = YYYY_MM
- if the file contains 3 months' worth of data", <date_stamp> = YYYY_MM, where MM is ~first month~ some month in the period (this is dictated by FMS).
- etc
Following CMIP6 vocab, <frequency> may be one of (where <int> is an integer):

fx subhr
hr day mon yr dec

Note, to deal with the fact that FMS prepends a "_" to the date stamp, I've used underscores between all the time-related fields (is everyone okay with this? @aekiss, @anton-seaice, @minghangli-uni)

Some examples:

MOM6, 1 year of monthly 3D variable access-om3.mom6.3d.agessc.1mon_mean_1900.nc

MOM6, 1 month of monthly 3D variable access-om3.mom6.3d.agessc.1mon_mean_1900_01.nc

MOM6, 1 month of monthly scalar variables access-om3.mom6.1d.scalar.1mon_mean_1900_01.nc

MOM6, 1 month of daily variables or multiple dimensionalities access-om3.mom6.md.heat_budget.1day_mean_1900_01.nc

CICE6, 1 month of daily variables access-om3.cice.1day_mean_1900_01.nc

WW3, 1 month of daily variables access-om3.ww3.1day_mean_1900_01.nc

minghangli-uni commented 1 month ago

Thanks @dougiesquire for wrapping this up!

This looks great to me!

aekiss commented 1 month ago

if the file contains 3 months' worth of data", = YYYY_MM, where MM is the first month

I don't think we get to choose MM, because FMS puts the date stamp in the middle of the file saving period (for means), or at the end (for snapshots) - see https://github.com/COSIMA/access-om2/issues/185#issuecomment-633341425

dougiesquire commented 1 month ago

I don't think we get to choose MM, because FMS puts the date stamp in the middle of the file saving period (for means), or at the end (for snapshots) - see https://github.com/COSIMA/access-om2/issues/185#issuecomment-633341425

Thanks for clarifying. I guess we use whatever FMS decides then? I've updated my comment above

aekiss commented 1 month ago

<variable_info> is only used for MOM output, where it is equal to

<variable_name>_<n_dimension> for single-file-per-variable output

<descriptor> for multi-variable output (see examples below).

Many MOM6 (and MOM5) diagnostics contain _ so we we need to disambiguate from that by only using non-underscore characters either side of the variable name (this is why we used - as the separator in ACCESS-OM2). I also prefer having the dimension part preceding the variable name, so the files will be sorted by dimensionality when listed alphabetically. So how about this instead?

<variable_info> is only used for MOM output, where it is equal to

<n_dimension>.<variable_name> for single-file-per-variable output (e.g. 3d.frazil_heat_tendency)
<n_dimension>.<descriptor> for multi-variable output all of the same dimensionality (e.g. 1d.scalar)
md.<descriptor> for multi-variable output of multiple dimensionalities

I suggest we include md so that all cases include dimension information in a prefix, so they can all be parsed the same way and will also be grouped by dimensionality if listed alphabetically.

dougiesquire commented 1 month ago

Sounds good to me @aekiss. I've updated the above comment accordingly

dougiesquire commented 1 month ago

Ugh, CICE and WW3 use dashes in their timestamp. So to minimise the mods we have to make to those files, we could do <date_stamp> as YYYY-MM-DD-SSSSS. It makes for a lot of separators (although I guess we already had a dash in access-om3), e.g.

access-om3.mom6.2d.tos.1day_mean_1900-01-01.nc

I'll wait for yays/nays before updating the above comment.

aekiss commented 1 month ago

If we're going to modify the filenames in post-processing, we might as well change the _ prior to the date to . and use . as the separator throughout, e.g.

access-om3.mom6.2d.tos.1day.mean.1900-01-01.nc

dougiesquire commented 1 month ago

I agree that looks much tidier, but it would be preferable not to have to post-process all files. It's just another thing that can go wrong.

Using access-om3.mom6.2d.tos.1day_mean_1900-01-01.nc at least means no post-processing of MOM output, and the CICE and WW3 caps could be modified to require no/minimal post-processing.

aekiss commented 1 month ago

Wait, so it's possible to make MOM6 output _1900-01-01 natively? I thought it could only do _1900_01_01.

dougiesquire commented 1 month ago

Yup, I think the _ is only appended to the start of the timestamp block. The separators in between are up to the user. E.g. currently we have:

"access-om3.mom6.h.native%4yr-%2mo", 1,  "months", 1, "days", "time", 1, "months"

dougiesquire commented 1 month ago

We could fork our own version of FMS and remove the prepended "_" 😆

aekiss commented 1 month ago

Sorry, I knew that. Brain is out of juice this late in the day.

dougiesquire commented 1 month ago

Oh wait, hang on... I'm mistaken. Even though we specify this is the diag_table:

"access-om3.mom6.h.native%4yr-%2mo", 1, "months", 1, "days", "time", 1, "months"

The output is this:

access-om3.mom6.h.native_1900_01.nc

Argh!

aekiss commented 1 month ago

I retract my apology :-) We use things like "ocean-3d-pot_rho_0-1yearly-mean-ym%4yr%2mo", 1, "years", 1, "days", "time", 1, "years" in OM2 and get ocean-3d-pot_rho_0-1yearly-mean-ym_1900_01 ie it puts the underscores in. How very "helpful"...

aekiss commented 1 month ago

I didn't realise FMS will replace other delimiters with underscores. That's weirdly controlling.

It also ignores anything following date notation, so you have no choice but to put the date at the end.

dougiesquire commented 1 month ago

Okay, my goal is to make a call and move forward on this today. We can always change things later.

I think I've come round to the idea of post-processing so that everything is consistent

THIS HAS BEEN UPDATED BELOW

MOM6

Output as access-om3.mom6.<n_dimension>.<variable_name/descriptor>.<frequency>.<time_cell_method><datestamp>.nc and post-process away the annoying underscores

E.g.

access-om3.mom6.3d.agessc.1day.mean_1900_01.nc is post-processed to access-om3.mom6.3d.agessc.1day.mean.1900-01.nc

What should we do for multi-variable files that have a range of time cell-methods? Should we just always drop the <time_cell_method> when there are multiple variables as was done for OM2?

CICE6

Output as access-om3.cice.h.<datestamp>.nc (current cap behaviour) and post-process to remove .h, concatenate daily files in months, and add <frequency> and <time_cell_method> info. (It may be more robust to change some of these in the cap, rather that post-process).

E.g.

access-om3.cice.h.1900-01.nc is post-processed to access-om3.cice.1mon.mean.1900-01.nc
access-om3.cice.h.1900-01-*.nc is post-processed to access-om3.cice.1day.mean.1900-01.nc

WW3

I'm not really across these outputs, but with nuopc I think the outputs currently look like access-om3.ww3.hi.YYYY-MM-DD-SSSSS.nc

I don't see why these couldn't be post-processed to a format like access-om3.ww3.1day.mean.YYYY-MM.nc but perhaps @ezhilsabareesh8 or @anton-seaice can comment?

anton-seaice commented 1 month ago

CICE6

Output as access-om3.cice.h.<datestamp>.nc (current cap behaviour) and post-process to remove .h, concatenate daily files in months, and add <frequency> and <time_cell_method> info. (It may be more robust to change some of these in the cap, rather that post-process).

E.g.
* `access-om3.cice.h.1900-01.nc`
  is post-processed to
  `access-om3.cice.1mon.mean.1900-01.nc`

* `access-om3.cice.h.1900-01-*.nc`
  is post-processed to
  `access-om3.cice.1day.mean.1900-01.nc`

We can add the frequency & cell method through the namelist option hist_suffix. To remove the .h we need to patch this one line in the driver. Which means we only need to post processing to concatenate daily data into monthly files.

(i.e. hist_suffix = "1day.mean", "1mon.mean", "x" , "x", "x" where histfreq = "d", "m", "x", "x", "x" )

COSIMA / access-om3

Essential diagnostics #190

Static 2D Grid Data

Monthly 3D Fields

Monthly 3D Squared Fields

monthly 2d fields

monthly 2d fields with different reduction methods

daily 2d fields

daily 2d maximum fields

daily 2d minimum fields

daily scalar snapshots

static 2d grid data

monthly 3d fields

monthly 3d squared fields

monthly 2d fields

monthly 2d fields with different reduction methods

daily 2d fields

daily 2d maximum fields

daily 2d minimum fields

daily scalar snapshots

monthly 2d snapshots

monthly 3d snapshots

MOM6

CICE6

WW3

CICE6