Issue with first hour chemical outputs

[zmoon]

Description:

In our recent runs, we noticed that in our outputs, the concentrations drop to low (species- and time-dependent) values in the first hour, but otherwise look normal. Someone in our group suggested that it could be that the data is getting written out before the initial condition from the previous cycle is read.

This occurs for chemical species in the dyn.f??? and aqm.prod files.

Some details for variables pm25_ave, o3_ave, no2_ave

From `dyn.f???` files :point_down:  Demonstrating the species and time dependence of the low 12z values (domain-mean time series at 12z) :point_down: ``` pm25_ave o3_ave no2_ave time 2019-07-21 12:00:00 0.153723 1.322011 0.019540 2019-07-22 12:00:00 0.146690 1.309766 0.021433 2019-07-23 12:00:00 0.140355 1.300713 0.021501 2019-07-24 12:00:00 0.142170 1.310394 0.022272 2019-07-25 12:00:00 0.153580 1.337568 0.022722 2019-07-26 12:00:00 0.146568 1.347833 0.023340 2019-07-27 12:00:00 0.149825 1.341450 0.020660 2019-07-28 12:00:00 0.153799 1.333693 0.019473 2019-07-29 12:00:00 0.152903 1.332540 0.021367 2019-07-30 12:00:00 0.152387 1.329104 0.022803 2019-07-31 12:00:00 0.151989 1.326439 0.022962 2019-08-01 12:00:00 0.154016 1.332624 0.023092 2019-08-02 12:00:00 0.156643 1.341431 0.023334 2019-08-03 12:00:00 0.156332 1.350270 0.021242 2019-08-04 12:00:00 0.159756 1.344413 0.020541 2019-08-05 12:00:00 0.159777 1.341120 0.023360 2019-08-06 12:00:00 0.160934 1.344035 0.022252 2019-08-07 12:00:00 0.159502 1.348051 0.022330 2019-08-08 12:00:00 0.157596 1.343187 0.021879 2019-08-09 12:00:00 0.154045 1.342660 0.022002 2019-08-10 12:00:00 0.150674 1.342777 0.021307 ``` From `aqm.prod` files 👇 

How To Reproduce:

Additional context:

Screenshots/Logs:

[zmoon]

@drnimbusrain could you add some info about the config etc.?

[drnimbusrain]

These are "community" runs we are doing on Hera, and the example for Zach's plots above is for a continuous/warm start run on Aug 01-10, 2019. Here is the configuration for these runs:

/scratch2/NAGAPE/arl/Patrick.C.Campbell/run_ufs-srw-app_online-cmaq/run_na_test_ufs-srw-app-arl-base_20230215/ush/config.yaml

metadata:
  description: config for Online-CMAQ, AQM_NA_13km, Community mode, 1cycle a day
user:
  RUN_ENVIR: community
  MACHINE: hera
  ACCOUNT: naqfc
workflow:
  USE_CRON_TO_RELAUNCH: true
  CRON_RELAUNCH_INTVL_MNTS: 3
  EXPT_BASEDIR: /scratch1/NCEPDEV/stmp4/Patrick.C.Campbell/expt_dirs
  EXPT_SUBDIR: aqm_cold_aqmna13_Aug01-Aug10_gfsv16_arl-base_no-canopy_fengsha_pt
  PREDEF_GRID_NAME: AQM_NA_13km
  CCPP_PHYS_SUITE: FV3_GFS_v16
  DATE_FIRST_CYCL: '2019080112'
  DATE_LAST_CYCL: '2019081012'
  FCST_LEN_HRS: 24
  PREEXISTING_DIR_METHOD: rename
  VERBOSE: true
  COMPILER: intel
  DIAG_TABLE_TMPL_FN: diag_table_aqm.FV3_GFS_v16
  FIELD_TABLE_TMPL_FN: field_table_aqm.FV3_GFS_v16
nco:
  NET: aqm
workflow_switches:
  RUN_TASK_MAKE_GRID: true
  RUN_TASK_MAKE_OROG: true
  RUN_TASK_MAKE_SFC_CLIMO: true
  RUN_TASK_RUN_POST: false
task_get_extrn_ics:
  EXTRN_MDL_NAME_ICS: FV3GFS
task_get_extrn_lbcs:
  EXTRN_MDL_NAME_LBCS: FV3GFS
  LBC_SPEC_INTVL_HRS: 6
task_run_fcst:
  DT_ATMOS: 180
  LAYOUT_X: 50
  LAYOUT_Y: 34
  BLOCKSIZE: 16
  RESTART_INTERVAL: 1 -1
  WTIME_RUN_FCST: 02:00:00
  QUILTING: true
  PRINT_ESMF: false
task_run_post:
  POST_OUTPUT_DOMAIN_NAME: 793
global:
  DO_ENSEMBLE: false
  NUM_ENS_MEMBERS: 2
  HALO_BLEND: 0
cpl_aqm_parm:
  CPL_AQM: true
  RUN_TASK_AQM_ICS: true
  RUN_TASK_AQM_LBCS: true
  RUN_TASK_NEXUS_GFS_SFC: true
  RUN_TASK_NEXUS_EMISSION: true
  RUN_TASK_FIRE_EMISSION: true
  RUN_TASK_POINT_SOURCE: true
  DO_AQM_CHEM_LBCS: true
  DO_AQM_GEFS_LBCS: true
  AQM_CONFIG_DIR: /scratch2/NCEPDEV/naqfc/RRFS_CMAQ/aqm/epa/data
  AQM_BIO_DIR: /scratch2/NCEPDEV/naqfc/RRFS_CMAQ/aqm/bio
  AQM_BIO_FILE: BEIS_RRFScmaq_C775.ncf
  DO_AQM_DUST: true
  AQM_DUST_DIR: /scratch1/RDARCH/rda-arl-gpu/Barry.Baker/emissions/nexus/FENGSHA/Online_CMAQ
  AQM_DUST_FILE_PREFIX: FENGSHA_p8_10km_inputs
  AQM_DUST_FILE_SUFFIX: .nc
  DO_AQM_CANOPY: false
  AQM_CANOPY_DIR: /scratch2/NCEPDEV/naqfc/RRFS_CMAQ/canopy
  AQM_CANOPY_FILE_PREFIX: gfs.t12z.geo
  AQM_CANOPY_FILE_SUFFIX: .canopy_regrid.nc
  AQM_FIRE_DIR: /scratch2/NCEPDEV/naqfc/Jianping.Huang/Data/emissions/GSCE/RAVE.in.793/RAVE_RT
  AQM_FIRE_FILE_PREFIX: Hourly_Emissions_regrid_NA_13km
  AQM_FIRE_FILE_SUFFIX: _h72.nc
  AQM_RC_FIRE_FREQUENCY: hourly
  DO_AQM_PRODUCT: true
  AQM_LBCS_DIR: /scratch2/NCEPDEV/naqfc/RRFS_CMAQ_NA13km/LBCS/RRFS_NA13km_AM4_v1/
  AQM_LBCS_FILES: am4_bndy.c793.2019<MM>.v1.nc
  AQM_GEFS_DIR: /scratch1/NCEPDEV/rstprod/nexus_emissions/RRFS-BOUNDARY/GEFS_Aerosol
  AQM_GEFS_FILE_PREFIX: gfs
  AQM_GEFS_FILE_CYC: 00
  NEXUS_INPUT_DIR: /scratch1/RDARCH/rda-arl-gpu/Barry.Baker/emissions/nexus
  NEXUS_FIX_DIR: /scratch2/NCEPDEV/naqfc/RRFS_CMAQ/nexus/fix
  NEXUS_GRID_FN: grid_spec_793.nc
  NUM_SPLIT_NEXUS: 3
  COLDSTART: false
  WARMSTART_CYCLE_DIR: /scratch1/NCEPDEV/stmp4/Patrick.C.Campbell/expt_dirs/aqm_cold_aqmna13_Jul21-Jul31_gfsv16_arl-base_no-canopy_fengsha_pt/2019073112

[JianpingHuang-NOAA]

What species?

[JianpingHuang-NOAA]

all chemical species except for CO (just recently added) represent hourly-averaged fields (backward). This is the reason that we need to ignore the first record (i.e., the IC) for model data evaluation and comparison.

[JianpingHuang-NOAA]

For the prod files, we do have hourly-averaged and instantaneous fields (e.g., O3 and PM2.5) for chemical species but all are for surface only.

[KaiWang-NOAA]

@zmoon @drnimbusrain As Jianping pointed out, please use f001-f024 for dyn files and record 01-24 in prod files for your analysis if you're using the hourly-averaged fields, since the hourly average was calculated backward.

[zmoon]

@KaiWang-NOAA @JianpingHuang-NOAA thanks for the explanation. Is this just temporary, or is this the way it's going to be going forward indefinitely?

[JianpingHuang-NOAA]

This is our final product for aqm v7.

On Wed, Mar 8, 2023 at 1:54 PM Zachary Moon @.***> wrote:

@KaiWang-NOAA https://github.com/KaiWang-NOAA @JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA thanks for the explanation. Is this just temporary, or is this the way it's going to be going forward indefinitely?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/67#issuecomment-1460699248, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI7A6XUINSURYS63RSLW3DIXFANCNFSM6AAAAAAVRTFO5Q . You are receiving this because you were mentioned.Message ID: @.***>

[zmoon]

use f001-f024 for dyn files for analysis

@KaiWang-NOAA Is this only for chemical species though? That is, for meteo variables like temperature and surface pressure should we still use f000-f023?

[zmoon]

we need to ignore the first record (i.e., the IC) for model data evaluation and comparison

Our results seem to suggest the first record isn't currently actually the IC though, at least not an IC based on the previous cycle, since the values are too low.

[drnimbusrain]

@KaiWang-NOAA @JianpingHuang-NOAA This seems very strange, and please provide more clarification as to why this is. The values for the first 000 hour are not zero as an initialization would suggest. The values are instead very low and not physical, but they are not zero. What averaging would cause such low values?

As Zach mentioned, the phy and dyn met variables are then inconsistent with the chem variables, and the theory of backwards averaging. All met values make sense across the 024 to 000 next day.

Further, if this is indeed true, how would any community user know this detail. It seems we should find a better fix to this as the output is awkward.

[JianpingHuang-NOAA]

Please ignore the first record for the average chemical fields. The first record doesn’t represent initial condition. For the meteorological variables, all represent instantaneous values. Please make sure both simulations and observations are consistent with the same time when you compare them.

On Wed, Mar 8, 2023 at 4:38 PM Zachary Moon @.***> wrote:

we need to ignore the first record (i.e., the IC) for model data evaluation and comparison

Our results seem to suggest the first record isn't currently actually the IC though, at least not an IC based on the previous cycle, since the values are too low.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/67#issuecomment-1460910984, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI7K2ZAV7FLY4CJWS4DW3D357ANCNFSM6AAAAAAVRTFO5Q . You are receiving this because you were mentioned.Message ID: @.***>

[drnimbusrain]

@JianpingHuang-NOAA Ah, OK, I see the issue. However, still very awkward to have the hourly average chem output not be continuous from the previous cycle. I think this will be an issue with community users of this system.

[JianpingHuang-NOAA]

@drnimbusrain The hourly chem output are continues. Here is an example. 1) 00z cycle run: hourly average chemical fields output at 01z (the 2nd fcst hr), 02z, 03z, 04z, 05z, 06z (i.e., the 6th fcst hr) 2) 06z cycle: hourly average chemical fields at 07z (the 2nd fcst hour), 08z, .....

Please ignore the 1st record for hourly average fields.

For community uses, they can follow what we are doing to verify hourly average O3 and PM2.5. If they want to look at instantaneous fields, they can add any chemical species back by modifying diag_table. We minimize the number of 3-D chemical fields (4 ave species: o3_ave, pm25_ave, no_ave, no2_ave + one instant. for co) to reduce 3-D file size for supporting AQM v7 operational implementation. The file size of each 3D file (dyn) was reduced from 20GB to 1.1. GB.

[drnimbusrain]

Ok Jianping, sounds fine to modify diag table, but expect to have questions from users to get instantaneous back. Currently, hour 000 is not continuous from previous day's forecast.

On Thu, Mar 9, 2023, 8:18 AM JianpingHuang-NOAA @.***> wrote:

@drnimbusrain https://github.com/drnimbusrain The hourly chem output are continues. Here is an example.

1. 00z cycle run: hourly average chemical fields output at 01z (the1st fcst hr), 02z, 03z, 04z, 05z, 06z (i.e., the 6th fcst hr)
2. 06z cycle: hourly average chemical fields at 07z (the 1st fcst hour), 08z, .....

For community uses, they can follow what we are doing to verify hourly average O3 and PM2.5. If they want to look at instantaneous fields, they can add any chemical species back by modifying diag_table. We minimize the number of 3-D chemical fields (4 ave species: o3_ave, pm25_ave, no_ave, no2_ave + one instant. for co) to reduce 3-D file size for supporting AQM v7 operational implementation. The file size of each 3D file (dyn) was reduced from 20GB to 1.1. GB.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/67#issuecomment-1462054475, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLFYNXHRJU7LXYBAOBANODW3HKB5ANCNFSM6AAAAAAVRTFO5Q . You are receiving this because you were mentioned.Message ID: @.***>

[JianpingHuang-NOAA]

That is a good point. I will create a separate diag_table file for the research community users later.

[bbakernoaa]

The first hour should not be zero. It should be the concentration at the previous hour. And in reality we should have the option to do hourly or instantaneous. @chan-hoo Once things calm down can we get this added to the workflow to add the option for instantaneous versus hourly averaged fields. The difference for most variables are negligible for the most part.

[chan-hoo]

@bbakernoaa, sure. I'll reach out to you once all my urgent tasks for AQMv7 are completed.

[bbakernoaa]

@chan-hoo Awesome! Thanks