Implementation of HEMCO in CESM/CAM

This issue is being posted to open discussions about the eventual implementation of HEMCO, the Harmonized Emissions Component (original implementation Keller et al., 2014, implementation in CESM Lin et al., 2021) in the CESM.

Intro

HEMCO is a versatile emissions and data component with the primary purpose of processing emissions data online with inventory selection, masking, and scaling functionality performed as described by a configuration file (see Lin et al., 2021 Figure 2). Emissions data is ingested from individual netCDF files of different inventories and combined into a collection of 3-D emissions containers for each chemical species. HEMCO also is capable of processing generic data containers that are used to provide data (e.g., UV albedo) for GEOS-Chem chemistry within CESM (working PR: https://github.com/ESCOMP/CAM/pull/484).

Motivation to implement HEMCO as an online emissions data source is to remove the need for re-processing offline emission datasets for simulations, instead allowing users to select on-the-fly emission inventories to use, their hierarchy/precedence and effective geographical region, and apply scale factors on-line. The original netCDF input data does not need to be modified, sub-setting and regridding are performed by HEMCO automatically to the target simulation domain.

In its current implementation, HEMCO within CESM can provide emissions and data to GEOS-Chem chemistry (#484) and CAM-chem chemistry (through source mods available here: https://github.com/jimmielin/HEMCO_CESM/tree/development/SourceMods to mo_extfrc.F90 and mo_srf_emissions.F90).

High level code description

The HEMCO-to-CESM interface (HEMCO_CESM) has been implemented as a module within CAM residing under src/hemco. The most recent version of the code is available at: https://github.com/jimmielin/HEMCO_CESM but will eventually be hosted under ESCOMP/HEMCO_CESM. It includes:

the main HEMCO source code as an external sourced from the HEMCO repository maintained by the GEOS-Chem Support Team (https://github.com/geoschem/HEMCO/),
a main driver (hemco_interface) designed to be called from control/cam_comp.F90, and
supporting routines (hco_cam_exports) designed to export data from the HEMCO interface.

HEMCO in CESM runs on its own separate grid and MPI decomposition (as initialized by hco_esmf_grid), currently limited to a rectilinear latitude-longitude grid (see limitations), and regrids its computed data to the CAM grid as described by cam_physics_mesh in recent versions of CAM.

3-D emissions can be computed. ESMF regridding is layer-by-layer, and vertical regridding is only performed when files are read into HEMCO_CESM.

Emissions and data computed are written into the CAM physics buffer (prefixed by HCO_) and retrieved by whatever component within CAM that requires this data.

Requirements

Apart from the main HEMCO source code, the following things were required:

For purposes of implementing HEMCO in CAM, the relevant code for supporting the physics mesh option was backported to an older version of CAM, but this can be easily removed.
USE_ESMF_LIB=true is required for compile and env_mach_specific.xml needs to point to an ESMF version supporting netCDF I/O (ncdfio and not defio)

Limitations and points of discussion

HEMCO uses its own input module, using "vanilla" netCDF routines. HEMCO supports the implementation of different input modules (e.g., for operation within MAPL for the GEOS model, HEMCO uses ExtData). Due to time limitations it is not certain when HEMCO will support CDEPS as the input module.
For HEMCO to replace the current CAM-chem emissions module, input datasets used presently by CAM-chem for emissions need to be ported to HEMCO in a combination of input files (if these emission inventories are unavailable in HEMCO; see Lin et al., 2021, Table 1) and configuration files to combine the inventories into an end result consistent with presently-used configurations (e.g., CEDS + BB4CMIP6). The responsibility of maintaining input data sets and configuration files would need to be discussed as well.

End notes and implementation roadmap

Presently, changes required for HEMCO to operate within CAM and GEOS-Chem are included as part of the GEOS-Chem PR (#484) but can be separated if needed. The sourcemods for CAM-chem are available, but not included by default (they're renamed to .G90 extension so kept part of the repository, but they need to be manually moved to sourcemods for use).

Please refer to a preliminary target branch here: https://github.com/CESM-GC/CAM/tree/HEMCO-CESM_rebased_on_cam6_3_045 based off cam6_3_045. Changes are minimal:

https://github.com/CESM-GC/CAM/commit/6cca709aaef7bc9f80b84ca19493bddb6e6acfc7 for including HEMCO,
changed CAM-chem sources for CAM-chem chemistry to support data read from HEMCO: https://github.com/CESM-GC/CAM/commit/125d9ae9f5f4cf691982e89f0658b08eed1cd6f5, and
everything else self-contained within the hemco external (in src/hemco) and of course the HEMCO external (in src/hemco/HEMCO) containing shared HEMCO code by all models.

I hope this issue can initiate discussion of a roadmap to implement HEMCO within CESM. We suggest implementing it in phases, first bringing HEMCO into CESM so GEOS-Chem can be operational, then implementing a transitional period for CAM-chem users to possibly use HEMCO, finally moving all CAM-chem compsets to full HEMCO usage.

Thank you for your help and please do not hesitate to provide any feedback.

Build instructions (work in progress)

Reference case, data, and results are WIP. I'll update here as soon as possible. Below please find building instructions for a test case with CAM-chem support:

Checkout ESCOMP/CESM at CESM-2.2 release

include modded CAM external. Edit Externals_CAM.cfg to source from testing branch, based off cam6_3_045

[cam]
branch = HEMCO-CESM_rebased_on_cam6_3_045
protocol = git 
repo_url = https://github.com/CESM-GC/CAM
local_path = components/cam
externals = Externals_CAM.cfg
required = True

./manage_externals/checkout_externals
cd cime/scripts
./create_newcase --case ~/hemco_dev_202201 --compset FCSD --res f09_f09_mg17 --run-unsupported --mach cheyenne --project project_number_here (change --case and --project as appropriate)
build with ESMF lib. at case directory, ./case.setup then ./xmlchange USE_ESMF_LIB=true. build.
setup configuration files. Copy necessary configuration files HEMCO_Diagn.rc (dummy) and HEMCO_Config.rc ("the HEMCO configuration file") to $scratch/$case, e.g.,
```
cp /glade/scratch/hplin/hemco_dev_202201/run/HEMCO_Diagn.rc .
cp /glade/scratch/hplin/hemco_dev_202201/run/HEMCO_Config.rc .
```
point location to HEMCO_Config.rc in user_nl_cam:
```
cam_physics_mesh = '/glade/p/cesmdata/cseg/inputdata/atm/cam/coords/fv0.9x1.25_esmf_141008.nc',
hemco_config_file = '/glade/scratch/hplin/hemco_dev_202201/run/HEMCO_Config.rc',
ext_frc_specifier = '',
srf_emis_specifier = '',
```
You may also want to empty ext_frc_specifier, srf_emis_specifier; they will be ignored if HEMCO_CESM is enabled, but emptying it will allow the pre-run phase to skip detection of these files. If the copy of CAM is old and does not have a default cam_physics_mesh value, it should also be provided.
run case for testing (./case.submit)
Reference output: Located in /glade/scratch/hplin/hemco_dev_202201/run/hemco_dev_202201.cam.h1.2005-01-01-00000.nc

Work is ongoing to make reference output match CAM-chem output that should be using the same emission inventory (CEDS) at /glade/scratch/hplin/camchem_run_202203/run

Notes

Note that -hemco defaults to 1 in config_files/definition.xml for testing only and should be changed to 0 before merge:

<entry id="hemco" valid_values="0,1" value="1">
Switch to turn on Harmonized Emissions Component (HEMCO): 0 => no, 1 => yes.
WARNING THIS IS ON BY DEFAULT FOR DEVELOPMENT - DO NOT SUBMIT THIS TO PRODUCTION CODE
</entry>

It is set to 1 for testing all compsets with HEMCO enabled without manually making a compset. CAM-chem with HEMCO compsets will need to be created in the future.

ESCOMP / CAM