ModelLev_Interpolate and MESSy error with NEI emissions

Lukemonr commented 3 months ago

Name and Institution (Required)

Name: Luke Monroe Institution: University of Missouri-Kansas City

Confirm you have reviewed the following documentation

[x] Support guidelines
[x] User manuals
[x] Debugging GEOS-Chem and HEMCO errors
[x] Current and past Github issues

Description of your issue or question

GC-ver = 14.3.0 HEMCO = 3.8.0

I've been poking around this problem for awhile with not much luck. Whenever I attempt to utilize the NEI emissions in the HEMCO_Config.rc file, the model kicks out the error at bottom of this post following its attempt for open the first NEI related file. Turn NEI off and the model runs without issue.

This problem has persisted across resolutions (4x5x 2x2.5, and 0.5x0.625_NA, as well as at 72 and 47 vertical levels). Members in my group using GC14.1.1 with similar HEMCO_Config switch settings have not had this issue. Looking through the change log for HEMCO, I wonder if HEMCO version 3.7.1 changes to when ModelLev_Interpolate and MESSy are called affects how we should be setting up our runs in a way we are missing.

Let me know if you need to me load any more information.

Log output error:

HEMCO: Opening /cluster/pixstor/aec6ry-lab/geos-chem/ExtData//HEMCO/AnnualScalar/v2014-07/AnnualScalar.geos.1x1.nc
HEMCO: Opening /cluster/pixstor/aec6ry-lab/geos-chem/ExtData//HEMCO/NEI2016/v2021-06/2016fh_16j_afdust_adj_0pt1degree_month_01.ncf

HEMCO ERROR:  ModelLev_Interpolate was called but MESSy should have been used: EPA16_BCPI__afdustPEC

HEMCO ERROR: ERROR 0
 --> LOCATION: ModelLev_Interpolate (hco_interp_mod.F90)

HEMCO ERROR: ERROR 16
 --> LOCATION: HCOIO_READ (HCOIO_READ_STD_MOD.F90)

HEMCO ERROR: Error encountered in routine HCOIO_Read!

HEMCO ERROR: Error in HCOIO_DATAREAD called from HEMCO ReadList_Fill: EPA16_BCPI__afdustPEC
 --> LOCATION: ReadList_Fill (HCO_ReadList_Mod.F90)

HEMCO ERROR: Error in ReadList_Fill (3) called from HEMCO ReadList_Read
 --> LOCATION: ReadList_Read (HCO_ReadList_Mod.F90)
 Error in ReadList_Read called from hco_run
.......................................................................................................................................................................
GEOS-Chem ERROR: Error encountered in "HCO_Run"!
 -> at HCOI_GC_Run (in module GeosCore/hco_interface_gc_mod.F90)

THIS ERROR ORIGINATED IN HEMCO!  Please check the HEMCO log file for
additional error messages!
.......................................................................................................................................................................

yantosca commented 3 months ago

Thanks @Lukemonr. Could you attach your HEMCO_Config.rc, HEMCO_Diagn.rc, and entire output log to this issue? I can take a look.

Also, I will transfer this issue to the HEMCO repository.

Lukemonr commented 3 months ago

Sure, not a problem. Here there are. There was a false start in the log where I forgot to adjust the save times for the output files. The error in question applies to the second run in the log file.

HEMCO_Config.rc.txt HEMCO_Diagn.rc.txt NEItestlog.txt

nicholasbalasus commented 3 months ago

Hi @Lukemonr - I can reproduce your error. This is the result of the changes I made in https://github.com/geoschem/HEMCO/commit/daf54d07ed16d0948302070855468b6b27fdd9a5.

Similar to https://github.com/geoschem/geos-chem/pull/1925, the fix is to change the dimensions of the 2D files from xyz to xy. By that I mean in the HEMCO config file:

0 EPA16_BCPI__afdustPEC          $ROOT/NEI2016/v2021-06/2016fh_16j_afdust_adj_0pt1degree_month_$MM.ncf                 PEC        2002-2020/1-12/1/0 RF xyz  kg/m2/s    BCPI   26/221/256/1007/70     1 50

Doing so fixes the problem, but things fail on the 3D NEI files still, as HEMCO tries to push them through MESSY but their sigma levels aren't provided. Looking at the metadata, these are supposed to the N lowest levels of the atmosphere (N = 3 or 11). The easiest fix for this is similar to what we did to handle the 36 level AEIC files with some simple IF statements in hco_interp_mod (starting with this comment and below).

https://github.com/geoschem/HEMCO/blob/f807e1a3828a57d663001297f8d8cf36978040c3/src/Core/hco_interp_mod.F90#L546-L549

@yantosca, if you agree, I can make a PR for this (both the config files in GEOS-Chem and in HEMCO).

nicholasbalasus commented 3 months ago

See https://github.com/geoschem/geos-chem/pull/2213 and https://github.com/geoschem/HEMCO/pull/268.

I tested the fix with this bash script. Without the fix, the simulation fails as described above. With the fix, it runs successfully.

#!/bin/bash

# Source env file
source /n/home06/nbalasus/envs/gcclassic.rocky+gnu10.minimal.env

for fix in yes no
do

# Create run dir
dir="/n/holyscratch01/jacob_lab/nbalasus/test-nei-${fix}-fix/"
mkdir -p "${dir}"
cd "${dir}"
git clone https://github.com/geoschem/GCClassic.git
cd GCClassic
git checkout dev/no-diff-to-benchmark
if [[ $fix = yes ]]; then
    sed -i -e "s|url = https://github.com/geoschem/geos-chem.git|url = https://github.com/nicholasbalasus/geos-chem.git|g" \
        -e "s|url = https://github.com/geoschem/hemco.git|url = https://github.com/nicholasbalasus/HEMCO.git|g" .gitmodules
    git submodule sync
fi
git submodule update --init --recursive
cd src/GEOS-Chem
if [[ $fix = yes ]]; then
    git checkout bugfix/nei-emissions
    cd ../../src/HEMCO
    git checkout bugfix/nei-emissions
fi
cd ../../run

# Compile GC for Full-Chem
run_dir="gc_run"
c="1\n1\n1\n1\n1\n${dir}\n${run_dir}\nn\n"
printf ${c} | ./createRunDir.sh
cd "${dir}/${run_dir}/build"
cmake ../CodeDir -DRUNDIR=.. -DMECH=fullchem
make -j
make install
cd "${dir}/${run_dir}"

# Modify config files
sed -i -e "s|NEI2016_MONMEAN        :       false|NEI2016_MONMEAN        :       true|g" HEMCO_Config.rc

# Modify and submit run script
cp runScriptSamples/operational_examples/harvard_cannon/geoschem.run .
sed -i -e "s|-c 8|-c 24|g" \
    -e "s|15000|64000|g" geoschem.run
sbatch geoschem.run

done

Lukemonr commented 3 months ago

Thanks for patching this up quickly!

yantosca commented 3 months ago

@Lukemonr @nicholasbalasus: Am closing this issue as the HEMCO PR #268 and GEOS-Chem https://github.com/geoschem/geos-chem/pull/2213 are now merged into the no-diff-to-benchmark development stream. This will ship with GEOS-Chem 14.3.1 / HEMCO 3.8.1.

nicholasbalasus commented 3 months ago

Thanks @yantosca!

geoschem / HEMCO