PM2.5 underprediction by the latest online-cmaq SRW workflow

[JianpingHuang-NOAA]

I completed a 22-day test run (July 4-25, 2022) by using the SRW workflow with a hash number of 5fdee93 (https://github.com/ufs-community/ufs-srweather-app/tree/online-cmaq) and noticed that PM2.5 was under-predicted especially in EPA-defined Regions , R6 and R4 while O3 predictions are fine.

This issue was observed by Kai recent run last week on Cactus.

After I turned on printing out ESMF log file, and also noticed the fire PM2.5 composition emissions are not read in correctly as illustrated by the following line example

================================ 20230126 231211.602 INFO PET0000 AQM: emissions: fire: read: PNCOM : min/max = 0.0000000 0.0000000

More details can be found from personal workflow and run dir on Dogwood

1) My workflow/ush: /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.53 (Dogwood)

2) An example case run for 20220717: /lfs/h2/emc/aqmtemp/para/tmp/run_fcst.id_1674697912_2022071712

@chan-hoo @bbakernoaa @KaiWang-NOAA

[JianpingHuang-NOAA]

C54 is our new run and C43 is the base case with the old workflow. Both C43 and C54 use the same ufs-weather-model code.

[chan-hoo]

@JianpingHuang-NOAA, what is the hash of the 'C43' workflow?

[JianpingHuang-NOAA]

3027910

On Fri, Jan 27, 2023 at 6:21 AM Chan-Hoo.Jeon-NOAA @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA, what is the hash of the 'C43' workflow?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/58#issuecomment-1406363395, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI2BWYNGBQ3AVKJBKDDWUOVS3ANCNFSM6AAAAAAUIDSXU4 . You are receiving this because you were mentioned.Message ID: @.***>

[chan-hoo]

The major differences (from the scientific perspective) between those two hashes are the new hash of nexus (workflow hash: f12cb5e) and the point source change (workflow hash: 2a19b16). @bbakernoaa, can you take a look at this issue?

[bbakernoaa]

I'll take a look but this is pretty clearly something with fires.

[ytangnoaa]

I completed a 22-day test run (July 4-25, 2022) by using the SRW workflow with a hash number of 5fdee93 (https://github.com/ufs-community/ufs-srweather-app/tree/online-cmaq) and noticed that PM2.5 was under-predicted especially in EPA-defined Regions , R6 and R4 while O3 predictions are fine.

This issue was observed by Kai recent run last week on Cactus.

After I turned on printing out ESMF log file, and also noticed the fire PM2.5 composition emissions are not read in correctly as illustrated by the following line example

================================ 20230126 231211.602 INFO PET0000 AQM: emissions: fire: read: PNCOM : min/max = 0.0000000 0.0000000

More details can be found from personal workflow and run dir on Dogwood

1. My workflow/ush: /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.53 (Dogwood)
2. An example case run for 20220717: /lfs/h2/emc/aqmtemp/para/tmp/run_fcst.id_1674697912_2022071712

@chan-hoo @bbakernoaa @KaiWang-NOAA

The fire emission appeared in other PET (0041 for instance) /lfs/h2/emc/aqmtemp/para/tmp/run_fcst.id_1674697912_2022071712/PET0041.ESMF_LogFile: line 5510780: 20230126 232657.462 INFO PET0041 AQM: emissions: fire: read: POC : min/max = 0.0000000 0.27282693E-09 though we need to ensure that it was processed correctly

[HaixiaLiu-NOAA]

The configuration of C54: same codes as C43 and latest workflow

[JianpingHuang-NOAA]

@BrianCurtis-NOAA Do you have an official UFSWM without personal ID for a test?

[BrianCurtis-NOAA]

My fork is official UFSWM minus my debug fixes. The official UFSWM has the debug issues still.

[JianpingHuang-NOAA]

Please provide hash # and link

[BrianCurtis-NOAA]

https://github.com/BrianCurtis-NOAA/ufs-weather-model/tree/bugfix/revert-debug

hash: https://github.com/BrianCurtis-NOAA/ufs-weather-model/commit/d3a84c6c7f44d1faf9bcec9235709c8e5983f2b8

[BrianCurtis-NOAA]

I just saw @yangfanglin comment from the meeting. Yes the runs are reproducible from regression testing. @JianpingHuang-NOAA Where are the two git directions I should be comparing located at?

[HaixiaLiu-NOAA]

@JianpingHuang-NOAA Please provide your local directories to Brian for him to compare

[JianpingHuang-NOAA]

The machine was switched. Is Brian able to access Dogwood now?

On Mon, Jan 30, 2023 at 11:20 AM HaixiaLiu @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA Please provide your local directories to Brian for him to compare

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/58#issuecomment-1408927870, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PIZLO7DHZ6XHEWQSOWTWU7S4RANCNFSM6AAAAAAUIDSXU4 . You are receiving this because you were mentioned.Message ID: @.***>

[JianpingHuang-NOAA]

Please ask Brian to ensure the regression test (RT) works properly. That is the key rather than to check my local working directories. The problem is not from my side.

On Mon, Jan 30, 2023 at 11:48 AM Jianping Huang - NOAA Affiliate < @.***> wrote:

The machine was switched. Is Brian able to access Dogwood now?

On Mon, Jan 30, 2023 at 11:20 AM HaixiaLiu @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA Please provide your local directories to Brian for him to compare

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/58#issuecomment-1408927870, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PIZLO7DHZ6XHEWQSOWTWU7S4RANCNFSM6AAAAAAUIDSXU4 . You are receiving this because you were mentioned.Message ID: @.***>

[chan-hoo]

@JianpingHuang-NOAA, please point us to your directories for C43 and C54. As I mentioned in the last meeting, I'd like to compare the input files of those two cases. If they are identical, this should not be an issue on the workflow itself.

[JianpingHuang-NOAA]

@Chan-Hoo Jeon - NOAA Affiliate @.***>

1) Workflow C54: /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.54/ Dogwood

2) INPUT/PUTS/ /lfs/h2/emc/aqmtemp/para/com/aqm/v7.0/c55* Dogwood.

I have provided those information when I created the issue, but no one has checked so far.

For C43, Kai did a test on Dogwood.. Kai, can you provide your local directories on Dogwood?

As I said, since the machine was switched today, we need to re-checkout everything on Cactus. That is what I am doing now.

On Mon, Jan 30, 2023 at 11:54 AM Chan-Hoo.Jeon-NOAA < @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA, please point us to your directories for C43 and C54. As I mentioned in the last meeting, I'd like to compare the input files of those two cases. If they are identical, this should not be an issue on the workflow itself.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/58#issuecomment-1408977141, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI2L54S5AM2A52YXQU3WU7W4DANCNFSM6AAAAAAUIDSXU4 . You are receiving this because you were mentioned.Message ID: @.***>

[chan-hoo]

@JianpingHuang-NOAA, You didn't provide any info for 'C43' and 'C54' but for 'C53'. Dogwood is not available. Please copy them to Cactus.

[chan-hoo]

@JianpingHuang-NOAA, @KaiWang-NOAA, What I want to check is not the workflows of C43 and C54 but their run directories where the input files exist (for example, 07/17/2022). Please point me to the run directories for the same date on Cactus.

[JianpingHuang-NOAA]

@Chan-Hoo Jeon - NOAA Affiliate @.***> INPUT/OUTPUTS C43: /NCEPDEV/emc-naqfc/5year/Jianping.Huang/AQMv7/aqm.v7.0.c43 (HPSS) C53: /NCEPDEV/emc-naqfc/5year/Jianping.Huang/AQMv7/aqm.v7.0.c53 (HPSS) C54: /NCEPDEV/emc-naqfc/5year/Jianping.Huang/AQMv7/aqm.v7.0.c54 (HPSS)

1)C43 Workflow /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.36/ush/config.yaml_c43xxx (Cactus) UFSWM: /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.36/sorc/ufs-weather-model

2) C54 Workflow /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.54/ush (both Cactus and Dogwood)

ufs-weather-model /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.54/sorc/ufs-weather-model

Note: C53 and C54 simulations were completed on Dogwood. and C43 was done on Cactus.

On Mon, Jan 30, 2023 at 12:04 PM Chan-Hoo.Jeon-NOAA < @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA, You didn't provide any info for 'C43' and 'C54' but for 'C53'. Dogwood is not available. Please copy them to Cactus.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/58#issuecomment-1408990882, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI5QJSG7DOXY6J3MVZDWU7YBRANCNFSM6AAAAAAUIDSXU4 . You are receiving this because you were mentioned.Message ID: @.***>

[chan-hoo]

@JianpingHuang-NOAA, I can't find any input files on the hpss directory: [hpsscore1]/NCEPDEV.../aqm.v7.0.c43/20220717/00->ls /NCEPDEV/emc-naqfc/5year/Jianping.Huang/AQMv7/aqm.v7.0.c43/20220717/00: aqm.20220717t00z.awpozcon.grib2.tar aqm.20220717t00z.max_ave.tar
aqm.20220717t00z.awpozcon.grib2.tar.idx aqm.20220717t00z.max_ave.tar.idx
aqm.20220717t00z.bc_o3_pm25.tar aqm.20220717t00z.met_sfc.tar
aqm.20220717t00z.bc_o3_pm25.tar.idx aqm.20220717t00z.met_sfc.tar.idx
aqm.20220717t00z.chem_sfc.tar aqm.20220717t00z.phy.tar
aqm.20220717t00z.chem_sfc.tar.idx aqm.20220717t00z.phy.tar.idx
aqm.20220717t00z.dyn.tar aqm.20220717t00z.pm25.grib2.tar
aqm.20220717t00z.dyn.tar.idx aqm.20220717t00z.pm25.grib2.tar.idx
aqm.20220717t00z.emis.tar aqm.20220717t00z.pm25.tar
aqm.20220717t00z.emis.tar.idx aqm.20220717t00z.pm25.tar.idx
aqm.20220717t00z.input.tar aqm.20220717t00z.restart.tar
aqm.20220717t00z.input.tar.idx aqm.20220717t00z.restart.tar.idx

htar -tvf /NCEPDEV/emc-naqfc/5year/Jianping.Huang/AQMv7/aqm.v7.0.c43/20220717/ 00/aqm.20220717t00z.input.tar [connecting to hpsscore1.fairmont.rdhpcs.noaa.gov/1217] Reading Index File from HPSS... /NCEPDEV/emc-naqfc/5year/Jianping.Huang/AQMv7/aqm.v7.0.c43/20220717/00/aqm.20220717t0 HTAR: -rw-r--r-- jianping.huang/physics 7545 2023-01-08 05:42 aqm.t00z.gfs_ctrl.nc HTAR: -rw------- jianping.huang/emc 256 2023-01-19 13:31 /var/tmp/pbs.28674914.cbqs01/HTAR_CF_CHK_633_1674135081 HTAR: Listing complete for /NCEPDEV/emc-naqfc/5year/Jianping.Huang/AQMv7/aqm.v7.0.c43/20220717/00/aqm.20220717t00z.input.tar, 2 files 2 total objects

I'd like to compare the fire emission data and LBC files between C43 and C54.

[chan-hoo]

@JianpingHuang-NOAA, On C43:

[ufs-weather-model]
protocol = git
#repo_url = https://github.com/ufs-community/ufs-weather-model
repo_url = https://github.com/rmontuoro/ufs-weather-model
#branch = develop
branch = feature/point-source
#hash = 3610672
local_path = sorc/ufs-weather-model
required = True

On C54:

[ufs-weather-model]
protocol = git
#repo_url = https://github.com/ufs-community/ufs-weather-model
#branch = develop
#hash = e051e0e
repo_url = https://github.com/BrianCurtis-NOAA/ufs-weather-model
hash = d3a84c6
local_path = sorc/ufs-weather-model
required = True

I can't find any evidence that you used the same ufs-weather-model between those two.

[JianpingHuang-NOAA]

C43: /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.36/sorc/ufs-weather-model

*commit 9afb82bfc00f4f143d0fba86fcdeb9d6327e76f2Author: Raffaele Montuoro @. @.>>Date: Thu Dec 22 17:58:27 2022 +0000*

C54: /lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.54/sorc/ufs-weather-model

*commit d3a84c6c7f44d1faf9bcec9235709c8e5983f2b8Author: Brian Curtis @. @.>*> Date: Thu Jan 26 16:18:24 2023 +0000

C43 code was checked out from Rafffaele's Github and C54 from Brian's reverted code. Basically they are the same. If they are not, please check with Brian.

On Mon, Jan 30, 2023 at 12:48 PM Chan-Hoo.Jeon-NOAA < @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA, On C43: [ufs-weather-model] protocol = git

repo_url = https://github.com/ufs-community/ufs-weather-model

repo_url = https://github.com/rmontuoro/ufs-weather-model Specify either a branch name or a hash but not both.

branch = develop

branch = feature/point-source

hash = 3610672

local_path = sorc/ufs-weather-model required = True

On 54: [ufs-weather-model] protocol = git

repo_url = https://github.com/ufs-community/ufs-weather-model

Specify either a branch name or a hash but not both.

branch = develop

hash = e051e0e

repo_url = https://github.com/BrianCurtis-NOAA/ufs-weather-model hash = d3a84c6 local_path = sorc/ufs-weather-model required = True

I can't find any evidence that you used the same ufs-weather-model between those two.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/58#issuecomment-1409055346, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI5W2SMRRN3EL6HFOODWU75FDANCNFSM6AAAAAAUIDSXU4 . You are receiving this because you were mentioned.Message ID: @.***>

[chan-hoo]

I suspect that the difference came from the different ufs weather models. Here is the point. For C54, you cloned the specific hash of the ufs weather model from Brian's repo. However, for C43, you cloned the specific branch (NOT hash) of the ufs weather model from Raffaele's repo. You might think they should be the same, but it totally depends on when you cloned it. If Raffaele modified something between C43 and C54, they would not be the same each other any more.

[BrianCurtis-NOAA]

I'm still unsure of whether the PM2.5 issue was seen with my branch/hash. @JianpingHuang-NOAA can you update me about that? I know you ran 30 hours, but was unsure if you knew from that if the PM2.5 issue was fixed/addressed.

If it's not seen there, can we run the 30 day test to see if it shows there?

[JianpingHuang-NOAA]

I am testing the branch that you provided last Friday with the latest workflow on Cactus. It looks good so far based on the checking with ncview. I will update the results once the simulations are done for the period of July 15-25, 2022. Thanks ! @Brian Curtis - NOAA Affiliate @.***>

On Mon, Jan 30, 2023 at 2:18 PM Brian Curtis @.***> wrote:

I'm still unsure of whether the PM2.5 issue was seen with my branch/hash. @JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA can you update me about that? I know you ran 30 hours, but was unsure if you knew from that if the PM2.5 issue was fixed/addressed.

If it's not seen there, can we run the 30 day test to see if it shows there?

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/58#issuecomment-1409202724, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI2NC7XKCOVYHIRS7HLWVAHYPANCNFSM6AAAAAAUIDSXU4 . You are receiving this because you were mentioned.Message ID: @.***>

[JianpingHuang-NOAA]

@BrianCurtis-NOAA The test was completed with your branch. The results are similar to that of C43 except for some small difference on July 7-13, 2022. Please consider it to merge it to the UFSWM. Thanks !

I am closing it now

[BrianCurtis-NOAA]

We typically close issues as they are fixed in the official repository. Can we keep this open so we can track this inside a PR?

[JianpingHuang-NOAA]

I am reopening it now. I was considering the code merge is another issue which may require further refinement of the regression test since it failed to detect the problem last time

[BrianCurtis-NOAA]

There is a correction to current testing methods that can be made to help stop the issue we had from happening again. I can get into that privately or at the next group meeting.

[HaixiaLiu-NOAA]

The issue was fixed already, so I am closing this one