Limited RAVE Emissions for NA13km Domain on WCOSS2

drnimbusrain commented 2 years ago

@chan-hoo @JianpingHuang-NOAA @KaiWang-NOAA

The ability to run Online-CMAQ on WCOSS2 is indeed great news! Thank you!

However, I noticed that there is only a very limited RAVE emissions dataset available on WCOSS2 for running the NA13 km domain: /lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT 20190805 20190806 20190807

Can the entire dataset for the NA13 domain be brought over from Hera to WCOSS2 soon for longer testing? Specifically, the data for August-September 2020 case would be excellent.

Thank you.

JianpingHuang-NOAA commented 2 years ago

What time period data do you need? We do not have real-time RAVE data files for the large domain until next Monday. But we do have RAVE data for the large domain but based on the interpolation of CONUS data.

Jianping

On Wed, Oct 12, 2022 at 12:02 PM Patrick Campbell @.***> wrote:

@chan-hoo https://github.com/chan-hoo @JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA @KaiWang-NOAA https://github.com/KaiWang-NOAA

The ability to run Online-CMAQ on WCOSS2 is indeed great news! Thank you!

However, I noticed that there is only a very limited RAVE emissions dataset available on WCOSS2 for running the NA13 km domain:

/lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT 20190805 20190806 20190807

Can the entire dataset for the NA13 domain be brought over from Hera to WCOSS2 soon for longer testing?

Thank you.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PIZMXQXE4K3XUB4YPCTWC3OJVANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA Thank you for your reply. I am running a base test case for September 2020, and have moved the data manually to my own WCOSS2 location. However, it of course would be good to similarly house all Online-CMAQ inputs on WCOSS2 the same as on Hera.

KaiWang-NOAA commented 2 years ago

Hi Patrick,

Just wanted to make sure that you got the latest version of RAVE data. The files are located at /scratch2/NCEPDEV/naqfc/Kai.Wang/GBBEPX_fire/RAVE_NA13km/forUFSCMAQrun/G793 on Hera. For example, Hourly_Emissions_regrid_NA_13km_20201004_t12z_h72.nc is for 72 hour forecast and Hourly_Emissions_regrid_NA_13km_20201004_t12z_h24.nearest.nc is for 24 hour. The RAW emissions are in /scratch2/NCEPDEV/naqfc/Kai.Wang/GBBEPX_fire/RAVE_NA13km/OUTPUT1

Thanks, Kai

On Thu, Oct 13, 2022 at 10:00 AM Patrick Campbell @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA Thank you for your reply. I am running a base test case for September 2020, and have moved the data manually to my own WCOSS2 location. However, it of course would be good to similarly house all Online-CMAQ inputs on WCOSS2 the same as on Hera.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1277660640, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2A5ALXZTMSW6HOCZ7LEMOTWDAIV5ANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

drnimbusrain commented 2 years ago

@KaiWang-NOAA Thank you, as we indeed did not have the latest version of RAVE data. Can those files please be made available at the WCOSS location?

AQM_FIRE_DIR: /lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT AQM_FIRE_FILE: Hourly_Emissions_regrid_NA_13km AQM_FIRE_FILE_SUFFIX: _h72.nc

KaiWang-NOAA commented 2 years ago

Patrick,I don't have the write permission to /lfs/h2/emc/lam, so I made a copy at /lfs/h2/emc/physics/noscrub/kai.wang/RAVE_fire/202009 on WCOSS2.

On Thu, Oct 13, 2022 at 10:44 AM Patrick Campbell @.***> wrote:

@KaiWang-NOAA https://github.com/KaiWang-NOAA Thank you, as we indeed did not have the latest version of RAVE data. Can those files please be made available at the WCOSS location?

AQM_FIRE_DIR: /lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT AQM_FIRE_FILE: Hourly_Emissions_regrid_NA_13km AQM_FIRE_FILE_SUFFIX: _h72.nc

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1277730616, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2A5ALS6IXVKFEXXHVHZKNDWDAN4XANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

drnimbusrain commented 2 years ago

Thank you @KaiWang-NOAA

drnimbusrain commented 2 years ago

@KaiWang-NOAA It seems your RAVE emissions have been moved or deleted from this location, and our Sep 2020 runs are failing on WCOSS2 again. Can you update us on this?

@bbakernoaa

bbakernoaa commented 2 years ago

@KaiWang-NOAA @HaixiaLiu-NOAA We really need to get this sorted out. We cannot have emission files moving directories without letting everyone know.

We really need to think about moving all emissions (anthropogenic) to a fix directory and the fire emissions to a static directory and/or HPSS and retrieve them in the workflow. This would cut down on errors like this drastically.

KaiWang-NOAA commented 2 years ago

Patrick, the folder has been moved to /lfs/h2/emc/physics/noscrub/kai.wang/RAVE_fire/RAVE_CONUS/202009 in order to differentiate with the new RAVE NA dataset from NESDIS. Sorry about that.

On Mon, Oct 24, 2022 at 9:59 AM Patrick Campbell @.***> wrote:

@KaiWang-NOAA https://github.com/KaiWang-NOAA It seems your RAVE emissions have been moved or deleted from this location, and our Sep 2020 runs are failing on WCOSS2 again. Can you update us on this?

@bbakernoaa https://github.com/bbakernoaa

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1289079991, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2A5ALVNGMJNS5ZOEFGENPDWE2I23ANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

drnimbusrain commented 2 years ago

@KaiWang-NOAA Why is it in "RAVE_CONUS"? We are ONLY running the larger NA domain. Where are those files?

KaiWang-NOAA commented 2 years ago

Sorry for the typo. It should be /lfs/h2/emc/physics/noscrub/kai.wang/RAVE_fire/RAVE_NA/202009

On Mon, Oct 24, 2022 at 10:07 AM Kai Wang - NOAA Affiliate < @.***> wrote:

Patrick, the folder has been moved to /lfs/h2/emc/physics/noscrub/kai.wang/RAVE_fire/RAVE_CONUS/202009 in order to differentiate with the new RAVE NA dataset from NESDIS. Sorry about that.

On Mon, Oct 24, 2022 at 9:59 AM Patrick Campbell @.***> wrote:

@KaiWang-NOAA https://github.com/KaiWang-NOAA It seems your RAVE emissions have been moved or deleted from this location, and our Sep 2020 runs are failing on WCOSS2 again. Can you update us on this?

@bbakernoaa https://github.com/bbakernoaa

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1289079991, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2A5ALVNGMJNS5ZOEFGENPDWE2I23ANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

KaiWang-NOAA commented 2 years ago

Barry, Thanks for pointing this out. The new hourly real time RAVE emissions from NESDIS just kicked in last week. We did some major updates of the scripts/procedures to deal with those data. So it is necessary to make changes of folders and also it's taking some time for us to stabilize the data flow. In the near future, we will put the fire emissions in a more static location on HPSS and update your guys. Thanks.

On Mon, Oct 24, 2022 at 10:02 AM Barry Baker @.***> wrote:

@KaiWang-NOAA https://github.com/KaiWang-NOAA @HaixiaLiu-NOAA https://github.com/HaixiaLiu-NOAA We really need to get this sorted out. We cannot have emission files moving directories without letting everyone know.

We really need to think about moving all emissions (anthropogenic) to a fix directory and the fire emissions to a static directory and/or HPSS and retrieve them in the workflow. This would cut down on errors like this drastically.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1289085169, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2A5ALW7E7FSPN6AXO6EWGTWE2JIDANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

KaiWang-NOAA commented 2 years ago

Hi all, the NRT RAVE fire emissions over the NA domain together with 202009 (for retro case) are now available at /NCEPDEV/emc-naqfc/2year/Kai.Wang/RAVE_fire/RAVE_NA on HPSS.

On Mon, Oct 24, 2022 at 10:18 AM Kai Wang - NOAA Affiliate < @.***> wrote:

Barry, Thanks for pointing this out. The new hourly real time RAVE emissions from NESDIS just kicked in last week. We did some major updates of the scripts/procedures to deal with those data. So it is necessary to make changes of folders and also it's taking some time for us to stabilize the data flow. In the near future, we will put the fire emissions in a more static location on HPSS and update your guys. Thanks.

On Mon, Oct 24, 2022 at 10:02 AM Barry Baker @.***> wrote:

@KaiWang-NOAA https://github.com/KaiWang-NOAA @HaixiaLiu-NOAA https://github.com/HaixiaLiu-NOAA We really need to get this sorted out. We cannot have emission files moving directories without letting everyone know.

We really need to think about moving all emissions (anthropogenic) to a fix directory and the fire emissions to a static directory and/or HPSS and retrieve them in the workflow. This would cut down on errors like this drastically.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1289085169, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2A5ALW7E7FSPN6AXO6EWGTWE2JIDANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

drnimbusrain commented 2 years ago

@KaiWang-NOAA Thank you for putting all RAVE emissions there.

However, at some point hopefully @chan-hoo (or whoever has write access) can move all RAVE emissions to the following Online-CMAQ WCOSS2 location for common emissions: /lfs/h2/emc/lam/noscrub/RRFS_CMAQ/emissions/GSCE/RAVE.in.793/RAVE_RT We really shouldn't have to do extra step to pull down emissions from HPSS and store locally for the official workflow on each system, correct?

FYI. The current workflow and aqm.rc template needs the emissions stored in YYYYMMDD sub-directory format here too. See: https://github.com/ufs-community/ufs-srweather-app/blob/online-cmaq/ush/create_aqm_rc_file.py#L67

chan-hoo commented 2 years ago

@drnimbusrain, due to the space issue, I put some specific data for the engineering test (sample_config/WE2E_test) in the 'noscrub/RRFS_CMAQ' directory. Otherwise, I'll receive a warning from the system administrator. I don't have permission to write data files in the centralized locations on HPCs.

drnimbusrain commented 2 years ago

@chan-hoo Thank you for clarifying. Seems this is a larger issue then with WCOSS2 for us then. @bbakernoaa

drnimbusrain commented 2 years ago

@KaiWang-NOAA @JianpingHuang-NOAA @chan-hoo @HaixiaLiu-NOAA

We have been continuing our online-cmaq NA domain run for Sep 2020 with latest model and RAVE fire emissions provided by @KaiWang-NOAA . While most of the areas of CONUS look OK after spin-up, there seems to be an issue with the input fire emissions magnitude and extremely low PM2.5 in the west. You can see our regional analysis here: https://docs.google.com/presentation/d/1AKhSlbVjPsy9aIBVyPRUYwQfoEgtCe4tFwxtBT4neLE/edit?usp=sharing

For example, while the non-fire regions look OK in the northeast:

The fire regions in the west and CA appear to account for the RAVE emissions, but they are much too low:

The spatial mean bias plots for PM2.5 also indicate an issue in the western fire regions:

RAVE fire emissions are being read from: /lfs/h2/emc/physics/noscrub/kai.wang/RAVE_fire/RAVE_NA/202009

bbakernoaa commented 2 years ago

@KaiWang-NOAA from the figures you can see that the emissions are being read in and processed but the magnitude is very low. This indicates that the units may be different than indicated in the aqm.rc. We need to get to the bottom of this for the NEW fire emission product and not the test data previously used.

KaiWang-NOAA commented 2 years ago

@bbakernoaa @drnimbusrain @JianpingHuang-NOAA I compiled several slides that I did before when QA the RAVE data regridding. https://docs.google.com/presentation/d/1Xq8rwSXsqwRwOujKaypH8VbM9JtA0y70Vfa5XwBKxSI/edit#slide=id.g1841e488be7_0_193. See slides 6 and 8 for the fluxes with the units consistent with the model ready emissions. It looks to be ok for me. We may need to look further at what exactly cause the low PM2.5

KaiWang-NOAA commented 2 years ago

@bbakernoaa @bbakernoaa Here are some slides that Jianping presented at the end of Sept. which covers the same Sept. 2020 episode. https://docs.google.com/presentation/d/12Jse29OvM4eqcB6Dkev1U_N2SCkSh0Pw/edit#slide=id.g15c34760966_0_118 c8 used the same set of RAVE emissions with cold start from Sept. 1, 2020. As you can see on slide 4, the PM2.5 concentration kept building up, which is much different than your results.

I'm not sure if @JianpingHuang-NOAA still has the run folder with related configure/aqm.rc files. It's worth looking into it.

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA This would be very helpful if you could provide any information on your Sample Config or aqm.rc file for your previous run. While you PM2.5 is still very low (purple line CONUS ave ~ 10 ug/m3 by Sep 18) and likely not doing well in the fire regions (although there is no regional analysis for R9 for example):

It is indeed about 2x higher than my run (green line CONUS ave ~ 5 ug/m3 by Sep 18)

However @KaiWang-NOAA , I wouldn't say it is "much different than our results", as the 2x higher PM2.5 is still likely much lower than the fire regions PM2.5 during this period, and thus indicates potentially a similar problem with RAVE fire emissions in both our runs. We really should discuss to get to the bottom of this.

JianpingHuang-NOAA commented 2 years ago

@Patrick Campbell - NOAA Affiliate @.***> This is the one that I used for the v7.0.c8 run on WCOSS2 (/lfs/h2/emc/physics/noscrub/jianping.huang/nwdev/packages/aqm.v7.0.4/regional_workflow/ush/config.sh). You are right. The UFS-AQM predictions were lower than operational forecasts (NAM-CMAQ) which used BlueSky fire emissions. We had a meeting with the groups of Shobha and Xiaoyang on RAVE emissions during Sept 12-18,

To me, it seems that RAVE data do not represent the real wildfire emissions well, but Fangjun (Xiaoyang's group) and RAVE suggested that high PM2.5 was a combined effect of wildfire emissions and local circulation during that time period.

On Mon, Nov 7, 2022 at 11:34 AM Patrick Campbell @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA This would be very helpful if you could provide any information on your Sample Config or aqm.rc file for your previous run. While you PM2.5 is still very low (purple line CONUS ave ~ 10 ug/m3 by Sep 18) and likely not doing well in the fire regions (although there is no regional analysis for R9 for example): [image: image] https://user-images.githubusercontent.com/26631222/200363193-27d56db5-1c1c-48dd-9c5d-9bfbebd0c16f.png

It is indeed about 2x higher than my run (green line CONUS ave ~ 5 ug/m3 by Sep 18) [image: image] https://user-images.githubusercontent.com/26631222/200363513-81017015-3629-4b37-9715-c93de6c7f2b4.png

However @KaiWang-NOAA https://github.com/KaiWang-NOAA , I wouldn't say it is "much different than our results", as the 2x higher PM2.5 is still likely much lower than the fire regions PM2.5 during this period, and thus indicates potentially a similar problem with RAVE fire emissions in both our runs. We really should discuss to get to the bottom of this.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1305874037, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PIZ3FUU5X7UO42XWPLTWHEVPFANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA Yes, thank you for your comments. I agree there is some issue in RAVE fire emissions during this time, but not sure of the cause. It seems a larger issue than what has initially been suggested. If you look at the region R9 or R10, or really any of our analysis, it is not even realistic:

bbakernoaa commented 2 years ago

@JianpingHuang-NOAA it would be helpful if you could repeat this experiment using the latest code and files to make sure that we are not setting this up incorrectly. What we are seeing here is a massive under prediction of fires that is beyond what has been shown in any of our simulations with online or offline versions. Again we do see a signal but it is far below what we expect and far away from the source region everything looks more reasonable. Please repeat even a small section of this experiment. It would drastically help settle this issue

JianpingHuang-NOAA commented 2 years ago

@Barry Baker - NOAA Federal @.> I will do. Meanwhile, I was working on the NRT testing these two days and noticed the NEXUS has some issues with the latest workflow. I am still testing it. @Patrick Campbell - NOAA Affiliate @.> Can you point me to the location of your runs again?

On Tue, Nov 8, 2022 at 9:37 AM Barry Baker @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA it would be helpful if you could repeat this experiment using the latest code and files to make sure that we are not setting this up incorrectly. What we are seeing here is a massive under prediction of fires that is beyond what has been shown in any of our simulations with online or offline versions. Again we do see a signal but it is far below what we expect and far away from the source region everything looks more reasonable. Please repeat even a small section of this experiment. It would drastically help settle this issue

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1307321712, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI2CIXG264JIHCDYARDWHJQS3ANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA Thank you for having a check. My current test run continuing on in Sep 18-20, 2020, is found here: _/lfs/h2/oar/ptmp/Patrick.C.Campbell/expt_dirs/aqm_cold_aqmna13_1day_arl_base_wetdepfix_firex1000_2020091820200920

Please ignore in the aqm.rc file that I am testing an arbitrary increase (x1000) in fire emissions to gauge the impact on the extremely low PM2.5 concentrations.

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA @KaiWang-NOAA You can see how much better our operational model with GBBEPx is doing compared to my online-cmaq run with RAVE for R9 and R10:

@JianpingHuang-NOAA Do you see any issues with my aqm.rc or config.yaml files for my run? From my vantage point, it seems like the RAVE 2020 emissions do not even see the major fires in the west, as increasing the PM2.5 emissions by 1000 (after Sep 18th in my run) does not even make an impact on PM2.5 concentrations.

JianpingHuang-NOAA commented 2 years ago

@Patrick Campbell - NOAA Affiliate @.> Thanks for the update. Does "oper" represent NAM-CMAQ simulations? I am not sure what wildfire emissions were used by "Oper". BlueSky or GBBEPx? @Ho-Chun Huang @.> Do you remember?

In addition, I noticed that your NEXUS emissions have +37 values. Something must be wrong with your anthropogenic emission processing. I tried to rerun the case with the new workflow but it failed to download the gfs files from HPSS. It was ok with the first release version on WCOSS2 by Chan-Hoo. I just got an update of the new workflow from Chan-Hoo this morning. I am going to test it today and let you know if I find any issues. Meanwhile, please have a double check with your NEXIS emission files which are not correct to me.

Thanks,

Jianping

On Wed, Nov 9, 2022 at 9:24 AM Patrick Campbell @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA @KaiWang-NOAA https://github.com/KaiWang-NOAA You can see how much better our operational model with GBBEPx is doing compared to my online-cmaq run with RAVE for R9 and R10:

[image: image] https://user-images.githubusercontent.com/26631222/200855289-01476eb9-c5fd-4390-b403-98740193b00f.png

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1308842213, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PIYGK4C4LFUBZXV4NBLWHOXZHANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

Ho-ChunHuang-NOAA commented 2 years ago

is opr NAM-CMAQ or current operational CMAQ? in Sept 2020 NAM-CMAQ is the operational AQM. GFS-CMAQ is the operational AQM after July 2021.

If NAm-CMAQ then it is HMS-based Bluesky fire emission If GFS-CMAQ then it is daily GBBEPx fire emissions with prescribed diurnal profile for PM25 and FRP+ wild-fire only with spatial filter.

Ho-ChunHuang-NOAA commented 2 years ago

@JianpingHuang-NOAA @drnimbusrain Sorry I reply in github issue directly, hope you received notice.

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA @Ho-ChunHuang-NOAA The "opr" here is the current operational model, NACC-CMAQ. This as you know uses GBBEPx fire emissions with prescribed diurnal profile.

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA I also do not think this issue necessarily has to do with the anthropogenic emissions, as other regions outside of fires look OK for PM2.5, and the ozone looks reasonable in our runs.
Outside of the west the PM2.5 looks reasonable, such as the non-fire northeast R2 (albeit slightly underpredicted) Ozone CONUS looks reasonable (albeit overpredicted)

JianpingHuang-NOAA commented 2 years ago

I can show the problem via google meeting if you like.

On Wed, Nov 9, 2022 at 12:06 PM Patrick Campbell @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA I also do not think this issue necessarily has to do with the anthropogenic emissions, as other regions outside of fires look OK for PM2.5, and the ozone looks reasonable in our runs. [image: image] https://user-images.githubusercontent.com/26631222/200894120-b30500cc-6e83-49a2-830d-3934e62496a8.png Non-Fire PM2.5 Region [image: image] https://user-images.githubusercontent.com/26631222/200894278-90679ab6-8219-413f-8c12-554cca9ffd38.png

Ozone CONUS: [image: image] https://user-images.githubusercontent.com/26631222/200894357-824d215f-f24c-4f9e-a6eb-67b7d6defad9.png

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1309069004, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI2ZQYQYSIEQJH5RTTDWHPKZPANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

bbakernoaa commented 2 years ago

@JianpingHuang-NOAA please put the problem here so that everyone can see it. if you know what the issue is we all need to know.

JianpingHuang-NOAA commented 2 years ago

@Barry Baker - NOAA Federal @.***> Please see the message that I posed at 10:40 am this morning

".... In addition, I noticed that your NEXUS emissions have +37 values. Something must be wrong with your anthropogenic emission processing. I tried to rerun the case with the new workflow but it failed to download the gfs files from HPSS. It was ok with the first release version on WCOSS2 by Chan-Hoo. I just got an update of the new workflow from Chan-Hoo this morning. I am going to test it today and let you know if I find any issues. Meanwhile, please have a double check with your NEXIS emission files which are not correct to me."

On Wed, Nov 9, 2022 at 12:28 PM Barry Baker @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA please put the problem here so that everyone can see it. if you know what the issue is we all need to know.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1309096176, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI3DH7A3S23DGBA64MTWHPNKDANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA This is not clear as to what the issue may be with anthropogenic emissions in my run, and does not seem related to potential wildfire emissions issue. Can you please further clarify/show here this issue you found?

bbakernoaa commented 2 years ago

@drnimbusrain I completely agree. This does not seem to have anything related to the anthropogenic emissions. It is clearly something with the wildfire emissions

JianpingHuang-NOAA commented 2 years ago

I am checking your *aqm.t12z.NEXUS_Expt.nc http://aqm.t12z.NEXUS_Expt.nc

file at / lfs/h2/oar/ptmp/Patrick.C.Campbell/expt_dirs/aqm_cold_aqmna13_1day_arl_base_wetdepfix_firex1000_20200918_20200920/2020091912/INPUT

Please see the issue with the line highlighted with red color in the figure below.

[image: image.png]

On Wed, Nov 9, 2022 at 12:52 PM Barry Baker @.***> wrote:

@drnimbusrain https://github.com/drnimbusrain I completely agree. This does not seem to have anything related to the anthropogenic emissions. It is clearly something with the wildfire emissions

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1309124019, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI44J4LJRGVO5TZN25TWHPQENANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA There is no figure here to reference. Please further clarify.

bbakernoaa commented 2 years ago

@KaiWang-NOAA can you help out out and repeat this simulation? We really need an independent verification of this.

JianpingHuang-NOAA commented 2 years ago

@Barry Baker - NOAA Federal @.***> I guess that NEXUS has some issues with the new workflow. Please see O3 prediction on Slide 2.

https://docs.google.com/presentation/d/1z_o1WET8mU590CREuWMWdjT1SUbziNXNDPxyLgn25w4/edit#slide=id.g16f20c7e4e2_0_9

On Wed, Nov 9, 2022 at 3:38 PM Barry Baker @.***> wrote:

@KaiWang-NOAA https://github.com/KaiWang-NOAA can you help out out and repeat this simulation? We really need an independent verification of this.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1309348007, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PIZVJLYZ4NVF46OAWKDWHQDU5ANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

bbakernoaa commented 2 years ago

@JianpingHuang-NOAA Nothing has really changed in the nexus step. I don't understand what you are showing here or what you think should be happening. There is no details at all to what you mention may be a problem.

JianpingHuang-NOAA commented 2 years ago

Please check the emission file, aqm.t12z.NEXUS_Expt.nc generated by NEXUS at /scratch2/NCEPDEV/stmp3/Jianping.Huang/emc.para/com/aqm/v7.0/aqm.v7.0.c15.20190801/12 (Hera)

You will see a similar issue to what I saw from Patrick's run. The emissions show +37 values for CO, ISOP, etc.

On Thu, Nov 10, 2022 at 10:26 AM Barry Baker @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA Nothing has really changed in the nexus step. I don't understand what you are showing here or what you think should be happening. There is no details at all to what you mention may be a problem.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1310458893, or unsubscribe https://github.com/notifications/unsubscribe-auth/ANA2PI7K5UPPHR243RXMZX3WHUHYRANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

bbakernoaa commented 2 years ago

@JianpingHuang-NOAA please create a separate issue for this. It is not answering problems related to the issue described above

KaiWang-NOAA commented 2 years ago

Hi Barry and Patrick,

I think there are two separate issues that we're discussing here.

1) The first one is about RAVE emissions. Initially you and Patrick indicated the potential unit issue for the RAVE emis files from me. However as I pointed out in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1303886667, the regridding and unit conversions were done correctly from raw RAVE. So I don't see any problems with the RAVE files. See below for more information on the RAVE raw emissions that will be more relevant to underprediction of PM2.5 that we're concerned about.

2) The 2nd one is about the potential issue of NEXUS files generated in Patrick's simulations. As I showed in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1304062794, your simulations predicted much lower PM2.5 concentrations than Jianping's recent runs using the same set of RAVE emissions. As Jianping indicated, the anomalies (e.g., 10e+37) of PM2.5 emissions in NEXUS files could lead to the PM2.5 emissions not being read in correctly in the model. So we may need to fix it if it turned out to be an issue.

I would acknowledge even if we resolve issue 2), the PM2.5 could still be significantly underpredicted during Sept. 12-16, 2020 in the current UFS-CMAQ, which is consistent with the earlier test from Jianping when we run the model over the CONUS (see 150a in the attached slides). We have discussed this with Fangjun and Xiaoyang before (I included Shobha's team here since they may not be aware of our ongoing discussions on github). From our discussion, the fire events during the period of Sept. 12-16 detected by ABI/VIIRS are much lower than what we're expecting over R9 and R10 (see the attached slides). The large PM2.5 underpredictions are more likely due to that the model can't resolve the long-range transport of SMOKE that circulated back from the Pacific ocean to R9 and R10 caused by the synoptic circulation. CMAQ_RAVE_PM25_17May2022.pptx https://drive.google.com/file/d/1VoaWr8ZZNMMrpqwcyOnXEAoaux_uR1qH/view?usp=drive_web

Thanks,

Kai

On Thu, Nov 10, 2022 at 10:42 AM Barry Baker @.***> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA please create a separate issue for this. It is not answering problems related to the issue described above

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1310484997, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2A5ALVE4ZVOODDAVG3ELZDWHUJWFANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

drnimbusrain commented 2 years ago

@KaiWang-NOAA

I agree there are two separate issues, and disagreed with @JianpingHuang-NOAA adding the NEXUS anthropogenic issue here. Hopefully, he will open up a separate issue on that.

For the main issue (1) about RAVE emissions test in our Sep 2020 retro case run. I understand that you have done QA/QC on the units/regridding. However, if they are correct, and the online-cmaq workflow reads them in "correctly", I disagree with the following consensus with your discussion with Fangjun and Xiaoyang: "The large PM2.5 underpredictions are more likely due to that the model can't resolve the long-range transport of SMOKE that circulated back from the Pacific ocean to R9 and R10 caused by the synoptic circulation."

I think we would all agree that the online-cmaq PM2.5 predictions in the west during the strong fires are not even realistic in my runs, and such underpredictions cannot be attributed to long-range transport issues. For example, as shown below, the operational (NACC-CMAQ) model with GBBEPx appears to do much better than RAVE (which appears to not even see the fires):

If either you or @JianpingHuang-NOAA can do an independent test of the same RAVE Sep 2020 emissions for this case, and you still see such an unrealistic PM2.5 prediction in the west, I think we really need to involve others (including Fangjun and Xiaoyang) again for another discussion.

KaiWang-NOAA commented 2 years ago

Thanks Kai. Model runs with RAVE seem to be doing very well in other regions, except in Region 9 and 10. Now, how do you guys verify transport in the model?

Shobha

On 11/10/2022 6:03 PM, Kai Wang - NOAA Affiliate wrote:

Hi Barry and Patrick,

I think there are two separate issues that we're discussing here.

1) The first one is about RAVE emissions. Initially you and Patrick indicated the potential unit issue for the RAVE emis files from me. However as I pointed out in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1303886667 https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1303886667, the regridding and unit conversions were done correctly from raw RAVE. So I don't see any problems with the RAVE files. See below for more information on the RAVE raw emissions that will be more relevant to underprediction of PM2.5 that we're concerned about.

2) The 2nd one is about the potential issue of NEXUS files generated in Patrick's simulations. As I showed in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1304062794 https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1304062794, your simulations predicted much lower PM2.5 concentrations than Jianping's recent runs using the same set of RAVE emissions. As Jianping indicated, the anomalies (e.g., 10e+37) of PM2.5 emissions in NEXUS files could lead to the PM2.5 emissions not being read in correctly in the model. So we may need to fix it if it turned out to be an issue.

I would acknowledge even if we resolve issue 2), the PM2.5 could still be significantly underpredicted during Sept. 12-16, 2020 in the current UFS-CMAQ, which is consistent with the earlier test from Jianping when we run the model over the CONUS (see 150a in the attached slides). We have discussed this with Fangjun and Xiaoyang before (I included Shobha's team here since they may not be aware of our ongoing discussions on github). From our discussion, the fire events during the period of Sept. 12-16 detected by ABI/VIIRS are much lower than what we're expecting over R9 and R10 (see the attached slides). The large PM2.5 underpredictions are more likely due to that the model can't resolve the long-range transport of SMOKE that circulated back from the Pacific ocean to R9 and R10 caused by the synoptic circulation. CMAQ_RAVE_PM25_17May2022.pptx https://drive.google.com/file/d/1VoaWr8ZZNMMrpqwcyOnXEAoaux_uR1qH/view?usp=drive_web

Thanks,

Kai

On Thu, Nov 10, 2022 at 10:42 AM Barry Baker @. @.>> wrote:
@JianpingHuang-NOAA <https://github.com/JianpingHuang-NOAA> please
create a separate issue for this. It is not answering problems
related to the issue described above

—
Reply to this email directly, view it on GitHub
<https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1310484997>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/A2A5ALVE4ZVOODDAVG3ELZDWHUJWFANCNFSM6AAAAAARDN54RI>.
You are receiving this because you were mentioned.Message ID:
***@***.***>
--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@. @.>

(301) 683-3230

-- Shobha Kondragunta, Ph.D Lead, Aerosols and Atmospheric Composition Science Team GeoXO Atmospheric Composition Instrument Product Lead NOAA NESDIS Center for Satellite Applications and Research cell: 301-655-7311 twitter: @AerosolWatch https://www.star.nesdis.noaa.gov/star/Kondragunta_S.php

The contents of this message are mine personally and do not necessarily reflect any position of NOAA.

drnimbusrain commented 2 years ago

R9 and R10 are also the near source fire regions, and by virtue of ozone and PM2.5 doing well everywhere but the fire regions, seems like overall large scale transport is OK. Seems like something else in the fire regions in R9 and R10 is wrong. Has anyone else tested a major fire period with the RAVE 2020 emissions?

On Fri, Nov 11, 2022, 11:30 AM KaiWang-NOAA @.***> wrote:

Thanks Kai. Model runs with RAVE seem to be doing very well in other regions, except in Region 9 and 10. Now, how do you guys verify transport in the model?

Shobha

On 11/10/2022 6:03 PM, Kai Wang - NOAA Affiliate wrote:

Hi Barry and Patrick,

I think there are two separate issues that we're discussing here.

1) The first one is about RAVE emissions. Initially you and Patrick indicated the potential unit issue for the RAVE emis files from me. However as I pointed out in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1303886667 https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1303886667, the regridding and unit conversions were done correctly from raw RAVE. So I don't see any problems with the RAVE files. See below for more information on the RAVE raw emissions that will be more relevant to underprediction of PM2.5 that we're concerned about.

2) The 2nd one is about the potential issue of NEXUS files generated in Patrick's simulations. As I showed in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1304062794 https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1304062794, your simulations predicted much lower PM2.5 concentrations than Jianping's recent runs using the same set of RAVE emissions. As Jianping indicated, the anomalies (e.g., 10e+37) of PM2.5 emissions in NEXUS files could lead to the PM2.5 emissions not being read in correctly in the model. So we may need to fix it if it turned out to be an issue.

I would acknowledge even if we resolve issue 2), the PM2.5 could still be significantly underpredicted during Sept. 12-16, 2020 in the current UFS-CMAQ, which is consistent with the earlier test from Jianping when we run the model over the CONUS (see 150a in the attached slides). We have discussed this with Fangjun and Xiaoyang before (I included Shobha's team here since they may not be aware of our ongoing discussions on github). From our discussion, the fire events during the period of Sept. 12-16 detected by ABI/VIIRS are much lower than what we're expecting over R9 and R10 (see the attached slides). The large PM2.5 underpredictions are more likely due to that the model can't resolve the long-range transport of SMOKE that circulated back from the Pacific ocean to R9 and R10 caused by the synoptic circulation. CMAQ_RAVE_PM25_17May2022.pptx < https://drive.google.com/file/d/1VoaWr8ZZNMMrpqwcyOnXEAoaux_uR1qH/view?usp=drive_web

Thanks,

Kai

On Thu, Nov 10, 2022 at 10:42 AM Barry Baker @. @.>> wrote:

@JianpingHuang-NOAA https://github.com/JianpingHuang-NOAA please create a separate issue for this. It is not answering problems related to the issue described above

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1310484997, or unsubscribe < https://github.com/notifications/unsubscribe-auth/A2A5ALVE4ZVOODDAVG3ELZDWHUJWFANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@. @.>

(301) 683-3230

-- Shobha Kondragunta, Ph.D Lead, Aerosols and Atmospheric Composition Science Team GeoXO Atmospheric Composition Instrument Product Lead NOAA NESDIS Center for Satellite Applications and Research cell: 301-655-7311 twitter: @AerosolWatch https://www.star.nesdis.noaa.gov/star/Kondragunta_S.php

The contents of this message are mine personally and do not necessarily reflect any position of NOAA.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1311921943, or unsubscribe https://github.com/notifications/unsubscribe-auth/AGLFYNUMC2D2WC7IYCATFV3WHZYDXANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

KaiWang-NOAA commented 2 years ago

Hi Shobha,

The long-range transport is still a speculation for the moment. We will need to get the wind field evaluation from Ho-chun to better understand it.

Hi Patrick and Barry,

Jianping did a new c15 run for the 2020 episode. Please see some preliminary MONET analysis.

https://docs.google.com/presentation/d/1pjyWJIw9HqJNmo5FMARF3bWSQvpR7zkSxJU_aa1JqCU/edit#slide=id.g18c566fc633_0_17

The PM2.5 performance is much better than your testing despite the large underprediction of PM2.5 during Sept. 13-16. One thing I noticed is the NACC results shown in my slides are different from yours. The NACC I'm using is from /scratch2/BMC/rcm1/rhs/fv3/regional/data/nacc_cmaq/2020/09/aqm.20200915/aqm.t12z.aconc_sfc_24.ncf on Hera and I'm suspecting it's not the latest NACC run. I'm curious if you have a copy of your NACC results on Hera?

Thanks,

Kai

On Fri, Nov 11, 2022 at 11:30 AM Shobha Kondragunta < @.***> wrote:

Thanks Kai. Model runs with RAVE seem to be doing very well in other regions, except in Region 9 and 10. Now, how do you guys verify transport in the model?

Shobha

On 11/10/2022 6:03 PM, Kai Wang - NOAA Affiliate wrote:
Hi Barry and Patrick,

I think there are two separate issues that we're discussing here.

1) The first one is about RAVE emissions. Initially you and Patrick indicated the potential unit issue for the RAVE emis files from me. However as I pointed out in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1303886667 https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1303886667, the regridding and unit conversions were done correctly from raw RAVE. So I don't see any problems with the RAVE files. See below for more information on the RAVE raw emissions that will be more relevant to underprediction of PM2.5 that we're concerned about.

2) The 2nd one is about the potential issue of NEXUS files generated in Patrick's simulations. As I showed in https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1304062794 https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1304062794, your simulations predicted much lower PM2.5 concentrations than Jianping's recent runs using the same set of RAVE emissions. As Jianping indicated, the anomalies (e.g., 10e+37) of PM2.5 emissions in NEXUS files could lead to the PM2.5 emissions not being read in correctly in the model. So we may need to fix it if it turned out to be an issue.

I would acknowledge even if we resolve issue 2), the PM2.5 could still be significantly underpredicted during Sept. 12-16, 2020 in the current UFS-CMAQ, which is consistent with the earlier test from Jianping when we run the model over the CONUS (see 150a in the attached slides). We have discussed this with Fangjun and Xiaoyang before (I included Shobha's team here since they may not be aware of our ongoing discussions on github). From our discussion, the fire events during the period of Sept. 12-16 detected by ABI/VIIRS are much lower than what we're expecting over R9 and R10 (see the attached slides). The large PM2.5 underpredictions are more likely due to that the model can't resolve the long-range transport of SMOKE that circulated back from the Pacific ocean to R9 and R10 caused by the synoptic circulation. CMAQ_RAVE_PM25_17May2022.pptx < https://drive.google.com/file/d/1VoaWr8ZZNMMrpqwcyOnXEAoaux_uR1qH/view?usp=drive_web

Thanks,

Kai

On Thu, Nov 10, 2022 at 10:42 AM Barry Baker @. @.>> wrote:
@JianpingHuang-NOAA <https://github.com/JianpingHuang-NOAA> please
create a separate issue for this. It is not answering problems
related to the issue described above

—
Reply to this email directly, view it on GitHub
<https://github.com/NOAA-EMC/AQM/issues/20#issuecomment-1310484997>,
or unsubscribe
<
https://github.com/notifications/unsubscribe-auth/A2A5ALVE4ZVOODDAVG3ELZDWHUJWFANCNFSM6AAAAAARDN54RI . You are receiving this because you were mentioned.Message ID: @.***>

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@. @.>

(301) 683-3230
-- Shobha Kondragunta, Ph.D Lead, Aerosols and Atmospheric Composition Science Team GeoXO Atmospheric Composition Instrument Product Lead NOAA NESDIS Center for Satellite Applications and Research cell: 301-655-7311 twitter: @AerosolWatch https://www.star.nesdis.noaa.gov/star/Kondragunta_S.php

The contents of this message are mine personally and do not necessarily reflect any position of NOAA.

--

Kai Wang, Ph.D.

Lynker at NOAA/NWS/NCEP/EMC

5830 University Research Ct., Rm. 2029

College Park, MD 20740

@.***

(301) 683-3230

drnimbusrain commented 2 years ago

@JianpingHuang-NOAA @KaiWang-NOAA Thank you for performing your independent Sep 2020 case test. I agree that your run looks much better for PM2.5 in the western fire regions compared to our online-cmaq run, with the understanding that the major PM2.5 from fires in the west are still underpredicted (but not as bad as our run).

I assume Jianping's run is also a cold start, where the PM2.5 does quickly spin-up and reach background pretty quickly (~10 days). Our run does not spin-up in 10-days, and the PM2.5 stays flat for some reason. Can I see the aqm.rc and config.yaml files for your run? I am suspicious about how the runs are setup, which may suggest a separate problem in retro runs.

For the NACC-CMAQ comparison in your analysis, I would use the Para6d runs for this case (identical to the operational) found here on Hera: /scratch1/NCEPDEV/stmp4/Patrick.C.Campbell/emc-para6d

Thanks again.

NOAA-EMC / AQM