search

NOAA-EMC / rrfs-workflow

workflow for RRFSv1
Other
12 stars 27 forks source link

missing or all zero values for some variables in RRFS grib2 product #212

Open JiliDong-NOAA opened 7 months ago

JiliDong-NOAA commented 7 months ago

When working on the fix for missing reflectivity of rrfs members using inline post, I noticed there are several variables in the real-time rrfs_a products with all missing or zero values across the whole domain. I quickly scanned one of the real-time rrfs_a grib2 file and have these variables listed below (sample file: rrfs_a.20240112/00/control/rrfs.t00z.prslev.f012.conus_3km.grib2). Some of the variables have been discussed (i.e. REFZR and REFZI). Some of them will be fixed if not already (NCPCP). Some of them may be zero due to physics reasons (Hail and Snow melt?). But I still see quite a few variables which should have valid values but not: such as 10m potential temp/specific humidity, Blackadars Mixing Length Scale, Planetary Boundary Layer Regime, Upward Long-Wave Rad. Flux at TOA, albedo, u/v momentum fluxes etc.

I am curious if these variables are needed/required by stakeholders/forecasters/downstream users? If yes, these missing/zero values should be fixed. If not, should we consider removing these variables from grib2 product?

39:37724182:vt=2024011212:entire atmosphere (considered as a single layer):12 hour fcst:REFZR Equivalent radar reflectivity factor for rain [mm^6/m^3]:

40:37724399:vt=2024011212:entire atmosphere (considered as a single layer):12 hour fcst:REFZI Equivalent radar reflectivity factor for snow [mm^6/m^3]:

44:41753610:vt=2024011212:1 hybrid level:12 hour fcst:BMIXL Blackadars Mixing Length Scale [m]:

50:49689661:vt=2024011212:surface:12 hour fcst:PBLREG Planetary Boundary Layer Regime [-]:

755:656826058:vt=2024011212:surface:11-12 hour max fcst:HAIL Hail [m]:

813:727851200:vt=2024011212:10 m above ground:12 hour fcst:POT Potential Temperature [K]:

814:727851417:vt=2024011212:10 m above ground:12 hour fcst:SPFH Specific Humidity [kg/kg]:

821:730760576:vt=2024011212:surface:11-12 hour acc fcst:NCPCP Large-Scale Precipitation (non-convective) [kg/m^2]:

823:730761058:vt=2024011212:surface:11-12 hour acc fcst:LSPA Land Surface Precipitation Accumulation [kg/m^2]:

827:731587783:vt=2024011212:surface:11-12 hour acc fcst:SNOM Snow Melt [kg/m^2]:

829:731889480:vt=2024011212:surface:11-12 hour acc fcst:BGRUN Baseflow-Groundwater Runoff [kg/m^2]:

830:731889721:vt=2024011212:surface:11-12 hour acc fcst:WATR Water Runoff [kg/m^2]:

834:732426693:vt=2024011212:surface:11-12 hour acc fcst:FFLDRO Flash Flood Runoff (Encoded as an accumulation over a floating subinterval of time) [kg/m^2]:

836:732427179:vt=2024011212:surface:11-12 hour acc fcst:GWLOWS Group Water Lower Storage [kg/m^2]:

853:738704521:vt=2024011212:surface:11-12 hour acc fcst:EVP Evaporation [kg/m^2]:

858:742964340:vt=2024011212:surface:12 hour fcst:UFLX Momentum Flux, U-Component [N/m^2]:

859:742964557:vt=2024011212:surface:12 hour fcst:VFLX Momentum Flux, V-Component [N/m^2]:

884:763010498:vt=2024011212:entire atmosphere (considered as a single layer):12 hour fcst:SWHR Solar Radiative Heating Rate [K/s]:

885:763010715:vt=2024011212:entire atmosphere (considered as a single layer):12 hour fcst:LWHR Long-Wave Radiative Heating Rate [K/s]:

895:769389984:vt=2024011212:entire atmosphere (considered as a single layer):11-12 hour ave fcst:CDLYR Non-Convective Cloud Cover [%]:

913:793665411:vt=2024011212:top of atmosphere:12 hour fcst:ULWRF Upward Long-Wave Rad. Flux [W/m^2]:

918:797322084:vt=2024011212:surface:12 hour fcst:AEMFLX Atmosphere Emission Mass Flux [kg/m^2/s]:aerosol=Particulate Organic Matter Dry::

920:798311899:vt=2024011212:surface:12 hour fcst:CSDSF Clear Sky Downward Solar Flux [W/m^2]:

1075:1009953316:vt=2024011212:surface:12 hour fcst:ALBDO Albedo [%]:

1076:1009953533:vt=2024011212:surface:12 hour fcst:SNFALB Snow-Free Albedo [%]:
HuiyaChuang-NOAA commented 7 months ago

@JiliDong-NOAA thank you for reporting this issue.

@EricJames-NOAA @jaymes-kenyon @AndrewBenjamin-NOAA Jili reported a list of variables that has missing or zero values. Most of these variables were generated by our regional models and UPP just passed them through. Thus, I think we will just need to figure out whether model can output them and if so, turn on the switch in diag table.

Wen already left to visit family in China and won't be back till Feb. 21. I'm leaving soon for Taiwan and won't return till Feb. 23. Any chance you can take a look until then? Thank you!

HuiyaChuang-NOAA commented 7 months ago

@MatthewPyle-NOAA I've booked a meeting with Wen on 2/26 to follow up on this issue

MatthewPyle-NOAA commented 7 months ago

@HuiyaChuang-NOAA Hopefully we'll get this sorted out well before then, but thanks. Enjoy your time away!

HuiyaChuang-NOAA commented 7 months ago

@HuiyaChuang-NOAA Hopefully we'll get this sorted out well before then, but thanks. Enjoy your time away!

Thank you! Will be my first time to make Chinese New Year family reunion in 2 decades.

EricJames-NOAA commented 7 months ago

@HuiyaChuang-NOAA enjoy your vacation!

@AndrewBenjamin-NOAA should be able to tell us how many of these are actually needed for RRFS. We may be able to get rid of many of them, although I'm not as familiar with the NAM related ones.

AndrewBenjamin-NOAA commented 7 months ago

@EricJames-NOAA: yes I can.

Products that can be removed/retired:

  1. REFZR: Equivalent radar reflectivity factor from rain
  2. REFZI: Equivalent radar reflectivity factor for snow
  3. PBLREG Planetary Boundary Layer Regime
  4. NCPCP Large-Scale Precipitation (non-convective)
  5. LSPA Land Surface Precipitation Accumulation
  6. WATR Water Runoff
  7. FFLDRO Flash Flood Runoff (Encoded as an accumulation over a floating subinterval of time) (hourly accumulation)
  8. GWLOWS Group Water Lower Storage
  9. CDLYR Non-Convective Cloud Cover
  10. AEMFLX Atmosphere Emission Mass Flux
  11. BMIXL Blackadars Mixing Length Scale
  12. POT Potential Temperature: 10 m above ground
  13. SPFH Specific Humidity: 10m above ground
  14. SWHR Solar Radiative Heating Rate
  15. UFLX Momentum Flux, U-Component
  16. VFLX Momentum Flux, V-Component
  17. LWHR Long-Wave Radiative Heating Rate

Products that should be in RRFS:

  1. HAIL Hail (hourly max)
  2. EVP Evaporation *firewx nest
  3. SNOM Snow Melt (hourly accumulations) *firewx nest
  4. BGRUN Baseflow-Groundwater Runoff (hourly accumulations)
  5. CSDSF Clear Sky Downward Solar Flux
  6. ALBDO Albedo
  7. SNFALB Snow-Free Albedo
  8. ULWRF Upward Long-Wave Rad. Flux (Top of Atmos. instantaneous: average forecast on grib2 files)
EricJames-NOAA commented 7 months ago

@AndrewBenjamin-NOAA thanks! The only notes I would add are that the variables currently named FFLDRO and GWLOWS are fields that I have added which we don't want to remove. They are binary QPF exceedances of various precipitation thresholds. It makes sense they would all be zero for some times in the winter, but they will highlight grid points with heavy rainfall during the warm season. I think we were working on getting some new GRIB2 names for these fields, but I'm not sure where that stands.

Also, the AEMFLX is the smoke emission for the smoke component of RRFS, so it should not be zero. I'm going to look into what's going on there.

JacobCarley-NOAA commented 7 months ago

@EricJames-NOAA Smoke is currently off in RRFS_A owing to a bug. That likely explains why AEMFLX is currently zero.

EricJames-NOAA commented 7 months ago

@EricJames-NOAA Smoke is currently off in RRFS_A owing to a bug. That likely explains why AEMFLX is currently zero.

@JacobCarley-NOAA thanks! That makes sense. Let's retain that output field.

jaymes-kenyon commented 7 months ago

Regarding the "HAIL" field, I am finding non-zero values in our beta retros. So, the missing values in some RRFS members may be limited to members with non-standard physics. Even so, the "HAIL" values that I am seeing are quite small (fractions of a mm), even during periods of confirmed large hail. I am investigating.

AndrewBenjamin-NOAA commented 7 months ago

Thanks for the follow ups everyone. I am based my list on what was currently in operations. So if products were already agreed upon to be added, then it obviously makes sense to keep them. AEMFLX would be an example of this.

@AndrewBenjamin-NOAA thanks! The only notes I would add are that the variables currently named FFLDRO and GWLOWS are fields that I have added which we don't want to remove. They are binary QPF exceedances of various precipitation thresholds. It makes sense they would all be zero for some times in the winter, but they will highlight grid points with heavy rainfall during the warm season. I think we were working on getting some new GRIB2 names for these fields, but I'm not sure where that stands.

Also, the AEMFLX is the smoke emission for the smoke component of RRFS, so it should not be zero. I'm going to look into what's going on there.

@EricJames-NOAA are these the same values that we added to the NCEP grib2 tables last year? Take a look at https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-2-1-1.shtml

196 and 197 are new exceedance grid we coordinated on. If these are the same products, please use the new IDS in Table 4.2-1-1.

AndrewBenjamin-NOAA commented 7 months ago

What is the status of the other products that need to be fixed? Some of these I can remember seeing for some time in the parallels.

  1. HAIL Hail (hourly max)
  2. EVP Evaporation *firewx nest
  3. SNOM Snow Melt (hourly accumulations) *firewx nest
  4. BGRUN Baseflow-Groundwater Runoff (hourly accumulations)
  5. CSDSF Clear Sky Downward Solar Flux
  6. ALBDO Albedo
  7. SNFALB Snow-Free Albedo
  8. ULWRF Upward Long-Wave Rad. Flux (Top of Atmos. instantaneous: average forecast on grib2 files)
EricJames-NOAA commented 7 months ago

@AndrewBenjamin-NOAA I think Tanya and should be able to get BGRUN working. I was just looking at CSDSF in UPP...I see this message in the code where the field is set to missing instead of read in:

! inst incoming clear sky sfc shortwave ! FV3 do not output instant incoming clear sky sfc shortwave !$omp parallel do private(i,j) do j=jsta_2l,jend_2u do i=ista_2l,iend_2u rswinc(i,j) = spval enddo enddo

Does anyone know if FV3 can output clear sky downward solar irradiance?

AndrewBenjamin-NOAA commented 7 months ago

@EricJames-NOAA Thats good news on BGRUN, please keep me posted.

I also noticed that WEASD is only being output hourly, with no run-total accumulations written to the grib2 files i.e a 0-7 hour acc fcst at f007). This was previously on the parallels, but is now missing. Was this taken offline for some reason?

EricJames-NOAA commented 7 months ago

@AndrewBenjamin-NOAA I think we moved to TSNOWP for the run total accumulated snow output. WEASD is now only used for the snow water equivalent on the ground. Hopefully @ericaligo-NOAA agrees with this explanation.

ericaligo-NOAA commented 7 months ago

@AndrewBenjamin-NOAA I think we moved to TSNOWP for the run total accumulated snow output. WEASD is now only used for the snow water equivalent on the ground. Hopefully @ericaligo-NOAA agrees with this explanation.

That's correct. Note TSNOWP does not use the variable precip ice density. It's just the water equivalent of snow at the surface directly from the microphysics. There's a run-time accumulated field, and one that's emptied according to the defined buckets.

EricJames-NOAA commented 7 months ago

@AndrewBenjamin-NOAA is CSDSF an instantaneous flux? I don't see that in RAP/HRRR....is it something we are carrying over from NAM? Thanks.

AndrewBenjamin-NOAA commented 7 months ago

@EricJames-NOAA, Yes CSDSF is an instantaneous flux. And you are correct, CSDSF is an operational product from the NAMnest

EricJames-NOAA commented 6 months ago

@AndrewBenjamin-NOAA with the latest UPP code we can get instantaneous ULWRF at top of atmosphere.

Tanya reports that BGRUN will be all zero with the RUC LSM since it is not coupled to hydrology. So it should be OK to remove this field.

I believe @jaymes-kenyon was looking into HAIL.

I'm working on the other ones.

jaymes-kenyon commented 6 months ago

@AndrewBenjamin-NOAA — Regarding HAIL, the UPP code seems to be working fine. There are some runs with universal zero values, but most most runs contain nonzero instances. However, unrelated to the UPP code, there is an underlying FV3 bug (in the HAILCAST routine) that is dividing hail sizes by 1000, yielding gridded values that are just fractions of a millimeter. A PR to fix this has been introduced: https://github.com/NOAA-GFDL/GFDL_atmos_cubed_sphere/pull/320

@EricJames-NOAA — Since HAIL is in good shape (in UPP), let me know if I can help tackle any of the remaining trouble spots. Thanks!

EricJames-NOAA commented 6 months ago

@jaymes-kenyon thanks! I will touch base with Tanya next week and let you know if we need help with anything.