search

dtcenter / MET

Model Evaluation Tools
https://dtcenter.org/community-code/model-evaluation-tools-met
Apache License 2.0
78 stars 24 forks source link

Update GRIB tables in MET based on wgrib2 versions 3.1.4 and 3.4.0 #2780

Open JohnHalleyGotway opened 10 months ago

JohnHalleyGotway commented 10 months ago

Describe the Enhancement

Note that this work was performed using the GRIB2 table input from:

This issue was discussed during the METplus NOAA User Telecon on Jan 8, 2024 (see meeting notes - limited access). Gwen indicated that wgrib2 version 3.1.4 is under development and will include updates to some of the GRIB2 tables. Note that wgrib2 has migrated to GitHub at NOAA-EMC/wgrib2.

NCEP GRIB2 tables have been updated with new WMO-approved variables (27 freezing/melting potential, 28 melt onset date, and 29 freeze onset date in Table 4.2-10-2 and Level 172 in Table 4.5).

A new release v3.1.4 will be installed on Acorn soon for testing. If successful, it will be delivered to WCOSS2. Use the v3.4.0 release from 8/15/24 instead.

This issue is to:

Recommend requesting that Gwen (Li-Chuan Chen) serve as the tester for these table updates.

Time Estimate

1 day.

Sub-Issues

Consider breaking the enhancement down into sub-issues. None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

Labels

Milestone and Projects

Define Related Issue(s)

Consider the impact to the other METplus components.

Enhancement Checklist

See the METplus Workflow for details.

JohnHalleyGotway commented 9 months ago

Note that discussion dtcenter/METplus#2465 points out that the FSNOWC variable from GRIB2 table 4.2-0-1 is also missing. Double-check to confirm that updates to this table are included in MET version 12.0.0.

On 10/31/2024, confirmed that FSNOWC is included in this update:

grib2_all.txt:0 1 0 255 0 0 1 121 "FSNOWC" "Fraction of Snow Cover" "Proportion"
JohnHalleyGotway commented 5 months ago

At the METplus NOAA User Telecon on 5/13/24, @MarcelCaron-NOAA requested that grib2_mrms.txt be updated to include ~10 new lines for FRANA for this sample of the 24-hour flat ice product: MRMS_EXP_FRANA_Flat_1440min_00.00_20240123-000000.grib2.gz

Test to confirm that the ice product data is read from these files well.

JohnHalleyGotway commented 1 week ago

For this issue, recommend updating additional static data sources:

JohnHalleyGotway commented 5 days ago

Details about updating the GRIB2 tables:

Ran the following commands to reformat the existing data files to match the current format in MET:

for file in `find wgrib2 -name "*gribtable.dat"`; do
  echo $file
  echo GRIB2 > NEW/`basename $file`
  cat $file | egrep -i '^ *{' | sed 's/^ *{ *//g' | \
  sed 's/ *, */ /g' | sed 's/ *} *//g' >> \
  NEW/`basename $file`
done

When updating the existing tables, make sure:

  1. There is no embedded whitespace in variable names
  2. There is no embedded whitespace in unit strings (found some in MRMS)
  3. There are 3 strings after each set of integers... variable name, long name, units (long name and units out of order in MRMS)

Had to carefully merge existing grib2_mrms.txt with updated MRMS_gribtable.dat. First confirmed that the updated MRMS_gribtable.dat file does NOT introduce any new table entries by checking the first 8 columns. It just updates variable names, descriptions, and units.

echo new GRIB2 > grib2_mrms_new.txt
while read LINE; do
  hdr=`echo $LINE | cut -d' ' -f1-8`
  if [[ `grep "${hdr} " MRMS_gribtable.dat | wc -l` -eq 1 ]]; then
    grep "${hdr} " MRMS_gribtable.dat >> grib2_mrms_new.txt
  else
    grep "${hdr} " grib2_mrms.txt >> grib2_mrms_new.txt
  fi
done < grib2_mrms.txt

Made no updates to grib2_ndfd.txt since it was only changing empty units strings to ?? which doesn't seem like an improvement.

Wrote up NOAA-EMC/wgrib2#285 to alert them of 2 minor GRIB2 table file issues I encountered.

GwenChen-NOAA commented 5 days ago

Wesley's source code of wgrib2 v3.1.4 is on Hera:/home/Wesley.Ebisuzaki/grib2/wgrib2. The GRIB2 tables in this version have been updated with the NCEP GRIB2 documentation https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-1.shtml. The /home/Wesley.Ebisuzaki/grib2/wgrib2/gribtables/ncep/gribtable.dat file contains ALBDOICE and SICEVOL. The /home/Wesley.Ebisuzaki/grib2/wgrib2/Level.c file has the new mixed layer depth (Level 172 in Table 4.5 https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.shtml ).

THE NCEPLIB team may have taken an older version of wgrib2 code and developed from there. I saw you already opened a wgrib2 issue (#285 https://github.com/NOAA-EMC/wgrib2/issues/285) on suspicious GRIB2 table entries. Please relay this information to them.

Thanks, Gwen

On Wed, Oct 30, 2024 at 5:09 PM John Halley Gotway @.***> wrote:

@GwenChen-NOAA https://github.com/GwenChen-NOAA, I did update the GRIB2 tables for MET based on the contents of this wgrib2 gribtables https://github.com/NOAA-EMC/wgrib2/tree/develop/wgrib2/gribtables directory. However, I spot-checked against this GRIB2 table documentation:

https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-2-10-2.shtml

I was looking for the new WMO-approved variables (27 freezing/melting potential, 28 melt onset date, and 29 freeze onset date in Table 4.2-10-2 https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-2-10-2.shtml and Level 172 in Table 4.5 https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.shtml)... that are mentioned in the description of this issue.

On line 1122 https://github.com/NOAA-EMC/wgrib2/blob/cfc151dc94e39eeb44d700ed0ced06316c1d4ce6/wgrib2/gribtables/ncep/gribtable.dat#L1122 of ncep/gribtable.dat, I see an entry for SNOWTSI... but no entries following that for ALBDOICE and SICEVOL for example.

So it would appear that the GRIB2 tables in the NOAA-EMC/wgrib2 repo are NOT up to date with what's published in the GRIB2 documentation. Do you know of an authoritative source of these tables? Or do I really just need to scrape content from the website documentation?

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2780#issuecomment-2448386118, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWXF3DEOIKA7CDHY4GL7LYDZ6FDG5AVCNFSM6AAAAABQT2MC4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGM4DMMJRHA . You are receiving this because you were mentioned.Message ID: @.***>

GwenChen-NOAA commented 5 days ago

I do see ALBDOICE on Line 32 https://github.com/NOAA-EMC/wgrib2/blob/cfc151dc94e39eeb44d700ed0ced06316c1d4ce6/wgrib2/gribtables/ncep/gribtable.dat#L32 and SICEVOL on Line https://github.com/NOAA-EMC/wgrib2/blob/cfc151dc94e39eeb44d700ed0ced06316c1d4ce6/wgrib2/gribtables/ncep/gribtable.dat#L1073 1073 https://github.com/NOAA-EMC/wgrib2/blob/cfc151dc94e39eeb44d700ed0ced06316c1d4ce6/wgrib2/gribtables/ncep/gribtable.dat#L1073 of ncep/gribtable.dat https://github.com/NOAA-EMC/wgrib2/blob/cfc151dc94e39eeb44d700ed0ced06316c1d4ce6/wgrib2/gribtables/ncep/gribtable.dat and the new mixed layer depth (Level 172 in Table 4.5 https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.shtml) on Line 198 of Level.c https://github.com/NOAA-EMC/wgrib2/blob/cfc151dc94e39eeb44d700ed0ced06316c1d4ce6/wgrib2/Level.c#L198. You may want to double check.

On Wed, Oct 30, 2024 at 5:51 PM L. Gwen Chen @.***> wrote:

Wesley's source code of wgrib2 v3.1.4 is on Hera:/home/Wesley.Ebisuzaki/grib2/wgrib2. The GRIB2 tables in this version have been updated with the NCEP GRIB2 documentation https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-1.shtml. The /home/Wesley.Ebisuzaki/grib2/wgrib2/gribtables/ncep/gribtable.dat file contains ALBDOICE and SICEVOL. The /home/Wesley.Ebisuzaki/grib2/wgrib2/Level.c file has the new mixed layer depth (Level 172 in Table 4.5 https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.shtml ).

THE NCEPLIB team may have taken an older version of wgrib2 code and developed from there. I saw you already opened a wgrib2 issue (#285 https://github.com/NOAA-EMC/wgrib2/issues/285) on suspicious GRIB2 table entries. Please relay this information to them.

Thanks, Gwen

On Wed, Oct 30, 2024 at 5:09 PM John Halley Gotway < @.***> wrote:

@GwenChen-NOAA https://github.com/GwenChen-NOAA, I did update the GRIB2 tables for MET based on the contents of this wgrib2 gribtables https://github.com/NOAA-EMC/wgrib2/tree/develop/wgrib2/gribtables directory. However, I spot-checked against this GRIB2 table documentation:

https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-2-10-2.shtml

I was looking for the new WMO-approved variables (27 freezing/melting potential, 28 melt onset date, and 29 freeze onset date in Table 4.2-10-2 https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-2-10-2.shtml and Level 172 in Table 4.5 https://www.nco.ncep.noaa.gov/pmb/docs/grib2/grib2_doc/grib2_table4-5.shtml)... that are mentioned in the description of this issue.

On line 1122 https://github.com/NOAA-EMC/wgrib2/blob/cfc151dc94e39eeb44d700ed0ced06316c1d4ce6/wgrib2/gribtables/ncep/gribtable.dat#L1122 of ncep/gribtable.dat, I see an entry for SNOWTSI... but no entries following that for ALBDOICE and SICEVOL for example.

So it would appear that the GRIB2 tables in the NOAA-EMC/wgrib2 repo are NOT up to date with what's published in the GRIB2 documentation. Do you know of an authoritative source of these tables? Or do I really just need to scrape content from the website documentation?

— Reply to this email directly, view it on GitHub https://github.com/dtcenter/MET/issues/2780#issuecomment-2448386118, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWXF3DEOIKA7CDHY4GL7LYDZ6FDG5AVCNFSM6AAAAABQT2MC4CVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDINBYGM4DMMJRHA . You are receiving this because you were mentioned.Message ID: @.***>

JohnHalleyGotway commented 5 days ago

@GwenChen-NOAA, aha, thanks for pointing out my mistake. I had assumed that they'd be listed in order in the gribtable file, but they are not. I'll keep that in mind and be more careful in my checking.

Pulled GRIB tables from hera:/home/Wesley.Ebisuzaki/grib2/wgrib2/gribtables, processed them, and note the following:

  1. No differences found for: BOM_gribtable.dat, KMA_gribtable.dat, NDFD_gribtable.dat, dwd_gribtable.dat, tigge_gribtable.dat, ECMWF_gribtable.dat
  2. Many updates for ncep/gribtable.dat increasing the number of entries to 1669.
  3. Has fewer entries for MRMS_gribtable.dat
JohnHalleyGotway commented 4 days ago

At the METplus NOAA User Telecon on 5/13/24, @MarcelCaron-NOAA requested that grib2_mrms.txt be updated to include ~10 new lines for FRANA for this sample of the 24-hour flat ice product: MRMS_EXP_FRANA_Flat_1440min_00.00_20240123-000000.grib2.gz

Test to confirm that the ice product data is read from these files well.

@MarcelCaron-NOAA, I can't find a definition for this data. Here's the output from wgrib2:

> wgrib2 MRMS_EXP_FRANA_Flat_1440min_00.00_20240123-000000.grib2 
1:0:d=2024012300:var discipline=209 center=161 local_table=1 parmcat=5 parm=8:500 m above mean sea level:anl:

But no corresponding entry exists in the MRMS GRIB tables:

Can you please advise where the definitions for the FRANA Flat Ice Products can be found?

MarcelCaron-NOAA commented 4 days ago

@JohnHalleyGotway My understanding is this is a brand new product and I'm not aware of any existing MRMS GRIB table definitions for FRANA ... apart from what I've custom-defined and tested (WCOSS2): /lfs/h2/emc/vpppg/noscrub/marcel.caron/test/frana/grib2_mrms.txt

 209 10 0 255 161 1 5 4  "FRANA_Flat_60min"  "Freezing Rain Accumulation National Analysis - Flat Ice 1-hour"  "mm"
 209 10 0 255 161 1 5 5  "FRANA_Flat_180min"  "Freezing Rain Accumulation National Analysis - Flat Ice 3-hour"  "mm"
 209 10 0 255 161 1 5 6  "FRANA_Flat_360min"  "Freezing Rain Accumulation National Analysis - Flat Ice 6-hour"  "mm"
 209 10 0 255 161 1 5 7  "FRANA_Flat_720min"  "Freezing Rain Accumulation National Analysis - Flat Ice 12-hour"  "mm"
 209 10 0 255 161 1 5 8  "FRANA_Flat_1440min"  "Freezing Rain Accumulation National Analysis - Flat Ice 24-hour"  "mm"
 209 10 0 255 161 1 5 11  "FRANA_Radial_60min"  "Freezing Rain Accumulation National Analysis - Radial Ice 1-hour"  "mm"
 209 10 0 255 161 1 5 12  "FRANA_Radial_180min"  "Freezing Rain Accumulation National Analysis - Radial Ice 3-hour"  "mm"
 209 10 0 255 161 1 5 13  "FRANA_Radial_360min"  "Freezing Rain Accumulation National Analysis - Radial Ice 6-hour"  "mm"
 209 10 0 255 161 1 5 14  "FRANA_Radial_720min"  "Freezing Rain Accumulation National Analysis - Radial Ice 12-hour"  "mm"
 209 10 0 255 161 1 5 15  "FRANA_Radial_1440min"  "Freezing Rain Accumulation National Analysis - Radial Ice 24-hour"  "mm"

Does that help, or would you need definitions from an operational MRMS GRIB table?

JohnHalleyGotway commented 4 days ago

Details about updating the GRIB1 tables:

egrep "^const struct ParmTable" wgrib.c | cut -d'[' -f1 > names.txt
while read NAME; do
  out_file="`echo $NAME | awk '{print $4}' | sed 's/parm_table/grib1/g'`.txt"
  echo "Writing: ${out_file}"
  center=`echo $out_file | cut -d'_' -f2`
  subcenter=`echo $out_file | cut -d'_' -f3 | cut -d'.' -f1`
  if [[ "$center" == "ecmwf" ]]; then
    center_id=98
  elif [[ "$center" == "dwd" ]]; then
    center_id=78
  elif [[ "$center" == "cptec" ]]; then
    center_id=46
  elif [[ "$center" == "ncep" || "$center" == "nceptab" ]]; then
    center_id=7
  elif [[ "$center" == "jra55" ]]; then
    center_id=34
  else
    center_id=$center
  fi
  echo GRIB1 > $out_file
  egrep "$NAME" wgrib.c -A 256 | egrep -v "ParmTable" | tr -d '*' > lines.txt
  while read LINE; do
    n=`echo $LINE | awk '{print $2}'`
    var=`echo $LINE | cut -d'"' -f2`
    desc=`echo $LINE | cut -d'"' -f4 | cut -d'[' -f1 | sed -r 's/ +$//g'`
    units=`echo $LINE | cut -d'"' -f4 | cut -d'[' -f2 | cut -d']' -f1`
    if [[ "${desc}" == "${units}" ]]; then
      units=""
    fi
    echo "${n} ${subcenter} ${center_id} -1 \"${var}\" \"${desc}\" \"${units}\"" >> ${out_file}
  done < lines.txt
done < names.txt

Compare these newly generated files with the existing tables and make any changes that are warranted:

Eliminate all embedded whitespace from units string. Also replace all instanced of ** with ^ to consistently indicate exponentiation in a more human-readable way. For example, replace m**-1s with m/s:

cat grib1_*.txt | cut -d'"' -f6 | egrep ' |\*\*' | sort -u > units.txt
rm units_sed.txt
while read LINE; do
  SEARCH=`echo ${LINE} | sed 's%\*%\\\\*%g'`
  REPLACE=`echo ${LINE} | sed 's%\*\*%^%g'`
  echo sed \'s%\"${SEARCH}\"%\"${REPLACE}\"%g\' \| \\ >> units_sed.txt
done < units.txt

Manually edit units_sed.txt to define the conversions and save it as: fix_units.sh.gz

JohnHalleyGotway commented 4 days ago

Thanks @MarcelCaron-NOAA! Since those definitions don't conflict with any existing usage, adding them to the grib2_mrms.txt file in my feature branch sounds fine to me, which I did with this commit. I don't know the process for getting them officially accepted and the MRMS GRIB2 table updated, but it may eventually involve submitting a new wgrib2 issue.