Enhance vx_data_ugrid library to read in tripolar projected datasets

j-opatz commented 7 months ago

Describe the Enhancement

Due to its non-standard gridding methodology (i.e. inconsistent distance between latitude and longitude lines), tripolar projection is unable to be utilized by any of MET's tools. This has historically meant that tripolar projection datasets cannot be used in any way in MET. However if the tripolar projection is approached as a system of points, we can then use a k-d tree distance method for analyzing the tripolar "grid" points and point observations. This methodology would open up the ability to verify tripolar projected forecasts with point observations in tools like Point-Stat.

To do this, we will need to enhance the vx_data_ugrid library to support this new projection.

The work in this issue is the point-verification side of #2658, which will deal with the grid-verification (including interpolation).

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues.

[ ] Add a checkbox for each sub-issue here.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

None yet.

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[ ] Review default alert labels
[ ] Select component(s)
[ ] Select priority
[ ] Select requestor(s)

Milestone and Projects

[ ] Select Milestone as the next official version or Backlog of Development Ideas
[ ] For the next official version, select the MET-X.Y.Z Development project

Define Related Issue(s)

Consider the impact to the other METplus components.

[ ] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

Enhancement Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Development issue Select: Milestone as the next official version Select: MET-X.Y.Z Development project for development toward the next official release
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

AliciaBentley-NOAA commented 6 months ago

@JohnHalleyGotway @j-opatz Can this tripolar grid MET issue be marked as required for the official release?

j-opatz commented 1 month ago

@hsoh-u I located the three tripolar file types we need to test against. All of the locations are on Seneca:

/d1/personal/jopatz/workbench/tripolar/CICE_fcst
/d1/personal/jopatz/workbench/tripolar/GFS_ocean
/d1/personal/jopatz/workbench/tripolar/RTOFS_fcst

There are various files in each directory, but for testing purposes I'm interested in commands for the following 2:

/d1/personal/jopatz/workbench/tripolar/GFS_ocean/gfs.ocean.t00z.6hr_avg.f006.nc
/d1/personal/jopatz/workbench/tripolar/RTOFS_fcst/rtofs_glo_2ds_f006_ice.nc

The reasoning for testing two files is the GFS ocean file is not CF-compliant and will require a configuration file to tell MET which variable field(s) to use for coordinate systems. The RTOFS file is CF-compliant and will cover the behavior of CICE. I want to make sure both methods (configuration file and command line) work properly.

Please provide the two necessary commands to run these two files. I'll test them out with basic settings and see if I can create intelligible output for EMC to review.

hsoh-u commented 1 month ago

The enhancement is allowing two dimension names instead of the cell dimension name. For GFS tripolar data:

ugrid_metadata_map = [
   { key = "dim_face";  val = "xh,yh"; },
   ...
];

Testing GFS case:

  out_dir=out_grid_stat_ugrid
  [ ! -d $out_dir ] && mkdir -p $out_dir

  fcst_name="/d1/personal/jopatz/workbench/tripolar/GFS_ocean/gfs.ocean.t00z.6hr_avg.f006.nc"

  TMP_OUTDIR="`pwd`/$out_dir"
  export METPLUS_CAT_THRESH=""
  export METPLUS_CENSOR_THRESH=""
  export METPLUS_CENSOR_VAL=""
  export METPLUS_CLIMO_CDF_DICT=""
  export METPLUS_CLIMO_MEAN_DICT=""
  export METPLUS_CLIMO_STDEV_DICT=""
  export METPLUS_DESC=""
  export METPLUS_DISTANCE_MAP_DICT=""
  export METPLUS_FCST_FIELD="field = [{ name=\"SST\"; level=\"(0,*,*)\"; }];"
  export METPLUS_FCST_FILE_TYPE="file_type = NETCDF_UGRID;"
  export METPLUS_FOURIER_DICT=""
  export METPLUS_GRID_WEIGHT_FLAG=""
  export METPLUS_HSS_EC_VALUE=""
  export METPLUS_INTERP_DICT=""
  export METPLUS_MASK_DICT=""
  export METPLUS_MET_CONFIG_OVERRIDES=""
  export METPLUS_MODEL=""
  export METPLUS_NBRHD_COV_THRESH=""
  export METPLUS_NBRHD_SHAPE=""
  export METPLUS_NBRHD_WIDTH=""
  export METPLUS_NC_PAIRS_FLAG_DICT=""
  export METPLUS_NC_PAIRS_VAR_NAME=""
  export METPLUS_OBS_FIELD="field = [{ name=\"SST\"; level=\"(0,*,*)\"; }];"
  export METPLUS_OBS_FILE_TYPE="file_type = NETCDF_UGRID;"
  export METPLUS_OBTYPE=""
  export METPLUS_OUTPUT_FLAG_DICT="output_flag = {ctc = BOTH;}"
  export METPLUS_OUTPUT_PREFIX=""
  export METPLUS_REGRID_DICT="regrid = {to_grid = \"G231\";}"
  export METPLUS_SEEPS_P1_THRESH=""
  export METPLUS_UGRID_COORDINATES_FILE=""
  export METPLUS_UGRID_DATASET=""
  export METPLUS_UGRID_MAX_DISTANCE_KM=""
  export MET_TMP_DIR="$TMP_OUTDIR"
  export OMP_NUM_THREADS="1";
  #export METPLUS_UGRID_COORDINATES_FILE="ugrid_coordinates_file = \"$fcst_name\";"
  #export METPLUS_UGRID_DATASET="ugrid_dataset = \"GFS\";"

/d1/personal/hsoh/git/pull_request/MET_feature_2859_tripolar_to_ugrid/bin/grid_stat \
   /d1/personal/jopatz/workbench/tripolar/GFS_ocean/gfs.ocean.t00z.6hr_avg.f006.nc  \
   /d1/personal/jopatz/workbench/tripolar/GFS_ocean/gfs.ocean.t00z.6hr_avg.f006.nc  \
   /d1/personal/hsoh/data/UGrids/GridStatConfig_wrapped \
   -ugrid_config /d1/personal/hsoh/data/UGrids/UGridConfig_GFS -outdir out_grid_stat_ugrid

The observation file should be changed.

Three UGrid configurations (dimension and variable mappings) are created at seneca:/d1/personal/hsoh/data/UGrids directory: /d1/personal/hsoh/data/UGrids/UGridConfig_CICE /d1/personal/hsoh/data/UGrids/UGridConfig_GFS /d1/personal/hsoh/data/UGrids/UGridConfig_RTOFS

JohnHalleyGotway commented 1 month ago

As discussed on 10/17/24, @hsoh-u please add the following unit tests to unit_ugrid.xml:

Run grid_stat to compare RTOFS_fcst UGRID forecast data to GFS_ocean UGRID ocean data as the observation.
- Regrid each to a global lat/lon grid (regrid.to_grid = "G004"; is 0.5 deg global lat/lon grid)
- Only need to write NetCDF matched pairs output file, not .stat output needed.
Run grid_stat to read CICE_fcst UGRID data and compare to some gridded polar stereographic observation data with regrid.to_grid = "OBS";. @j-opatz will add a comment pointing @hsoh-u to that PS obs data.

Note that the "regridding" from UGRID locations to "to_grid" locations only supports nearest neighbor at this time.

hsoh-u commented 1 month ago

The first one does not work because only one metadata mapping is passed to grid_stat via -ugrid_config option. They have to be the same dataset (RTOFS to RTOFS or GFS to GFS).

j-opatz commented 1 month ago

@hsoh-u I was able to find gridded observational data in polar stereographic projection for the GFS-ocean comparison. On seneca, /d1/personal/jopatz/workbench/tripolar/ice_observation/seaice_conc_daily_nh_1978_v04r00.nc Utilize the cdr_seaice_conc variable field. I've already tested this file in Plot-Data-Plane and confirmed it produces output.

j-opatz commented 1 month ago

Performing a Grid-Stat test revealed that there are some concerns to be addressed in the logic of the tripolar reprojection process.

For my test, I ran an RTOFS forecast file to verify the ice concentration field against the observation SMMR dataset ice concentration field from NSIDC. When using ncdump on both files it is clear there are values >0 at various points, indicating the presence of ice. While the RTOFS file is not viewable via METplus, the SMMR file shows approximately what both files should look like. This image was taken from a Plot-Data-Plane call for the ice coverage field: SMMR_obs

The projection of the data onto the globe highlights the first error: MET will usually attempt to place data on a lat/lon projection which will distort data with a polar stereographic projection, which the SMMR file has. However, it should still appear in the northern hemisphere of a lat/lon projection, while the attached image shows the data floating somewhere over Latin America.

Moving forward, the Grid-Stat call finished successfully and created output. However, on reviewing the raw forecast field the second problem is revealed: RTOFS_fcst

Once again, the data is projected somewhere over Latin America. This time, however, the RTOFS data file shows all 0 values for the entire domain. I verified that the input RTOFS dataset does have values for the ice concentration field, so somewhere in the reprojection process, MET seems to be dropping the values attached to the various lat/lon pairs.

These two issues (incorrect projection of data with Plot-Data-Plane and incorrect reprojection of tripolar data values) need to be addressed.

hsoh-u commented 4 days ago

It looks like the data image is flipped vertically and horizontally (with IDV plot below). I think there are problems from the input data itself.(seneca:/d1/personal/jopatz/workbench/tripolar/ice_observation/seaice_conc_daily_nh_1978_v04r00.nc).

This is a plot by ncview:

And by IDV. WIth different color table:

dtcenter / MET