Enhance MET to use point observations falling between the first and last columns of a global grid.

[JohnHalleyGotway]

Describe the Enhancement

During the METplus MetOffice telecon on 6/10/21, @robdarvell described an issue he's run into when doing point verification on a global grid. He found 8 points located near the prime meridian that were counted by Point-Stat as being 'off the grid' and therefore excluded from the statistics. The MET library interpolation code translates those points from lat/lon space to grid x/y space and the grid-x value for these points are -1 < grid-x < 0.

This issue is to enhance MET to incorporate those points along the seam into the verification. Specifically, when processing global grids, handle points with x locations between -1 and 0 or between Nx-1 and Nx. Recommend addressing the following issues:

• Request test data from @robdarvell to replicate this behavior.
  • Please find sample data from Rob: kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta2/feature_1823/input_data
• Check to see what handling of global grids currently exists (e.g. shifting global grids in MODE by some number of degrees). Is there any automated or configurable logic for identifying global grids? If not, consider adding both... so that they are automatically identified, but that identification can be overridden by a config file option.
• When stating "global" grids, we are talking about the x-dimension, indicating that the longitudes wrap. The latitudes need not extend all the way to the poles for this logic to be required.
• When processing these points on the seam, enhance the interpolation logic to interpolate between the first and last columns of grid values.

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the enhancement down into sub-issues. No sub-issues required.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

2799991

Define the Metadata

Assignee

• [x] Select engineer(s) or no engineer required: John HG
• [x] Select scientist(s) or no scientist required: Rob D for now. Could have another METplus scientist advise on the changes and ask Rob to test.

Labels

• [x] Select component(s)
• [x] Select priority
• [x] Select requestor(s)

Projects and Milestone

• [x] Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• [x] Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• [x] METplus, MET, METdatadb, METviewer, METexpress, METcalcpy, METplotpy If configurable options are added, coordinate them with METplus wrappers.

Enhancement Checklist

See the METplus Workflow for details.

• [x] Complete the issue definition above, including the Time Estimate and Funding Source.
• [x] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
• [x] Complete the development and test your changes.
• [x] Add/update log messages for easier debugging.
• [x] Add/update unit tests.
• [x] Add/update documentation.
• [x] Push local changes to GitHub.
• [x] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
• [x] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Linked issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
• [x] Iterate until the reviewer(s) accept and merge your changes.
• [x] Delete your fork or branch.
• [x] Close this issue.

[robdarvell]

I have placed a UM forecast file, plus corresponding obs netcdf file and my config file (more or less uses the default master config) within the ftp anonymous account (darvell_data directory). I note the config is still METplus3, but works with 4. Was also run under MET10.

[JohnHalleyGotway]

Thanks Rob. I retrieved the sample data you sent and stored it locally on a project machine (kiowa:/d1/projects/MET/MET_pull_requests/met-10.1.0/met-10.1.0_beta2/feature_1823/input_data).

[mpm-meto]

This problem also affects regrid_data_plane. Here is an example of taking UM model output and regridding it to the GPM grid. The image shows the missing data column along the meridian, i.e. regrid_data_plane is not interpolating across 0 meridian. Note the rain along the front is split in the plot on the right. The original forecast is 0 to 360 whilst the output from regrid_data_plane is -180 to 180.

[mpm-meto]

I think the fact that the GPM data is -180 to 180 exposes this problem, as the model data is 0 to 360.

[JohnHalleyGotway]

@robdarvell and @mpm-meto, I've made some progress on this issue this week. My feature branch (feature_1823_global) now uses point observations in point_stat that fall in the "seam" of a global grid and the regridding no longer introduces a strip of missing data. As seen below. Left: MET version 10.0.0, Right: feature_1823_global branch:

While the changes are working as expected. I'm a bit leery about the number of changes I've made and want to do lots more testing to make sure I haven't broken anything else.

I am wondering if I should try to tackle the "global MODE" issue along with this, enabling objects to straddle the edge of a global grid? This is essentially the same issue right?

[JohnHalleyGotway]

Discussed at MetOffice meeting on 9/16/21.

• What name should we use internally instead of "isGlobal" since it doesn't need to include all the latitudes? Perhaps "wrapLon" would be better.
• Yes, please write up an issue to handle MODE wrapping the globe.
• Consider adding a minor issue to remove the existing "shift" option since it will no longer be necessary.

[robdarvell]

John, Was thinking of having a play with these changes in our gridstat and pointstat jobs we are running to examine what has been done at our end. What is the best plan of attack to do this sort of thing (given I have not done anything like this from Git before)? I did create a single-branch of your branch and cloned the stuff across to my local machine, but then I ran into compile issues as it didn't like the configure option of the script I run. (my thought was a similar approach to how I install the main releases that I have been doing (command I was trying was "/configure --prefix="${PREFIX}" --enable-grib2 --enable-python")). Probably doing something wrong in the very basic way, so best way to approach welcome. Or is this something I should wait until you have placed it onto the Develop branch? Rob

[JohnHalleyGotway]

@robdarvell the steps for compiling directly from the repo differ from compiling from a tarfile. @TaraJensen has noted this as a frustration for some users and and as a result, we defined this development task to restructure the repo accordingly: dtcenter/MET#1920.

However we do NOT want to restructure the repo for a minor release (MET version 10.1.0) and will instead do it for the next major release (MET version 11.0.0). This of course doesn't have anything to do with testing global grids though.

For dtcenter/MET#1823, I'd recommend that you wait until next week when we'll have this feature included in the met-10.1.0_beta3 release. We'll post a tarfile for that, and you should be able to compile that, assuming you were able to compile previous releases.

After submitting a pull request for the initial version of the gen_ens_prod tool yesterday, I am circling back around to dtcenter/MET#1823 today. Does that timeline work for you?

[robdarvell]

@JohnHalleyGotway thanks for this. I am more than happy to wait to until the Beta3 release gets done (I was just playing with something with some of our areas, and thought it would be a good time to test out what you had done). I have installed the Beta2 release here, so if anyone wanted to use it (eg: the Ocean area) then it is there, so adding Beta3 when that appears should work

[JohnHalleyGotway]

@robdarvell and @j-opatz, please give this logic for determining whether or not to wrap the longitudes some thought.

Is it sufficient?

For example, we could have a degenerate grid that includes a lot more than 360 degrees of longitude, and perhaps in that case wrapLon should be false? Of course there would likely be other problems too!

Or much more likely, we could have a grid that is not equally spaced over the "wrap". For example, a grid with longitudes = 0, 100, 200, 300. With dlon = 100, this would satisfy the wrapLon = TRUE logic. However the spacing between the first/second longitudes (100 degrees) is a LOT different from the spacing between the last/first longitudes (60 degrees). In this case, simply computing x % Nx wouldn't work well.

For example, when choosing (x,y) points in computing bilinear interpolation we apply that modulo logic. This implicitly assumes that spacing in the grid units is consistent with lat/lon spacing. But for dlon = 100, that's not true!

With most global grids (e.g. 1.0, 0.5, or 0.25 degrees), the longitude spacing does remain consistent over the wrap point. But I'm wondering if I should add logic to check that? And if not, set wrapLon = FALSE? Or is that overkill?

[robdarvell]

@JohnHalleyGotway and @j-opatz

If I understood the question at the end correctly, I started thinking whether or not I knew of any Global models which used variable resolutions, around the meridian area. Have to say that I couldn’t think of any. I know we have, what we call, regional models, which have variable resolutions (eg: our 1.5km UK model, which is at 4km on the outside and 1.5km over the UK) but in that we just verify the 1.5km bit after it is ‘cut’ from the full model. I know some would like to verify the full model, but that isn’t possible in VER. Marion then said that the ARPEGE model is a Global model which has variable resolution, down to around 5km over the French/Europe area down from around 25km (approximately). However, we don’t verify that. So over the ‘wrap’ point it is likely to be the same either side, just might not be the same value from North Pole to South Pole. So, at this stage, I am wondering whether or not it is a very good question to raise, and perhaps document (store an issue) for later, but at this time not worth trying to work out a fix to what may happen.