Allow observation anomaly replacement in Anomaly Correlation Coefficient (ACC) calculation

[j-opatz]

Describe the New Feature

In discussions with Gwen Chen from EMC, there seems to be an opportunity for expansion with MET's calculation of Anomaly Correlation Coefficient (ACC).

As it currently stands, ACC is calculated with the following equation: Where f denotes the forecast, o for observation, and c for climatology. The (o - c) term is more simply the observation anomaly; likewise, the (f - c) term is the forecast anomaly.

During Sea Surface Height (SSH) verification within MET, Gwen noticed that the MET values for ACC were not matching the legacy system outputs (mismatch at approx. the tenths value). The legacy system is part of Todd Spindler's website, given here: https://polar.ncep.noaa.gov/global/class-1/ (view the SSH tab). This was due to the MET calculation method and the datasets available: using the AVISO as the verification dataset, the Absolute Dynamic Topography (ADT) and Sea Level Anomaly (SLA) are the only variables available. ADT is instantaneous and cannot be used to verify SSH, and SLA is, as its name implies, an anomaly. In MET, the equation provided above does not allow any substitutions for the anomalies it calculates, instead using forecast-observation pairs and climatology.

Ideally, there would be a way for users to provide anomaly datasets (forecast or observation) and have those be substituted in the appropriate areas for statistics. Given the large coding overhaul that would require, it might be more prudent to create a new CNT column that calculates this alternative method of ACC.

The biggest hurdle to this is implementation. Even by restricting this to the tool currently being used for analysis (Grid-Stat), it would require another file input, regridding, and possibly more configuration options available to the forecast and observation fields.

Acceptance Testing

Gwen provided datasets for testing the functionality. Forecasts come from RTOFS, Observations come from AVISO, and climatology is provided from HYCOM_NCEP. These are housed on Seneca, under /d1/personal/jopatz/workbench/ACC_work

Time Estimate

Estimate the amount of work required here. Issues should represent approximately 1 to 3 days of work.

Sub-Issues

Consider breaking the new feature down into sub-issues.

• [ ] Add a checkbox for each sub-issue here.

Relevant Deadlines

This would ideally be completed before the EVS deadline of Dec. 1st, but with the current workload on MET and necessary lead time to get new features tested and loaded over to NOAA machines, that may not be an option.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

• [ ] Select engineer(s) or no engineer required
• [ ] Select scientist(s) or no scientist required

Labels

• [ ] Select component(s)
• [ ] Select priority
• [ ] Select requestor(s)

Projects and Milestone

• [ ] Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
• [ ] Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

• [ ] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy

New Feature Checklist

See the METplus Workflow for details.

• [ ] Complete the issue definition above, including the Time Estimate and Funding source.
• [ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
• [ ] Complete the development and test your changes.
• [ ] Add/update log messages for easier debugging.
• [ ] Add/update unit tests.
• [ ] Add/update documentation.
• [ ] Push local changes to GitHub.
• [ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
• [ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Linked issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
• [ ] Iterate until the reviewer(s) accept and merge your changes.
• [ ] Delete your fork or branch.
• [ ] Close this issue.

[j-opatz]

This issue was discussed in the 10/20/22 MET engineer meeting. The consensus is that given the backlog of work that needs to be completed before the MET v11.0.0 release, in addition to funding available and deadline for the EVS, this work will not meet the December 1st deadline nor the v11.0.0 release, and will need to be added to work for the next upcoming release. Milestone was adjusted accordingly.

John HG questioned if this work belongs in MET coding, or if it is best solved by Python embedding. If this is a common issue (i.e. verification datasets are often already in anomaly form), then it should be solved with MET coding. Perhaps a flag that allows the observation data to be treated as an anomaly in calculations? This could cause other difficulties, and open more doors (what about a forecast anomaly flag, how do you preserve the other 99+ statistics that are not using an anomaly, etc) than wanted. If this is not a common issue, then Python embedding can be made to calculate this in the manner desired. However, it would need to be a METCalcPy/User Script application, which would require more tools to be run for the other statistics.

The answer to that question will be determined by discussions with EMC, which will be started by email and pursued in more detail ~mid-December.

[JohnHalleyGotway]

On 4/3/23, Gwen notes that we'll likely need support for configuring with Forecast Climatology AND Observation Climatology, rather than just a single climatology field. We could consider making the specification of a forecast climatology mean field as being optional. If provided, use it. If not, use the observation climatology instead?

[GwenChen-NOAA]

Broader features to be considered for future development:

1. Capability to read in separate climatology mean fields for forecast and observation (i.e., clim_fcst and clim_obs). Forecast climatology (or model climatology) is a function of init/valid date and forecast lead time at a given location/grid point (x, y).

2. Capability to calculate forecast and observation anomalies using separate climatology mean fields, that is, anom_fcst = fcst - clim_fcst and anom_obs = obs - clim_obs and use them for Anomaly Correlation Coefficient (ACC) calculation.

3. Capability to read in observation anomalies (anom_obs) from a dataset, such as Sea Level Anomaly (SLA) from the AVISO dataset, and use it for ACC calculation.

4. Capability to read in forecast anomalies (anom_fcst) from a dataset and use it for ACC calculation.

5. Options to ask MET to calculate forecast/observation anomaly using climatology mean field, or to directly use forecast/observation anomaly from an input dataset.

[JohnHalleyGotway]

Discussed at NOAA METplus User telecon on 11/13/23. Recommend putting this issue on hold until EMC provides more direction.

[AliciaBentley-NOAA]

@JohnHalleyGotway @j-opatz Following up with EMC developers to see if this is critical for the MET-12.0.0 release or if we can work around it.

[AliciaBentley-NOAA]

@JohnHalleyGotway @j-opatz @TaraJensen Samira Ardani (the RTOFS developer who replaced Gwen) feels that this update is required for RTOFS verification in EVS v2.0 and needs to be included in MET-12.0.0. I would still put this issue lower than adding wind direction verification, reading in tripolar grids, and reading in unstructured grids. Thank you!