Enhance the point pre-processing tools to support data censoring and conversion

Describe the Enhancement

During the NOAA METplus User Telecon on 12/12/2022, the idea of supporting data censoring in the MET point pre-processing tools arose.

It would be useful when processing AirNow datasets in ascii2nc where observed concentration values can be reported as being slightly negative due to the observation uncertainty logic applied. The negative values should be reassigned a value of 0. While this data censoring can already be performed in the Point-Stat and Ensemble-Stat tools downstream, providing the option to handle this during the pre-processing step would be convenient. Similarly, if adding support for censor_thresh and censor_val, we should also support convert(x) to handle unit conversions.

This task is to add new configuration options for this functionality to the point pre-processing tools. Recommend modeling it after the existing time_summary dictionary which is supported across all these tools. Pick a good name and define an array of dictionaries, each defining data censoring and/or conversion logic for one or more obs_var names.

data_processing = [
  {
    obs_var       = [];
    censor_thresh = [];
    censor_val    = [];
    convert(x)    = x;
    units = ""; // Define a units string that should be written to the output NetCDF file, if applicable.
  }
];

An empty array means that no data_processing steps should be performed. Otherwise, apply them in the order they are listed in the config file.

Time Estimate

2 days.

Sub-Issues

Consider breaking the enhancement down into sub-issues. None needed.

Relevant Deadlines

List relevant project deadlines here or state NONE.

Funding Source

Define the source of funding and account keys here or state NONE.

Define the Metadata

Assignee

[ ] Select engineer(s) or no engineer required
[ ] Select scientist(s) or no scientist required

Labels

[x] Select component(s)
[x] Select priority
[x] Select requestor(s)

Projects and Milestone

[x] Select Repository and/or Organization level Project(s) or add alert: NEED PROJECT ASSIGNMENT label
[x] Select Milestone as the next official version or Future Versions

Define Related Issue(s)

Consider the impact to the other METplus components.

[x] METplus, MET, METdataio, METviewer, METexpress, METcalcpy, METplotpy
[ ] Define a METplus issue to make these options configurable.

Enhancement Checklist

See the METplus Workflow for details.

[ ] Complete the issue definition above, including the Time Estimate and Funding Source.
[ ] Fork this repository or create a branch of develop. Branch name: feature_<Issue Number>_<Description>
[ ] Complete the development and test your changes.
[ ] Add/update log messages for easier debugging.
[ ] Add/update unit tests.
[ ] Add/update documentation.
[ ] Push local changes to GitHub.
[ ] Submit a pull request to merge into develop. Pull request: feature <Issue Number> <Description>
[ ] Define the pull request metadata, as permissions allow. Select: Reviewer(s) and Development issues Select: Repository level development cycle Project for the next official release Select: Milestone as the next official version
[ ] Iterate until the reviewer(s) accept and merge your changes.
[ ] Delete your fork or branch.
[ ] Close this issue.

dtcenter / MET