Data retrieval for observations

Description

Observations for data assimilation and verification should be incorporated into the retrieve_data.py tool, and hooked into the workflow tasks.

Currently, some observations do have retrieval tasks in the workflow, but they only work with HPSS on RDHPCS platforms. These observations include MRMS, CCPA, and NDAS observations, and are the types used for verification.

The list needs to be extended to include bufr and prepbufr files from RAP, lightning, AMDAR rejection lists, Mesonet use lists, VIIRS Green Vegetation Fraction, snow observations, and SST analyses.

Solution

The capabilities to retrieve these types of files should be included in the retrieve_data.py tool. Currently, the tool is only meant to interact with the YAML file that includes analysis and forecast gridded data from other modeling systems.

Each of these observation data types should be included in the YAML file, along with their known sources on HPSS, AWS, etc. Any known supported data streams on supported platforms should be included in the machine files under the data section.

Acceptance Criteria (Definition of Done)

Any of the data sets listed above can be retrieved with the retrieve_data.py tool, whether from a remote source or data stream. The associated tasks have been incorporated into the workflow to gather the data sets, as needed by verification and RRFS data assimilation.

ufs-community / ufs-srweather-app