Learn more about how the API returns data

[mgwalker]

We've observed some unexpected behavior when using the API and we should understand better how it's deriving data in order to help decide whether the API should be our primary data source.

A specific example. We know that observation station K1HW exists:

• https://api.weather.gov/stations/K1HW

From the data returned, we can see that the station is located at -102.83ºE by 41.32ºN. The station is named "Hotel (Gurley)". If we open a map to that location, we'll see that it is indeed near a town called Gurley, Nebraska.

If we now query the API for information about the point where the weather station is located:

• https://api.weather.gov/points/41.32,-102.83

We see that this point is located in WFO CYS (Cheyenne, WY), in grid cell x=178, y=17. We will also get a URL that points to a list of observation stations for this point:

• https://api.weather.gov/gridpoints/CYS/178,17/stations

Immediately notice that station K1HW is not in the list, despite being literally at this location.

Jittering the location a little bit produces essentially the same result, where K1HW is not in the list of observation stations, but a station over 100 miles away is listed.

• How are observation stations related to WFO grid points?
  • Why might an observation station be present in the API but not show up in any WFO grid points?
  • When fetching a list of observation stations for a WFO grid point, what order are they returned in?
• Is it possible to short-circuit out the WFO grids and get a list of observation stations directly from a point instead?

[colinmurphy01]

Adding to backlog as I assume this will be something we should answer either this sprint or next.

[colinmurphy01]

Schedule convo with Kolly

[mgwalker]

• Every observation station in MADIS is represented in the /stations endpoint
• Observation stations are filtered out of the grids endpoints if:
  • it has not reported within a certain amount of time (probably 3 hours)
  • the WFO that controls the grid point has decided to exclude it
• Stations are listed in the grids endpoint in order of increasing distance from...?
  • this could be one of the corners of the grid or the center of the grid
  • grids being 2.5km x 2.5km, this difference shouldn't be more than ~1 mile
• Some geographic areas have more sparse observation coverage (particularly in rural parts of the west)
• Some geographic areas have dramatic altitude changes over short distances, making it more important to get the nearest station.
• A station's data is only included in the API when it has been QC'ed by MADIS
  • QC feed from MADIS is piecemeal: as soon as a single sensor at the station is QC'ed, the entire observation gets sent out. Repeat as each sensor at the station is QC'ed
  • Instead of using that feed, the API pulls a CSV, which partially contributes to latency, but most of the latency is on the MADIS side
• MADIS is run by NWS, not (like Greg thought) a university partner

[mgwalker]

From experimentation, we don't want to rely on MADIS's public-facing data feeds. They're slow, a little clunky, and don't provider fresher data than api.weather.gov. The private-facing feeds via LDM and NetCDF may be a different story, though.

That said, MADIS has a contract with Synoptic to do some of the QC and distribution work. The western region is currently using a Synoptic API to show current observations (see the map at https://www.weather.gov/wrh/; in the overlays panel on the right, click the "Observations" tab and turn it on – that is fetching data from the Synoptic API). Maybe we can see about hooking into that API?

[mgwalker]

Gonna close this. There's info in this thread that's useful and we'll probably have ongoing learnings about the API, but there's no reason to just hold this open forever. We accomplished the main goal of the issue, which was figuring out which observation stations the API covers and how it determines which observation stations to include in a list for a given point.