Add wind (and dew point?) to missing observation log in NewRelic

[eric-gade]

Description

With #534 we implemented a strategy for current conditions, whereby we check the first observation station in the sorted list provided by the API, and if that seems to be "down," we go to the next one, and repeat, so on, for 3 times.

The logic of the "down" part needs to be fleshed out more, because for now it only checks for the presence of a temperature reading. This presents a problem, because the current conditions block is also looking for fields like textDescription and icon to know what to display in multiple places within its template.

Below is an example of a "bad result" observation station response that passes our current validation, but presents problems when being displayed on the page:

(See textDescription and icon values in the observation)

{
  "@context": [
    "https://geojson.org/geojson-ld/geojson-context.jsonld",
    {
      "@version": "1.1",
      "wx": "https://api.weather.gov/ontology#",
      "s": "https://schema.org/",
      "geo": "http://www.opengis.net/ont/geosparql#",
      "unit": "http://codes.wmo.int/common/unit/",
      "@vocab": "https://api.weather.gov/ontology#",
      "geometry": {
        "@id": "s:GeoCoordinates",
        "@type": "geo:wktLiteral"
      },
      "city": "s:addressLocality",
      "state": "s:addressRegion",
      "distance": {
        "@id": "s:Distance",
        "@type": "s:QuantitativeValue"
      },
      "bearing": {
        "@type": "s:QuantitativeValue"
      },
      "value": {
        "@id": "s:value"
      },
      "unitCode": {
        "@id": "s:unitCode",
        "@type": "@id"
      },
      "forecastOffice": {
        "@type": "@id"
      },
      "forecastGridData": {
        "@type": "@id"
      },
      "publicZone": {
        "@type": "@id"
      },
      "county": {
        "@type": "@id"
      }
    }
  ],
  "type": "FeatureCollection",
  "features": [
    {
      "id": "https://api.weather.gov/stations/SLEC1/observations/2023-12-20T20:18:00+00:00",
      "type": "Feature",
      "geometry": {
        "type": "Point",
        "coordinates": [
          -120.86,
          39.64
        ]
      },
      "properties": {
        "@id": "https://api.weather.gov/stations/SLEC1/observations/2023-12-20T20:18:00+00:00",
        "@type": "wx:ObservationStation",
        "elevation": {
          "unitCode": "wmoUnit:m",
          "value": 2033
        },
        "station": "https://api.weather.gov/stations/SLEC1",
        "timestamp": "2023-12-20T20:18:00+00:00",
        "rawMessage": "",
        "textDescription": "",
        "icon": null,
        "presentWeather": [],
        "temperature": {
          "unitCode": "wmoUnit:degC",
          "value": 4.44,
          "qualityControl": "V"
        },
        "dewpoint": {
          "unitCode": "wmoUnit:degC",
          "value": 2.47,
          "qualityControl": "V"
        },
        "windDirection": {
          "unitCode": "wmoUnit:degree_(angle)",
          "value": 145,
          "qualityControl": "V"
        },
        "windSpeed": {
          "unitCode": "wmoUnit:km_h-1",
          "value": 11.268,
          "qualityControl": "V"
        },
        "windGust": {
          "unitCode": "wmoUnit:km_h-1",
          "value": null,
          "qualityControl": "Z"
        },
        "barometricPressure": {
          "unitCode": "wmoUnit:Pa",
          "value": null,
          "qualityControl": "Z"
        },
        "seaLevelPressure": {
          "unitCode": "wmoUnit:Pa",
          "value": null,
          "qualityControl": "Z"
        },
        "visibility": {
          "unitCode": "wmoUnit:m",
          "value": null,
          "qualityControl": "Z"
        },
        "maxTemperatureLast24Hours": {
          "unitCode": "wmoUnit:degC",
          "value": null
        },
        "minTemperatureLast24Hours": {
          "unitCode": "wmoUnit:degC",
          "value": null
        },
        "precipitationLast3Hours": {
          "unitCode": "wmoUnit:mm",
          "value": null,
          "qualityControl": "Z"
        },
        "relativeHumidity": {
          "unitCode": "wmoUnit:percent",
          "value": 87.032575256473,
          "qualityControl": "V"
        },
        "windChill": {
          "unitCode": "wmoUnit:degC",
          "value": 1.7373908915,
          "qualityControl": "V"
        },
        "heatIndex": {
          "unitCode": "wmoUnit:degC",
          "value": null,
          "qualityControl": "V"
        },
        "cloudLayers": []
      }
    }
  ]
}

Related #610

Acceptance criteria

• [ ] todo

[colinmurphy01]

@greg-does-weather can you help me evaluate the importance of this.

This feels medium - high to me

[greg-does-weather]

Very important but not necessarily high priority. If I recall correctly from @shadkeene, the text description and icon will only be available if the observation station has the right set of equipment on it. The ones at airports all have it, but not all of the others. I suspect a "right" answer here might involve building up a complete set of current conditions from a few observation stations. Like, we check the first one, determine it doesn't have everything we need, but we keep the things it does have. On the second station, we fill in any gaps. If we need to go to a third station, we fill in some more.

[colinmurphy01]

Should we discuss the above solution with the API team on wednesday?

[colinmurphy01]

Development on this is blocked.

Next step for this ticket is to investigate what is available to us in MADIS, Other APIs, and via Other Observation Station Networks.

[colinmurphy01]

Other tickets have been created that are related:

• 763

• 764

• 765

[colinmurphy01]

Added to risk tracker to evaluate alternative data sources, value, costs, etc.

[colinmurphy01]

• What does it mean to have an invalid station? (minimal amount of observations)
  • Right now it's only temperature. If temperature is null, we move to the next station
• Should we combine observations from multiple stations?

Let's take a crack at this and then review with a lead from Obs/MADIS?

[mgwalker]

Maybe a heuristic where we weight a handful of required properties and if the observation's score is below some threshold, we reject it. For example, if only the temperature is missing, that observation may be our best option rather than moving to the next one. But if the obs is missing temperature, wind, dewpoint, are all missing, then perhaps it's better to move on.

[coreypieper]

@shadkeene and I spoke today about this, and we're comfortable with sticking w/ just T for now and evaluating as we go.

If we do have the opportunity to weight the parameters, this would be our suggested order:

• Temp
• Dewpoint
• Wind
• sky/wx

We know the 1st fail rate is low now (<3% I think), and we're comfortable with up to 5%.

[colinmurphy01]

@coreypieper could you clarify how the weighting would work.

Would it be if we reach a 5% failure rate if temp is missing, then we look to all 3 of these to see if they exist and keep it?

[coreypieper]

Right now only T is considered and the failure rate is acceptably low. Shad and I agreed on this order of importance:

• T
• Wind
• Dewpoint (regarding heat index, etc)

So, a next step may be adding wind as a trigger for a missing ob, and see how that impacts failure rate. ...then DP, pending how T + wind goes.

[colinmurphy01]

Consideration: build in tests to NewRelic to log these unique error rates.