search

jdemaeyer / brightsky

JSON API for DWD's open weather data.
https://brightsky.dev/
MIT License
290 stars 18 forks source link

Critical: Historical data missing #108

Closed jdemaeyer closed 3 years ago

jdemaeyer commented 3 years ago

Seems like we're not parsing historical data sources properly:

$ curl "https://api.brightsky.dev/weather?lat=52&lon=7.6&date=2020-12-31T23:00:00&last_date=2021-01-01T01:00" | jq .
{
  "weather": [
    {
      "timestamp": "2020-12-31T23:00:00+00:00",
      "source_id": 7003,
      "precipitation": null,
      "pressure_msl": null,
      "sunshine": null,
      "temperature": null,
      "wind_direction": null,
      "wind_speed": null,
      "cloud_cover": null,
      "dew_point": null,
      "relative_humidity": null,
      "visibility": null,
      "wind_gust_direction": 280,
      "wind_gust_speed": 9.4,
      "condition": null,
      "icon": "clear-night"
    },
    {
      "timestamp": "2021-01-01T00:00:00+00:00",
      "source_id": 7003,
      "precipitation": 0,
      "pressure_msl": 1005.2,
      "sunshine": 0,
      "temperature": 2,
      "wind_direction": 230,
      "wind_speed": 2.2,
      "cloud_cover": 87,
      "dew_point": 1.7,
      "relative_humidity": null,
      "visibility": 2700,
      "wind_gust_direction": 240,
      "wind_gust_speed": 3.6,
      "condition": "dry",
      "fallback_source_ids": {
        "wind_speed": 7561,
        "sunshine": 7561,
        "wind_direction": 7561,
        "dew_point": 7561,
        "condition": 7561,
        "pressure_msl": 7561,
        "visibility": 7561,
        "cloud_cover": 7561,
        "precipitation": 7561,
        "temperature": 7561
      },
      "icon": "cloudy"
    },
    {
      "timestamp": "2021-01-01T01:00:00+00:00",
      "source_id": 7561,
      "precipitation": 0,
      "pressure_msl": 1005.3,
      "sunshine": 0,
      "temperature": 1.9,
      "wind_direction": 270,
      "wind_speed": 3.6,
      "cloud_cover": 87,
      "dew_point": 1.9,
      "relative_humidity": 100,
      "visibility": 2500,
      "wind_gust_direction": 270,
      "wind_gust_speed": 5.4,
      "condition": "dry",
      "icon": "cloudy"
    }
  ],
  "sources": [
    {
      "id": 7003,
      "dwd_station_id": "01766",
      "observation_type": "historical",
      "lat": 52.1344,
      "lon": 7.6969,
      "height": 47.8,
      "station_name": "Münster/Osnabrück",
      "wmo_station_id": "10315",
      "first_record": "2010-01-01T00:00:00+00:00",
      "last_record": "2021-01-01T00:00:00+00:00",
      "distance": 16365
    },
    {
      "id": 7561,
      "dwd_station_id": "01766",
      "observation_type": "recent",
      "lat": 52.1344,
      "lon": 7.6969,
      "height": 47.8,
      "station_name": "Münster/Osnabrück",
      "wmo_station_id": "10315",
      "first_record": "2021-01-01T00:00:00+00:00",
      "last_record": "2021-03-21T23:00:00+00:00",
      "distance": 16365
    }
  ]
}

Source looks alright: https://opendata.dwd.de/climate_environment/CDC/observations_germany/climate/hourly/precipitation/historical/stundenwerte_RR_01766_19950901_20191231_hist.zip nevermind, historical records do not include 2020 yet

jdemaeyer commented 3 years ago

Parsing works as intended.

The culprit is that we delete recent records once we have any historical record for that timestamp, but the historical records have only been published for some parameters yet. Turns out I actually anticipated this during development and then failed to check after new historical records were published: https://github.com/jdemaeyer/brightsky/blob/master/brightsky/tasks.py#L77-L83.

Edit: 10-minute extreme wind records are the only parameter with published historical records for 2020 yet.

jdemaeyer commented 3 years ago

I have

That should temporarily mitigate this issue but is not a permanent solution. Importantly, it will break as soon as any new historical records are released.

jdemaeyer commented 3 years ago

Fixed via https://github.com/jdemaeyer/brightsky/commit/8ba7302df11f8fcbbfa803d4ff9f082b881938d1