NULL values for some locations.

poetaster commented 1 year ago

In the past months I've seen a lot of the below for locations like Strausberg Using GPS I've switched between the two stations here, and that makes no difference. It seems to be the past hours of the current day that are affected, but it may also be the early hours of the day?

precipitation: null,
pressure_msl: null,
sunshine: null,
temperature: null,
wind_direction: null,
wind_speed: null,
cloud_cover: null,
dew_point: null,
relative_humidity: null,
visibility: null,
wind_gust_direction: 140,
wind_gust_speed: 13.3,
condition: null,
precipitation_probability: null,
precipitation_probability_6h: null,

Currently I'm catching that, but it's become noticeable since overviews are now displaying 0 - 20 when the actual range was 7 -20.

jdemaeyer commented 1 year ago

Hi @poetaster, thanks a lot for the report! This may be related to how we prepare ten-minute measurements into hourly measurements, given that that's what we do for wind gusts only and those are the only non-null values in your example. Could you give me the actual request URL for the data you posted above?

poetaster commented 1 year ago

Ok, here it is:

https://api.brightsky.dev/weather?tz=Europe/Berlin&lat=52.58&lon=13.88&date=2023-05-23

jdemaeyer commented 1 year ago

Thanks, I'll try to have a look into this tomorrow!

Notes to myself

Why are the records for 0:00 through 2:00 not filled up from the current source (which does have data, see below)?
Why is the solar data available for 2:00 (i.e. 0:00 UTC) but not for the two records before?

Request for lat/lon

https://api.brightsky.dev/weather?tz=Europe/Berlin&lat=52.58&lon=13.88&date=2023-05-23

{
  "weather": [
    {
      "timestamp": "2023-05-23T00:00:00+02:00",
      "source_id": 7134,
      "precipitation": null,
      "pressure_msl": null,
      "sunshine": null,
      "temperature": null,
      "wind_direction": null,
      "wind_speed": null,
      "cloud_cover": null,
      "dew_point": null,
      "relative_humidity": null,
      "visibility": null,
      "wind_gust_direction": 140,
      "wind_gust_speed": 13.3,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "fallback_source_ids": {},
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T01:00:00+02:00",
      "source_id": 7134,
      "precipitation": null,
      "pressure_msl": null,
      "sunshine": null,
      "temperature": null,
      "wind_direction": null,
      "wind_speed": null,
      "cloud_cover": null,
      "dew_point": null,
      "relative_humidity": null,
      "visibility": null,
      "wind_gust_direction": 110,
      "wind_gust_speed": 11.9,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "fallback_source_ids": {},
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T02:00:00+02:00",
      "source_id": 7134,
      "precipitation": null,
      "pressure_msl": null,
      "sunshine": null,
      "temperature": null,
      "wind_direction": null,
      "wind_speed": null,
      "cloud_cover": null,
      "dew_point": null,
      "relative_humidity": null,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": null,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": 0,
      "fallback_source_ids": {
        "solar": 193071
      },
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T03:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 11.7,
      "wind_direction": 30,
      "wind_speed": 5,
      "cloud_cover": null,
      "dew_point": 8,
      "relative_humidity": 78,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 10.1,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "fallback_source_ids": {},
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T04:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 12.7,
      "wind_direction": 50,
      "wind_speed": 9,
      "cloud_cover": null,
      "dew_point": 8.1,
      "relative_humidity": 74,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 11.2,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "fallback_source_ids": {},
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T05:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 12.6,
      "wind_direction": 50,
      "wind_speed": 6.8,
      "cloud_cover": null,
      "dew_point": 8.3,
      "relative_humidity": 75,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 11.9,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "fallback_source_ids": {},
      "icon": "clear-day"
    },
    {
      "timestamp": "2023-05-23T06:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": 1011.9,
      "sunshine": 0,
      "temperature": 12.5,
      "wind_direction": 50,
      "wind_speed": 5,
      "cloud_cover": 78,
      "dew_point": 8.7,
      "relative_humidity": 78,
      "visibility": 32500,
      "wind_gust_direction": null,
      "wind_gust_speed": 11.9,
      "condition": "dry",
      "precipitation_probability": 12,
      "precipitation_probability_6h": null,
      "solar": 0.008,
      "fallback_source_ids": {
        "visibility": 1700,
        "condition": 1700,
        "pressure_msl": 1700,
        "cloud_cover": 1700,
        "solar": 1700,
        "precipitation_probability": 1700
      },
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T07:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": 1011.7,
      "sunshine": 0,
      "temperature": 13.1,
      "wind_direction": 10,
      "wind_speed": 2.2,
      "cloud_cover": 78,
      "dew_point": 10,
      "relative_humidity": 81,
      "visibility": 32700,
      "wind_gust_direction": null,
      "wind_gust_speed": 9,
      "condition": "dry",
      "precipitation_probability": 16,
      "precipitation_probability_6h": null,
      "solar": 0.058,
      "fallback_source_ids": {
        "visibility": 1700,
        "condition": 1700,
        "pressure_msl": 1700,
        "cloud_cover": 1700,
        "solar": 1700,
        "precipitation_probability": 1700
      },
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T08:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": 1011.6,
      "sunshine": 0,
      "temperature": 13.9,
      "wind_direction": 60,
      "wind_speed": 7.9,
      "cloud_cover": 71,
      "dew_point": 10,
      "relative_humidity": 77,
      "visibility": 32700,
      "wind_gust_direction": null,
      "wind_gust_speed": 13,
      "condition": "dry",
      "precipitation_probability": 15,
      "precipitation_probability_6h": 37,
      "solar": 0.136,
      "fallback_source_ids": {
        "visibility": 1700,
        "condition": 1700,
        "pressure_msl": 1700,
        "cloud_cover": 1700,
        "precipitation_probability_6h": 1700,
        "solar": 1700,
        "precipitation_probability": 1700
      },
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T09:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1011.5,
      "sunshine": 25,
      "temperature": 17.3,
      "wind_direction": 152,
      "wind_speed": 7.4,
      "cloud_cover": 75,
      "dew_point": 10.5,
      "relative_humidity": null,
      "visibility": 33100,
      "wind_gust_direction": null,
      "wind_gust_speed": 11.1,
      "condition": "dry",
      "precipitation_probability": 21,
      "precipitation_probability_6h": null,
      "solar": 0.242,
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T10:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1011.4,
      "sunshine": 25,
      "temperature": 18.2,
      "wind_direction": 224,
      "wind_speed": 7.4,
      "cloud_cover": 72,
      "dew_point": 10.7,
      "relative_humidity": null,
      "visibility": 31000,
      "wind_gust_direction": null,
      "wind_gust_speed": 13,
      "condition": "dry",
      "precipitation_probability": 19,
      "precipitation_probability_6h": null,
      "solar": 0.336,
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T11:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1011.5,
      "sunshine": 21,
      "temperature": 19,
      "wind_direction": 257,
      "wind_speed": 9.3,
      "cloud_cover": 71,
      "dew_point": 11.1,
      "relative_humidity": null,
      "visibility": 31400,
      "wind_gust_direction": null,
      "wind_gust_speed": 18.5,
      "condition": "dry",
      "precipitation_probability": 15,
      "precipitation_probability_6h": null,
      "solar": 0.403,
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T12:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1011.6,
      "sunshine": 17,
      "temperature": 19.8,
      "wind_direction": 270,
      "wind_speed": 13,
      "cloud_cover": 73,
      "dew_point": 11.3,
      "relative_humidity": null,
      "visibility": 31300,
      "wind_gust_direction": null,
      "wind_gust_speed": 22.2,
      "condition": "dry",
      "precipitation_probability": 15,
      "precipitation_probability_6h": null,
      "solar": 0.439,
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T13:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1011.7,
      "sunshine": 14,
      "temperature": 20.1,
      "wind_direction": 274,
      "wind_speed": 16.7,
      "cloud_cover": 77,
      "dew_point": 11.6,
      "relative_humidity": null,
      "visibility": 31200,
      "wind_gust_direction": null,
      "wind_gust_speed": 27.8,
      "condition": "dry",
      "precipitation_probability": 11,
      "precipitation_probability_6h": null,
      "solar": 0.436,
      "icon": "partly-cloudy-day"
    },
    {
      "timestamp": "2023-05-23T14:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0.4,
      "pressure_msl": 1011.9,
      "sunshine": 9,
      "temperature": 19.9,
      "wind_direction": 280,
      "wind_speed": 20.4,
      "cloud_cover": 84,
      "dew_point": 11.9,
      "relative_humidity": null,
      "visibility": 29400,
      "wind_gust_direction": null,
      "wind_gust_speed": 33.3,
      "condition": "rain",
      "precipitation_probability": 13,
      "precipitation_probability_6h": 48,
      "solar": 0.4,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T15:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0.1,
      "pressure_msl": 1012.2,
      "sunshine": 8,
      "temperature": 19.5,
      "wind_direction": 287,
      "wind_speed": 22.2,
      "cloud_cover": 86,
      "dew_point": 12.1,
      "relative_humidity": null,
      "visibility": 29400,
      "wind_gust_direction": null,
      "wind_gust_speed": 37,
      "condition": "rain",
      "precipitation_probability": 21,
      "precipitation_probability_6h": null,
      "solar": 0.358,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T16:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0.1,
      "pressure_msl": 1012.6,
      "sunshine": 6,
      "temperature": 19,
      "wind_direction": 290,
      "wind_speed": 24.1,
      "cloud_cover": 87,
      "dew_point": 12.3,
      "relative_humidity": null,
      "visibility": 28800,
      "wind_gust_direction": null,
      "wind_gust_speed": 40.8,
      "condition": "rain",
      "precipitation_probability": 22,
      "precipitation_probability_6h": null,
      "solar": 0.297,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T17:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0.1,
      "pressure_msl": 1013,
      "sunshine": 6,
      "temperature": 17.8,
      "wind_direction": 295,
      "wind_speed": 25.9,
      "cloud_cover": 86,
      "dew_point": 11.8,
      "relative_humidity": null,
      "visibility": 28300,
      "wind_gust_direction": null,
      "wind_gust_speed": 42.6,
      "condition": "rain",
      "precipitation_probability": 23,
      "precipitation_probability_6h": null,
      "solar": 0.242,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T18:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0.2,
      "pressure_msl": 1013.4,
      "sunshine": 5,
      "temperature": 17.4,
      "wind_direction": 300,
      "wind_speed": 25.9,
      "cloud_cover": 89,
      "dew_point": 11.7,
      "relative_humidity": null,
      "visibility": 28900,
      "wind_gust_direction": null,
      "wind_gust_speed": 42.6,
      "condition": "rain",
      "precipitation_probability": 31,
      "precipitation_probability_6h": null,
      "solar": 0.178,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T19:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0.1,
      "pressure_msl": 1013.9,
      "sunshine": 6,
      "temperature": 16.5,
      "wind_direction": 302,
      "wind_speed": 25.9,
      "cloud_cover": 91,
      "dew_point": 10.9,
      "relative_humidity": null,
      "visibility": 29200,
      "wind_gust_direction": null,
      "wind_gust_speed": 40.8,
      "condition": "rain",
      "precipitation_probability": 29,
      "precipitation_probability_6h": null,
      "solar": 0.119,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T20:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0.1,
      "pressure_msl": 1014.5,
      "sunshine": 5,
      "temperature": 15.4,
      "wind_direction": 304,
      "wind_speed": 24.1,
      "cloud_cover": 92,
      "dew_point": 10,
      "relative_humidity": null,
      "visibility": 31300,
      "wind_gust_direction": null,
      "wind_gust_speed": 40.8,
      "condition": "rain",
      "precipitation_probability": 26,
      "precipitation_probability_6h": 73,
      "solar": 0.064,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T21:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1015.2,
      "sunshine": 3,
      "temperature": 14.7,
      "wind_direction": 304,
      "wind_speed": 22.2,
      "cloud_cover": 91,
      "dew_point": 9.2,
      "relative_humidity": null,
      "visibility": 30900,
      "wind_gust_direction": null,
      "wind_gust_speed": 38.9,
      "condition": "dry",
      "precipitation_probability": 19,
      "precipitation_probability_6h": null,
      "solar": 0.019,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T22:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1015.8,
      "sunshine": 1,
      "temperature": 13.9,
      "wind_direction": 302,
      "wind_speed": 20.4,
      "cloud_cover": 89,
      "dew_point": 8.6,
      "relative_humidity": null,
      "visibility": 32300,
      "wind_gust_direction": null,
      "wind_gust_speed": 35.2,
      "condition": "dry",
      "precipitation_probability": 13,
      "precipitation_probability_6h": null,
      "solar": 0,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-23T23:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1016.4,
      "sunshine": 0,
      "temperature": 13.2,
      "wind_direction": 301,
      "wind_speed": 18.5,
      "cloud_cover": 87,
      "dew_point": 7.8,
      "relative_humidity": null,
      "visibility": 34400,
      "wind_gust_direction": null,
      "wind_gust_speed": 31.5,
      "condition": "dry",
      "precipitation_probability": 8,
      "precipitation_probability_6h": null,
      "solar": 0,
      "icon": "cloudy"
    },
    {
      "timestamp": "2023-05-24T00:00:00+02:00",
      "source_id": 1700,
      "precipitation": 0,
      "pressure_msl": 1016.8,
      "sunshine": 0,
      "temperature": 12.5,
      "wind_direction": 296,
      "wind_speed": 18.5,
      "cloud_cover": 83,
      "dew_point": 7.5,
      "relative_humidity": null,
      "visibility": 35300,
      "wind_gust_direction": null,
      "wind_gust_speed": 31.5,
      "condition": "dry",
      "precipitation_probability": 3,
      "precipitation_probability_6h": null,
      "solar": 0,
      "icon": "cloudy"
    }
  ],
  "sources": [
    {
      "id": 7134,
      "dwd_station_id": "03376",
      "observation_type": "historical",
      "lat": 52.5176,
      "lon": 14.1232,
      "height": 63,
      "station_name": "Müncheberg",
      "wmo_station_id": "F475",
      "first_record": "2010-01-01T00:00:00+00:00",
      "last_record": "2023-05-23T00:00:00+00:00",
      "distance": 17868
    },
    {
      "id": 193071,
      "dwd_station_id": "00427",
      "observation_type": "historical",
      "lat": 52.3807,
      "lon": 13.5306,
      "height": 45.59,
      "station_name": "Berlin Brandenburg",
      "wmo_station_id": "10385",
      "first_record": "2010-01-01T00:00:00+00:00",
      "last_record": "2023-05-23T00:00:00+00:00",
      "distance": 32456
    },
    {
      "id": 56854,
      "dwd_station_id": "03376",
      "observation_type": "current",
      "lat": 52.52,
      "lon": 14.13,
      "height": 63,
      "station_name": "MUENCHEBERG",
      "wmo_station_id": "F475",
      "first_record": "2023-05-21T07:00:00+00:00",
      "last_record": "2023-05-23T06:00:00+00:00",
      "distance": 18193
    },
    {
      "id": 1700,
      "dwd_station_id": null,
      "observation_type": "forecast",
      "lat": 52.58,
      "lon": 13.88,
      "height": 73,
      "station_name": "STRAUSBERG",
      "wmo_station_id": "X429",
      "first_record": "2023-05-23T04:00:00+00:00",
      "last_record": "2023-06-02T10:00:00+00:00",
      "distance": 0
    }
  ]
}

Request for `current` source

https://api.brightsky.dev/weather?tz=Europe/Berlin&source_id=56854&date=2023-05-23

{
  "weather": [
    {
      "timestamp": "2023-05-23T00:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 17.4,
      "wind_direction": 130,
      "wind_speed": 9,
      "cloud_cover": null,
      "dew_point": 6.9,
      "relative_humidity": 50,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 13,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T01:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 15.3,
      "wind_direction": 40,
      "wind_speed": 6.8,
      "cloud_cover": null,
      "dew_point": 7.4,
      "relative_humidity": 59,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 13,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T02:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 14,
      "wind_direction": 50,
      "wind_speed": 6.8,
      "cloud_cover": null,
      "dew_point": 7.5,
      "relative_humidity": 65,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 9,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T03:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 11.7,
      "wind_direction": 30,
      "wind_speed": 5,
      "cloud_cover": null,
      "dew_point": 8,
      "relative_humidity": 78,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 10.1,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T04:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 12.7,
      "wind_direction": 50,
      "wind_speed": 9,
      "cloud_cover": null,
      "dew_point": 8.1,
      "relative_humidity": 74,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 11.2,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-night"
    },
    {
      "timestamp": "2023-05-23T05:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 12.6,
      "wind_direction": 50,
      "wind_speed": 6.8,
      "cloud_cover": null,
      "dew_point": 8.3,
      "relative_humidity": 75,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 11.9,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-day"
    },
    {
      "timestamp": "2023-05-23T06:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 12.5,
      "wind_direction": 50,
      "wind_speed": 5,
      "cloud_cover": null,
      "dew_point": 8.7,
      "relative_humidity": 78,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 11.9,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-day"
    },
    {
      "timestamp": "2023-05-23T07:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 13.1,
      "wind_direction": 10,
      "wind_speed": 2.2,
      "cloud_cover": null,
      "dew_point": 10,
      "relative_humidity": 81,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 9,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-day"
    },
    {
      "timestamp": "2023-05-23T08:00:00+02:00",
      "source_id": 56854,
      "precipitation": 0,
      "pressure_msl": null,
      "sunshine": 0,
      "temperature": 13.9,
      "wind_direction": 60,
      "wind_speed": 7.9,
      "cloud_cover": null,
      "dew_point": 10,
      "relative_humidity": 77,
      "visibility": null,
      "wind_gust_direction": null,
      "wind_gust_speed": 13,
      "condition": null,
      "precipitation_probability": null,
      "precipitation_probability_6h": null,
      "solar": null,
      "icon": "clear-day"
    }
  ],
  "sources": [
    {
      "id": 56854,
      "dwd_station_id": "03376",
      "observation_type": "current",
      "lat": 52.52,
      "lon": 14.13,
      "height": 63,
      "station_name": "MUENCHEBERG",
      "wmo_station_id": "F475",
      "first_record": "2023-05-21T07:00:00+00:00",
      "last_record": "2023-05-23T06:00:00+00:00"
    }
  ]
}

poetaster commented 1 year ago

I just checked and current weather looks the same on a number of locations:

https://api.brightsky.dev/current_weather?tz=Europe/Berlin&lat=52.58&lon=13.88&date=2023-05-23

jdemaeyer commented 1 year ago

Hmm interesting, the current_weather endpoint uses very different data sources and shouldn't be affected by any "ten minutes to hourly" shenanigans. For that particular URL, increasing the range of possible fallback stations by adding &max_dist=100000 seems to work, can you try that for your other locations, too?

jdemaeyer commented 1 year ago

Hmm, scratch that, given how close Strausberg is to Berlin, increasing the fallback distance really shouldn't be necessary.

The synop records (which are the basis for the current_weather endpoint) are a little tricky because they publish little bits of data scattered throughout the day, and the different stations are not synchronized. That could be what's causing trouble here. (We already have data from Strausberg for 7:30 UTC but only from 7:00 UTC or 7:10 UTC for most other stations.)

poetaster commented 1 year ago

I'm not actually using current_weather so have not observed whether this is new behaviour of not. With the weather endpoint, this early morning nulls is a new development, sometime in the last 1-2 months.

poetaster commented 1 year ago

, increasing the range of possible fallback stations by adding &max_dist=100000

Hmmm. I wonder if this is because Müncheberg has some kind of exceptional role in the scheme of things. That works with the /weather endpoint as well. Hmmm.

poetaster commented 1 year ago

Oh. Just adding &max_dist=500 gets the correct data on the /weather endpoint.

jdemaeyer commented 1 year ago

Oh. Just adding &max_dist=500 gets the correct data on the /weather endpoint.

That's an optical illusion, unfortunately :sweat_smile: Reducing the fallback range by that much will just kick out all the real measurement stations and leave you with nothing but forecasts (including old forecasts for the past 3-4 hours)

jdemaeyer commented 1 year ago

There seem to be three separate but related issues at play here:

The historical ten-minute data (wind_gust_direction, wind_gust_speed, and solar) is published way earlier than all other historical data (0:50 UTC vs ~ 9:00 UTC), leaving us with an eight-hour span where we have almost-empty historical records for yesterday (UTC) and need to trigger our fallback mechanism for everything else.
Moreover, the solar radiation data contains data for midnight today, so there is an extra almost-empty historical record for that timestamp, which is empty except for solar.
A lot of stations do not measure solar, leading to them being disqualified for the second round of fallback, and thus to the empty fields remaining null.

That last issue should be the main culprit causing responses with many nulls after April 12 (when we added the solar parameter).

I have added a workaround for the second issue via 20438c6, and addressed the third issue via de2f53e. This should significantly improve the "a lot of nulls" issue at the price of slightly degraded completeness for solar.

@poetaster can you confirm that you're getting much more complete data now?

poetaster commented 1 year ago

@poetaster can you confirm that you're getting much more complete data now?

Yes. I've done some work to make the new precipitation values visible and noticed last week that the null values have disappeared. Thank you!

jdemaeyer commented 1 year ago

Follow-up: #150

jdemaeyer / brightsky