search

jdemaeyer / brightsky

JSON API for DWD's open weather data.
https://brightsky.dev/
MIT License
287 stars 18 forks source link

api output differs in demo and from endpoint. #118

Closed poetaster closed 3 years ago

poetaster commented 3 years ago

I've had http://github.com/harbour-dwd running for some time now and noticed that values from the demo (where I was looking for some inspiration in the javascript) where differing from my app. The I checked the endpoint in browser:

In the demo summary, 10 AM:

timestamp: "2021-06-23T10:00:00+02:00",
source_id: 1700,
precipitation: 0,
pressure_msl: 1019.1,
sunshine: 36,
temperature: 19.3,
wind_direction: 317,
wind_speed: 7.4,
cloud_cover: 67,
dew_point: 12.5,
relative_humidity: null,
visibility: 36000,
wind_gust_direction: null,
wind_gust_speed: 14.8,
condition: "dry",
icon: "partly-cloudy-day"

In a browser tab:

"timestamp": "2021-06-23T10:00:00+00:00",
"source_id": 1700,
"precipitation": 0.0,
"pressure_msl": 1019.3,
"sunshine": 32.0,
"temperature": 20.6,
"wind_direction": 282,
"wind_speed": 9.3,
"cloud_cover": 80,
"dew_point": 11.8,
"relative_humidity": null,
"visibility": 38100,
"wind_gust_direction": null,
"wind_gust_speed": 18.5,
"condition": "dry",
"icon": "cloudy"

The values from the demo look 'closer' to real than those from the raw endpoint? I did note the Timezone info but that doesn't seem relavent?

jdemaeyer commented 3 years ago

Heyho @poetaster

Those two records you posted contain weather records for different times: The first one is for 10 AM in UTC+2, i.e. Berlin time (notice the +02:00 at the end of the timestamp value), corresponding to 8 AM in UTC+0. The second one is for 10 AM in UTC+0 (which would be noon in Berlin).

By default Bright Sky will always return timestamps in UTC, the demo code specifically requests the Europe/Berlin timezone via the tz parameter.

poetaster commented 3 years ago

I tried to compensate by adding the UTC but it doesn't matter anyway. check the complete records:

This is https://api.brightsky.dev/weather?lat=52.517&lon=14.117&date=2021-06-23 (which is how my app asks)

{

    "weather": [
        {
            "timestamp": "2021-06-23T00:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1017.2,
            "sunshine": 0.0,
            "temperature": 11.6,
            "wind_direction": 200,
            "wind_speed": 2.9,
            "cloud_cover": 0,
            "dew_point": 11.2,
            "relative_humidity": 97,
            "visibility": 22600,
            "wind_gust_direction": null,
            "wind_gust_speed": 4.0,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "clear-night"
        },
        {
            "timestamp": "2021-06-23T01:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1017.4,
            "sunshine": 0.0,
            "temperature": 10.6,
            "wind_direction": 150,
            "wind_speed": 5.0,
            "cloud_cover": 0,
            "dew_point": 10.2,
            "relative_humidity": 97,
            "visibility": 11700,
            "wind_gust_direction": null,
            "wind_gust_speed": 6.1,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "clear-night"
        },
        {
            "timestamp": "2021-06-23T02:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1017.7,
            "sunshine": 0.0,
            "temperature": 10.9,
            "wind_direction": 280,
            "wind_speed": 2.9,
            "cloud_cover": 0,
            "dew_point": 10.5,
            "relative_humidity": 97,
            "visibility": 7700,
            "wind_gust_direction": null,
            "wind_gust_speed": 6.1,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "clear-night"
        },
        {
            "timestamp": "2021-06-23T03:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1018.0,
            "sunshine": 0.0,
            "temperature": 10.5,
            "wind_direction": 330,
            "wind_speed": 5.0,
            "cloud_cover": 0,
            "dew_point": 10.2,
            "relative_humidity": 98,
            "visibility": 4100,
            "wind_gust_direction": null,
            "wind_gust_speed": 6.1,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "clear-day"
        },
        {
            "timestamp": "2021-06-23T04:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1018.2,
            "sunshine": 27.0,
            "temperature": 11.0,
            "wind_direction": 240,
            "wind_speed": 2.2,
            "cloud_cover": 88,
            "dew_point": 10.7,
            "relative_humidity": 98,
            "visibility": 12000,
            "wind_gust_direction": null,
            "wind_gust_speed": 6.8,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "cloudy"
        },
        {
            "timestamp": "2021-06-23T05:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1019.0,
            "sunshine": 49.0,
            "temperature": 13.6,
            "wind_direction": 350,
            "wind_speed": 5.0,
            "cloud_cover": 50,
            "dew_point": 11.8,
            "relative_humidity": 89,
            "visibility": 17100,
            "wind_gust_direction": null,
            "wind_gust_speed": 7.9,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T06:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1019.2,
            "sunshine": 60.0,
            "temperature": 15.8,
            "wind_direction": 360,
            "wind_speed": 7.9,
            "cloud_cover": 100,
            "dew_point": 11.6,
            "relative_humidity": 76,
            "visibility": 29200,
            "wind_gust_direction": null,
            "wind_gust_speed": 13.0,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "cloudy"
        },
        {
            "timestamp": "2021-06-23T07:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1019.8,
            "sunshine": 60.0,
            "temperature": 17.8,
            "wind_direction": 60,
            "wind_speed": 6.8,
            "cloud_cover": 0,
            "dew_point": 11.7,
            "relative_humidity": 67,
            "visibility": 33700,
            "wind_gust_direction": null,
            "wind_gust_speed": 14.0,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "clear-day"
        },
        {
            "timestamp": "2021-06-23T08:00:00+00:00",
            "source_id": 56854,
            "precipitation": 0.0,
            "pressure_msl": 1019.7,
            "sunshine": 60.0,
            "temperature": 19.8,
            "wind_direction": 320,
            "wind_speed": 5.0,
            "cloud_cover": 13,
            "dew_point": 11.3,
            "relative_humidity": 58,
            "visibility": 48700,
            "wind_gust_direction": null,
            "wind_gust_speed": 15.8,
            "condition": "dry",
            "fallback_source_ids": {
                "cloud_cover": 6154,
                "pressure_msl": 6154,
                "condition": 6154,
                "visibility": 6154
            },
            "icon": "clear-day"
        },
        {
            "timestamp": "2021-06-23T09:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.1,
            "sunshine": 45.0,
            "temperature": 20.4,
            "wind_direction": 318,
            "wind_speed": 7.4,
            "cloud_cover": 70,
            "dew_point": 11.4,
            "relative_humidity": null,
            "visibility": 33400,
            "wind_gust_direction": null,
            "wind_gust_speed": 16.7,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T10:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.3,
            "sunshine": 40.0,
            "temperature": 20.7,
            "wind_direction": 302,
            "wind_speed": 7.4,
            "cloud_cover": 78,
            "dew_point": 11.3,
            "relative_humidity": null,
            "visibility": 35200,
            "wind_gust_direction": null,
            "wind_gust_speed": 18.5,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T11:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.3,
            "sunshine": 33.0,
            "temperature": 21.0,
            "wind_direction": 291,
            "wind_speed": 7.4,
            "cloud_cover": 80,
            "dew_point": 11.5,
            "relative_humidity": null,
            "visibility": 36500,
            "wind_gust_direction": null,
            "wind_gust_speed": 20.4,
            "condition": "dry",
            "icon": "cloudy"
        },
        {
            "timestamp": "2021-06-23T12:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.3,
            "sunshine": 25.0,
            "temperature": 21.2,
            "wind_direction": 282,
            "wind_speed": 9.3,
            "cloud_cover": 86,
            "dew_point": 11.8,
            "relative_humidity": null,
            "visibility": 35800,
            "wind_gust_direction": null,
            "wind_gust_speed": 20.4,
            "condition": "dry",
            "icon": "cloudy"
        },
        {
            "timestamp": "2021-06-23T13:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.2,
            "sunshine": 20.0,
            "temperature": 21.4,
            "wind_direction": 285,
            "wind_speed": 9.3,
            "cloud_cover": 84,
            "dew_point": 12.0,
            "relative_humidity": null,
            "visibility": 34600,
            "wind_gust_direction": null,
            "wind_gust_speed": 20.4,
            "condition": "dry",
            "icon": "cloudy"
        },
        {
            "timestamp": "2021-06-23T14:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.2,
            "sunshine": 16.0,
            "temperature": 21.4,
            "wind_direction": 294,
            "wind_speed": 9.3,
            "cloud_cover": 81,
            "dew_point": 12.1,
            "relative_humidity": null,
            "visibility": 34000,
            "wind_gust_direction": null,
            "wind_gust_speed": 18.5,
            "condition": "dry",
            "icon": "cloudy"
        },
        {
            "timestamp": "2021-06-23T15:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.0,
            "sunshine": 16.0,
            "temperature": 21.6,
            "wind_direction": 292,
            "wind_speed": 9.3,
            "cloud_cover": 78,
            "dew_point": 12.4,
            "relative_humidity": null,
            "visibility": 32500,
            "wind_gust_direction": null,
            "wind_gust_speed": 18.5,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T16:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.0,
            "sunshine": 20.0,
            "temperature": 21.6,
            "wind_direction": 306,
            "wind_speed": 9.3,
            "cloud_cover": 73,
            "dew_point": 12.2,
            "relative_humidity": null,
            "visibility": 33800,
            "wind_gust_direction": null,
            "wind_gust_speed": 18.5,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T17:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.1,
            "sunshine": 22.0,
            "temperature": 21.4,
            "wind_direction": 313,
            "wind_speed": 7.4,
            "cloud_cover": 66,
            "dew_point": 12.4,
            "relative_humidity": null,
            "visibility": 34700,
            "wind_gust_direction": null,
            "wind_gust_speed": 18.5,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T18:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.3,
            "sunshine": 23.0,
            "temperature": 20.7,
            "wind_direction": 319,
            "wind_speed": 7.4,
            "cloud_cover": 58,
            "dew_point": 12.6,
            "relative_humidity": null,
            "visibility": 35600,
            "wind_gust_direction": null,
            "wind_gust_speed": 16.7,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T19:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.3,
            "sunshine": 21.0,
            "temperature": 19.5,
            "wind_direction": 329,
            "wind_speed": 5.5,
            "cloud_cover": 52,
            "dew_point": 12.7,
            "relative_humidity": null,
            "visibility": 35800,
            "wind_gust_direction": null,
            "wind_gust_speed": 14.8,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },
        {
            "timestamp": "2021-06-23T20:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.5,
            "sunshine": 12.0,
            "temperature": 17.9,
            "wind_direction": 349,
            "wind_speed": 5.5,
            "cloud_cover": 50,
            "dew_point": 13.0,
            "relative_humidity": null,
            "visibility": 37000,
            "wind_gust_direction": null,
            "wind_gust_speed": 11.1,
            "condition": "dry",
            "icon": "partly-cloudy-night"
        },
        {
            "timestamp": "2021-06-23T21:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.7,
            "sunshine": 0.0,
            "temperature": 16.3,
            "wind_direction": 342,
            "wind_speed": 3.7,
            "cloud_cover": 49,
            "dew_point": 13.1,
            "relative_humidity": null,
            "visibility": 37400,
            "wind_gust_direction": null,
            "wind_gust_speed": 9.3,
            "condition": "dry",
            "icon": "partly-cloudy-night"
        },
        {
            "timestamp": "2021-06-23T22:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.7,
            "sunshine": 0.0,
            "temperature": 15.5,
            "wind_direction": 337,
            "wind_speed": 3.7,
            "cloud_cover": 50,
            "dew_point": 13.0,
            "relative_humidity": null,
            "visibility": 36500,
            "wind_gust_direction": null,
            "wind_gust_speed": 9.3,
            "condition": "dry",
            "icon": "partly-cloudy-night"
        },
        {
            "timestamp": "2021-06-23T23:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.8,
            "sunshine": 0.0,
            "temperature": 14.8,
            "wind_direction": 333,
            "wind_speed": 3.7,
            "cloud_cover": 51,
            "dew_point": 12.9,
            "relative_humidity": null,
            "visibility": 35000,
            "wind_gust_direction": null,
            "wind_gust_speed": 7.4,
            "condition": "dry",
            "icon": "partly-cloudy-night"
        },
        {
            "timestamp": "2021-06-24T00:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.6,
            "sunshine": 0.0,
            "temperature": 14.4,
            "wind_direction": 336,
            "wind_speed": 3.7,
            "cloud_cover": 57,
            "dew_point": 12.5,
            "relative_humidity": null,
            "visibility": 30900,
            "wind_gust_direction": null,
            "wind_gust_speed": 9.3,
            "condition": "dry",
            "icon": "partly-cloudy-night"
        }
    ],
    "sources": [
        {
            "id": 56854,
            "dwd_station_id": "03376",
            "observation_type": "current",
            "lat": 52.52,
            "lon": 14.13,
            "height": 63.0,
            "station_name": "MUENCHEBERG",
            "wmo_station_id": "F475",
            "first_record": "2021-06-21T09:00:00+00:00",
            "last_record": "2021-06-23T08:00:00+00:00",
            "distance": 942.0
        },
        {
            "id": 6154,
            "dwd_station_id": "03158",
            "observation_type": "current",
            "lat": 52.55,
            "lon": 14.55,
            "height": 12.0,
            "station_name": "MANSCHNOW",
            "wmo_station_id": "10396",
            "first_record": "2021-06-21T09:00:00+00:00",
            "last_record": "2021-06-23T08:00:00+00:00",
            "distance": 29550.0
        },
        {
            "id": 4139,
            "dwd_station_id": "03376",
            "observation_type": "forecast",
            "lat": 52.52,
            "lon": 14.12,
            "height": 63.0,
            "station_name": "MUENCHEBERG",
            "wmo_station_id": "F475",
            "first_record": "2021-06-23T06:00:00+00:00",
            "last_record": "2021-07-03T07:00:00+00:00",
            "distance": 391.0
        }
    ]

}

And this is the demo response data fetched 30 seconds later (just 12 noon).:

` {

weather: [
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {17 items},
    {16 items},
    {
        timestamp: "2021-06-23T12:00:00+02:00",
        source_id: 4139,
        precipitation: 0,
        pressure_msl: 1019.3,
        sunshine: 40,
        temperature: 20.7,
        wind_direction: 302,
        wind_speed: 7.4,
        cloud_cover: 78,
        dew_point: 11.3,
        relative_humidity: null,
        visibility: 35200,
        wind_gust_direction: null,
        wind_gust_speed: 18.5,
        condition: "dry",
        icon: "partly-cloudy-day"
    },
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items},
    {16 items}
],
sources: [
    {11 items},
    {11 items},
    {11 items},
    {11 items}
]

}

`

jdemaeyer commented 3 years ago

Hm I probably didn't bring my point across clearly enough. 10:00 UTC+0 corresponds to 12:00 UTC+2, and those look exactly the same to me :)

Your response:

        {
            "timestamp": "2021-06-23T10:00:00+00:00",
            "source_id": 4139,
            "precipitation": 0.0,
            "pressure_msl": 1019.3,
            "sunshine": 40.0,
            "temperature": 20.7,
            "wind_direction": 302,
            "wind_speed": 7.4,
            "cloud_cover": 78,
            "dew_point": 11.3,
            "relative_humidity": null,
            "visibility": 35200,
            "wind_gust_direction": null,
            "wind_gust_speed": 18.5,
            "condition": "dry",
            "icon": "partly-cloudy-day"
        },

Demo response:

    {
        timestamp: "2021-06-23T12:00:00+02:00",
        source_id: 4139,
        precipitation: 0,
        pressure_msl: 1019.3,
        sunshine: 40,
        temperature: 20.7,
        wind_direction: 302,
        wind_speed: 7.4,
        cloud_cover: 78,
        dew_point: 11.3,
        relative_humidity: null,
        visibility: 35200,
        wind_gust_direction: null,
        wind_gust_speed: 18.5,
        condition: "dry",
        icon: "partly-cloudy-day"
    },
poetaster commented 3 years ago

Don't get it. I'm getting 24 hours of data for a specific lat/lon? And there is no way for me to specify an offset? What are we supposed to do? The very link that the demo shows produces different output.

poetaster commented 3 years ago

ah, got it. TZ arg

jdemaeyer commented 3 years ago

If you do not specify a timezone, Bright Sky will always assume that you mean UTC.

So if you request https://api.brightsky.dev/weather?lat=52.517&lon=14.117&date=2021-06-23 you will get records from 2021-06-23 00:00 UTC through 2021-06-24 00:00 UTC (i.e. 25 records).

You can set a timezone other than UTC either by providing a more specific timestamp with offset, e.g. https://api.brightsky.dev/weather?lat=52.517&lon=14.117&date=2021-06-23%2000:00+2:00, or by supplying the tz argument: https://api.brightsky.dev/weather?lat=52.517&lon=14.117&date=2021-06-23&tz=Europe/Berlin.

The very link that the demo shows produces different output.

Ah, yes, the shown demo link is omitting the tz argument for clarity while internally it actually does supply the Europe/Berlin timezone.

poetaster commented 3 years ago

Ok, I've been showing people a double fencepost error. https://api.brightsky.dev/weather?lat=52.517&lon=14.117&date=2021-06-23%2000:00+2:00 does not seem to work? also shows without offset ?

jdemaeyer commented 3 years ago

https://api.brightsky.dev/weather?lat=52.517&lon=14.117&date=2021-06-23%2000:00+2:00 does not seem to work?

Ah, yes, apologies, the date selection does work as I meant (notice how the first record is for 2:00 UTC+0) but I forgot that the "displayed" timestamp is determined by the tz argument and that argument alone.

Ok, I've been showing people a double fencepost error.

Note that this "double fencepost" is deliberate and not an error. Most of the weather parameters are momentary, but some are cumulative: the precipitation parameter contains the amount of precipitation that fell during the preceding hour. If we're interested in the weather on June 23, we need both the record from 2021-06-23 0:00, which contains the temperature at 0:00, and the record from 2021-06-24 0:00 which contains the total amount of rain that fell between 2021-06-23 23:00 and 2021-06-24 0:00.

poetaster commented 3 years ago

Ah, ok. That precipitation is a value for the previous hour is good to know. So I need to suply the timezone such that: tz=Europe/Berlin (for instance). And then the hourly times correspond to the actual UTC+(2 in this instance). If only QtQml had a decent ecmascript implementation. sigh. and now comes the bloat.

jdemaeyer commented 3 years ago

Yep, if you set tz=Europe/Berlin Bright Sky will automatically select the correct offset (UTC+2 during summer, UTC+1 during winter)

poetaster commented 3 years ago

Ok I'll close this. Sadly, I'm using a QT which doesn't do modern timezone output (stuck at 5.6 for a bit). so I have to add a javascript library to do olson lookups. pita.