Weather condition

[mweinelt]

This is great! I was always wondering how to make DWD data actually usable. It's a shame they either can't (weird format) or won't (legal issues). Although the Warnwetter App seems to have a proper API, so it's probably the latter.

I'm currently dreaming up a home-assistant integration for brightsky and I'm missing a string describing the weather condition. This is an excerpt from the darksky component:

MAP_CONDITION = {
    "clear-day": "sunny",
    "clear-night": "clear-night",
    "rain": "rainy",
    "snow": "snowy",
    "sleet": "snowy-rainy",
    "wind": "windy",
    "fog": "fog",
    "cloudy": "cloudy",
    "partly-cloudy-day": "partlycloudy",
    "partly-cloudy-night": "partlycloudy",
    "hail": "hail",
    "thunderstorm": "lightning",
    "tornado": None,
}

Is that something that's available in the data, or is there only numeric data?

[jdemaeyer]

Hi Martin, thanks for the kind words :)

A qualitative description of the weather is available in the forecasts (code table) and in the current observations (code table), I have not yet been able to find it for the recent/historical observations.

I have been putting off looking into these so far because to make them useful like the Dark Sky ones, we would have to somehow have to reduce and normalize the two code tables (which are not equivalent). That would break a "just provide the DWD data as is" rule I've tried to follow so far. Also I'm not too happy about having the field always null for past observations.

Alternatively, we could come up with our own logic for a qualitative weather description based on the numeric data that we have (and clearly mark that this is our own interpretation). We are missing "type of precipitation" data for that currently but iirc that is available for all data sources so we can simply add it.

In summary: I'm very open to input or PRs on this :D

[mweinelt]

A qualitative description of the weather is available in the forecasts (code table) and in the current observations (code table), I have not yet been able to find it for the recent/historical observations.

That's something!

I have been putting off looking into these so far because to make them useful like the Dark Sky ones, we would have to somehow have to reduce and normalize the two code tables (which are not equivalent). That would break a "just provide the DWD data as is" rule I've tried to follow so far. Also I'm not too happy about having the field always null for past observations.

I sympathize a lot with this rule.

Alternatively, we could come up with our own logic for a qualitative weather description based on the numeric data that we have (and clearly mark that this is our own interpretation). We are missing "type of precipitation" data for that currently but iirc that is available for all data sources so we can simply add it.

Wouldn't it be easier to try and puzzle the two code tables to map similar conditions?

[EricMc1289]

Hallo and a great thanks for your work. I am asking here because i am wondering where to find the weather code in the response? I am just testing a little bit with the dev api and trying to get the weather code from this sample, but it isn't returned so far. https://api.brightsky.dev/weather?date=2020-07-11&last_date=2020-07-12&lat=51.58&lon=7.38 Is there anything i need to add to the url or am i missing something?

Thanks for any help

[jdemaeyer]

Hi Eric! We currently don't scrape the weather codes from the DWD data (hence this issue), so unfortunately there's no way to get them in your Bright Sky responses at the moment.

This is arguably a much requested feature, so I'm tempted to prioritize it over some other stuff I had on the near-term roadmap. I will keep you guys updated.

[mweinelt]

From the linked issues this documented looks very interesting as it models the conditions for the different weather conditions.

https://www.dwd.de/DE/forschung/wettervorhersage/num_modellierung/01_num_vorhersagemodelle/01c_wetterinterpretation/wetter_interpretation.pdf?__blob=publicationFile&v=6

[jdemaeyer]

Dropping a couple of thoughts into my pensieve after half a day of plowing through raw data and DWD specs:

1. As a first implementation I think we should try to reproduce Dark Sky's icon field, which can currently take the values snow, sleet, rain, wind, fog, cloudy, partly-cloudy-day, partly-cloudy-night, clear-day and clear-night.
2. Note that this does not include thunderstorm, hail, and tornado, which they mention they might add in the future but never did.
3. For a start I also want to merge clear-day and clear-night into just clear (and likewise for partly-cloudy) to avoid having to deal with sunrise and sunset times.
4. This leaves us with eight possible states (from highest to lowest priority/badness):
   • snow
   • sleet
   • rain
   • wind
   • fog
   • cloudy
   • partly-cloudy
   • clear
5. The icon field should be consistent across all observation types. This rules out directly reading any existing "present weather" field from the raw data, because such fields only exists for current and forecasted weather, but not for historical weather.
6. Using the "Wetterinterpretation" algorithm from above seems very heavy and requires a bunch of meteorological parameters that we don't have.
7. If we add precipitation_type to all our weather records we should be able to map our numerical weather parameters into one of the eight states with a simple decision tree:

   if precipitation:
       return precipitation_type
   elif wind > SOME_THRESHOLD:
       return 'wind'
   elif visibility < SOME_THRESHOLD:
       return 'fog'
   elif cloud_cover >= 75:
       return 'cloudy'
   elif cloud_cover >= 25:
       return 'partly-cloudy'
   return 'clear'

8. The precipitation type is available in all source types:
   1. Derived from "significant weather" (ww) in the MOSMIX data,
   2. As formOfPrecipitation in the SYNOP data,
   3. As present_weather in the current data,
   4. As WWTR in the recent/historical precipitation data.
9. The WWTR field is set to null every third hour in the recent/historical data (which is explained as "following SYNOP standard" but seems odd because it is available in the current data), we will need to come up with a "fill from previous/next hour" logic and clearly document that this field is not taken 1-to-1 from the raw DWD data.

I'm hoping to make significant progress on this tomorrow but don't wanna promise too much.

@mweinelt @EricMc1289 How annoying is the merging of clear-day and clear-night into clear for your use cases?

[mweinelt]

That sounds like a good approach, keeping it simple.

Merging of clear-day and clear-night just shifts that calculation downstream, I'm not sure what to tell you. :smile: If you think that it is a helpful state, since it basically says sunny vs moony you should probably add it upstream.

Home-assistant uses the astral library for that kind of calculation: https://astral.readthedocs.io/en/latest/index.html

Example:

>>> observer = Observer()
>>> observer.latitude = 49.8809
>>> observer.longitude = 8.67791
>>> observer.elevation = 162
>>> observer
Observer(latitude=49.8809, longitude=8.67791, elevation=162.0)
>>> s = sun(observer, datetime.datetime.now())
>>> s
{'dawn': datetime.datetime(2020, 7, 15, 2, 47, 52, 195183, tzinfo=<UTC>), 'sunrise': datetime.datetime(2020, 7, 15, 3, 31, 12, 974232, tzinfo=<UTC>), 'noon': datetime.datetime(2020, 7, 15, 11, 31, 17, tzinfo=<UTC>), 'sunset': datetime.datetime(2020, 7, 15, 19, 30, 46, 1101, tzinfo=<UTC>), 'dusk': datetime.datetime(2020, 7, 15, 20, 13, 53, 745449, tzinfo=<UTC>)}

The interesting parts are sunset and sunrise. So it should come down to a comparison like so:

def is_night(latitude: float, longitude: float, elevation: float, date: datetime.datetime) -> bool:
    observer = astral.Observer()
    observer.latitude = latitude
    observer.longitude = longitude
    observer.elevation = elevation
    today = astral.sun.sun(observer, date)
    if date < today.get('sunrise'):
        return True
    tomorrow = astral.sun.sun(observer, date + datetime.timedelta(days=1))
    return date > today.get('sunset') and date <= tomorrow.get('sunrise')

if precipitation:
    return precipitation_type
elif wind > SOME_THRESHOLD:
    return 'wind'
elif visibility < SOME_THRESHOLD:
    return 'fog'
elif cloud_cover >= 75:
    return 'cloudy'
elif cloud_cover >= 25:
    return 'partly-cloudy'
return 'clear-night' if is_night(station.lat, station.lon, station.height, date) else 'clear-day'

So whether I do that downstream or you do it upstream, I don't mind.

[jdemaeyer]

Merging of clear-day and clear-night just shifts that calculation downstream, I'm not sure what to tell you. :smile:

I was hoping you would say "we don't need the distinction anyways". :D

But yeah, after having slept over it the distinction will definitely be necessary for most use cases using this field, and implementing it later will give us a headache about backwards compatibility. I'll add it right away, thanks for the hint to astral!

Blocked by: #91

[jdemaeyer]

Aalrighty, weather records in the response will now contain an icon field holding one of the following values:

• clear-day
• clear-night
• partly-cloudy-day
• partly-cloudy-night
• cloudy
• fog
• wind
• rain
• sleet
• snow
• hail
• thunderstorm

[mweinelt]

Cool, I'll give it a test-drive ASAP.

[mweinelt]

Okay, this looks good! One minor headache I now need to address is how to get from 24 hourly conditions to one for a whole day. If you have a concept of how this is done, I'd appreciated it to hear it.

Edit: What I'm currently doing is pick the worst condition out of those that are projected over a day and go with that.

[jdemaeyer]

Great to hear that it seems to work!

Edit: What I'm currently doing is pick the worst condition out of those that are projected over a day and go with that.

Yeah, I think that's what Dark Sky does as well (from their FAQ)... I don't have any better idea right now unfortunately