Precipitation probability

[mweinelt]

I think that is probably the last major field that is currently still missing.

The home-assistant data provider is coming along well by the way:

[mweinelt]

I had an interesting observation as to the limitations of what brightsky can actually achieve yesterday. The situation was like this:

• I'm In Darmstadt, Hessen, Germany.
• It started raining
• Bright Sky said it was dry/cloudy

Fast forward 20 minutes:

• Rain had stopped a while back
• Bright Sky now says it is raining

I check the sources and it used the fallback Offenbach-Wetterpark instead of the local station in Darmstadt, which probably didn't provide the condition. Whether that is situational or and exception I'm not sure. Anyway, Offenbach is roughly 28km away, so I understand where the delayed rain condition is coming from.

The limitation that I'm seeing in bright sky now is, that there is no weather model that does interpolation based on the distance between stations around my requested position. And that is certainly a very complex task.

That in turn means that I can only make good use of bright sky when I'm close to a well-equipped DWD station. For Hessen those seem to be Frankfurt Airport, Fritzlar, Gießen, Offenbach and Wasserkuppe: https://www.dwd.de/DE/wetter/wetterundklima_vorort/hessen/hessen_node.html.

[jdemaeyer]

The home-assistant data provider is coming along well by the way

Awesome!

I think that is probably the last major field that is currently still missing.

The probability of precipitation is certainly an important forecast element that we should try to make available in Bright Sky. Unfortunately it is not provided in the MOSMIX_S data set that we currently parse, and switching to MOSMIX_L is a larger task.

The funding period for Bright Sky's development is coming to an end, and while I will certainly not stop supporting and developing it afterwards, I will naturally have less time. For now I would like to keep gauging interest to see which features should be pushed with highest priority. (In other words: I think this would be awesome but I don't plan to work on it within the next weeks :/)

I check the sources and it used the fallback Offenbach-Wetterpark instead of the local station in Darmstadt, which probably didn't provide the condition. Whether that is situational or and exception I'm not sure. Anyway, Offenbach is roughly 28km away, so I understand where the delayed rain condition is coming from.

As a mere "gateway" to the DWD data we adopt the limitations of that data. In this case, where you're looking to retrieve the weather right now, there is not only the distance between your location and the next fully-featured station at work, but also the delay of reported weather. Most SYNOP stations publish their observations (which form the base of the current_weather response) only every half hour. So even if there was a weather station in your backyard, any rainfall will at best be reported as "condition: rain" on the next full or half hour, i.e. with a delay of up to 30 minutes (and that condition will then stay unchanged for at least 30 minutes). A less-than-20-minutes-rainfall like you describe could even not be picked up at all (always reported as "condition: dry").

[elbotho]

For now I would like to keep gauging interest to see which features should be pushed with highest priority.

This is an important feature for me personally, but it's also the open issue with the most reactions by a lot. Sooo if you were searching for a new feature to add… :)

[jdemaeyer]

Precipitation probability for forecasts will land in the API very soon :)

MOSMIX provides four different time-ranges for precipitation probabilities:

• last hour
• last six hours
• last twelve hours (only available at 0:00, 6:00, 12:00, 18:00)
• last 24 hours (only available at 0:00, 12:00)

For each of these, the following eight precipitation probabilities for different volumes are available:

• Probability of precipitation
• Probability of precipitation > 0.1 mm
• Probability of precipitation > 0.2 mm
• Probability of precipitation > 0.3 mm
• Probability of precipitation > 0.5 mm
• Probability of precipitation > 0.7 mm
• Probability of precipitation > 1.0 mm
• Probability of precipitation > 5.0 mm

Giving us a grand total of 40 probabilities to choose from...

Here is an example of the different hourly probabilities:

{
  "precipitation_probability_1h": 24,
  "precipitation_probability_1h_01mm": 17,
  "precipitation_probability_1h_02mm": 10,
  "precipitation_probability_1h_03mm": 4,
  "precipitation_probability_1h_05mm": 4,
  "precipitation_probability_1h_10mm": 2,
}

Unordered thoughts:

• The different time-ranges cannot be deduced from one another since they are probabilities of non-independent events
• Six-hours and 24-hours seem to be the most sensible non-hourly values, often used in "day overview" or "week overview" layouts (together with 4-hours which is not available)
• I don't think we should use more than two - max three - values, for both storage and usability reasons

My instinct is to provide

• "Probability of precipitation > 0.1 mm during last hour" as precipitation_probability_1h, and
• "Probability of precipitation > 0.1 mm during last six hours" as precipitation_probability_6h.

Later we'll probably pick up the 24h-value as part of #117.

@mweinelt @elbotho @poetaster Thoughts?

[jdemaeyer]

"More than 0.1 mm" seems to be what OpenWeatherMap and open-meteo use. [1, 2]. DarkSky's (WeatherKit's) docs just say "the probability of precipitation during the hour". Wikipedia has some info on usage by various weather agencies: The US National Weather Service uses a ~0.25 mm threshold, Environment Canada uses a 0.2 mm threshold, the UK Met Office uses a 0.1 mm threshold.

EDIT:

Here's an example of the current forecast for Münster:

$ dwdparse MOSMIX_L_LATEST.kmz | jq .

# ...
{
  "timestamp": "2023-03-29 00:00:00+00:00",
  "precipitation_probability_wwP": 33,
  "precipitation_probability_wwP6": 55,
  "precipitation": 0,
  "precipitation_probability": 20,
  "precipitation_probability_6h": 29,
}
{
  "timestamp": "2023-03-29 01:00:00+00:00",
  "precipitation_probability_wwP": 26,
  "precipitation_probability_wwP6": 60,
  "precipitation": 0,
  "precipitation_probability": 11,
  "precipitation_probability_6h": null,
}
{
  "timestamp": "2023-03-29 02:00:00+00:00",
  "precipitation_probability_wwP": 27,
  "precipitation_probability_wwP6": 62,
  "precipitation": 0,
  "precipitation_probability": 9,
  "precipitation_probability_6h": null,
}
{
  "timestamp": "2023-03-29 03:00:00+00:00",
  "precipitation_probability_wwP": 20,
  "precipitation_probability_wwP6": 64,
  "precipitation": 0,
  "precipitation_probability": 5,
  "precipitation_probability_6h": null,
}
{
  "timestamp": "2023-03-29 04:00:00+00:00",
  "precipitation_probability_wwP": 18,
  "precipitation_probability_wwP6": 58,
  "precipitation": 0,
  "precipitation_probability": 7,
  "precipitation_probability_6h": null,
}
{
  "timestamp": "2023-03-29 05:00:00+00:00",
  "precipitation_probability_wwP": 12,
  "precipitation_probability_wwP6": 54,
  "precipitation": 0,
  "precipitation_probability": 4,
  "precipitation_probability_6h": null,
}
{
  "timestamp": "2023-03-29 06:00:00+00:00",
  "precipitation_probability_wwP": 12,
  "precipitation_probability_wwP6": 58,
  "precipitation": 0,
  "precipitation_probability": 6,
  "precipitation_probability_6h": 23,
}
# ...

where

• precipitation is the forecasted precipitation amount for the previous hour,
• precipitation_probability is the probability of more than 0.1 mm precipitation in the previous hour,
• precipitation_probability_6h is the probability of more than 0.2 mm precipitation in the previous six hours,
• precipitation_probability_wwP is the probability of any precipitation in the previous hour,
• precipitation_probability_wwP6 is the probability of any precipitation in the previous six hours,

Given that the forecasted total precipitation amount for the six-hour timespan is zero (or rather, below 0.1 mm), the 60 %-ish probabilities from wwP6 seem quite high to me. Then again, if it's drizzling the whole night, maybe that is the probability that feels more natural? Although the wwP values don't seem to suggest drizzling. Boy do I have a hard time wrapping my primate brain around these probabilities :sweat_smile:

[jdemaeyer]

Change of plans: Instead of discussing hypothetical scenarios and playing guessing games in this issue, I've just released these two fields as undocumented / experimental (i.e. subject to change without warning) in the public Bright Sky instance, so you guys can test them in your actual applications:

• precipitation_probability: probability of more than 0.1 mm precipitation in the previous hour
• precipitation_probability_6h: probability of more than 0.2 mm precipitation in the previous six hours (only available on 0:00, 6:00, 12:00, 18:00 UTC, this is a limitation we inherit from MOSMIX)

If the data looks reasonable from your side, I'll add documentation and mark them stable.

[poetaster]

Ah, this is cool. I think it'll be more accurate than my calculations. I'll see if I can test this the coming days. Thanks!

[jdemaeyer]

Closing as this is now officially part of the API :)