OWM precipitation from daily is not consistent with assumptions

[jeroenterheerdt]

What happened?

OWM precipitation from daily is not consistent with our assumptions / requirements for when people use a sensor. We expected it to be a daily forecast but it varies wildly during the day. Precipitation in our integration is defined as: daily["rain"] + daily["snow"]. I'd expect it to either stay 0 or increase when the day progresses, but it doesn't:

"data": [
          {
            "Windspeed": 0.0,
            "Pressure": 996.2924284049999,
            "Humidity": 100.0,
            "Temperature": 12.527777777777775,
            "Dewpoint": 10.82,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T00:34:39.708540"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 996.2924284049999,
            "Humidity": 100.0,
            "Temperature": 12.449999999999998,
            "Dewpoint": 11.35,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T01:34:40.198560"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 996.2924284049999,
            "Humidity": 100.0,
            "Temperature": 12.42222222222222,
            "Dewpoint": 11.22,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T02:34:40.211957"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 996.2924284049999,
            "Humidity": 100.0,
            "Temperature": 12.111111111111109,
            "Dewpoint": 10.65,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T03:34:40.203154"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 996.981904134,
            "Humidity": 100.0,
            "Temperature": 11.922222222222222,
            "Dewpoint": 10.37,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T04:34:40.604191"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 996.981904134,
            "Humidity": 100.0,
            "Temperature": 11.61111111111111,
            "Dewpoint": 9.76,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T05:34:40.331183"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 997.671379863,
            "Humidity": 100.0,
            "Temperature": 11.261111111111113,
            "Dewpoint": 9.51,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T06:34:40.212081"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 997.671379863,
            "Humidity": 100.0,
            "Temperature": 11.172222222222222,
            "Dewpoint": 9.55,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T07:34:40.229681"
          },
          {
            "Windspeed": 0.0,
            "Pressure": 999.73980705,
            "Humidity": 100.0,
            "Temperature": 12.177777777777779,
            "Dewpoint": 9.74,
            "Precipitation": 0.0,
            "retrieved": "2023-09-26T08:34:40.255309"
          },
          {
            "Windspeed": 0.6656764568994703,
            "Pressure": 1014.0000058602027,
            "Humidity": 90,
            "Temperature": 11.83,
            "Dewpoint": 10.24,
            "Precipitation": 6.44,
            "retrieved": "2023-09-26T09:10:40.109577"
          },
          {
            "Windspeed": 0.6656764568994703,
            "Pressure": 1014.0000058602027,
            "Humidity": 88,
            "Temperature": 13.56,
            "Dewpoint": 11.61,
            "Precipitation": 6.33,
            "retrieved": "2023-09-26T10:10:40.002459"
          },
          {
            "Windspeed": 1.002254440725045,
            "Pressure": 1014.0000058602027,
            "Humidity": 82,
            "Temperature": 14.79,
            "Dewpoint": 11.75,
            "Precipitation": 8.1,
            "retrieved": "2023-09-26T11:10:39.907213"
          },
          {
            "Windspeed": 1.002254440725045,
            "Pressure": 1015.000005865982,
            "Humidity": 82,
            "Temperature": 14.91,
            "Dewpoint": 11.87,
            "Precipitation": 8.33,
            "retrieved": "2023-09-26T12:10:40.013252"
          },
          {
            "Windspeed": 1.675410408376195,
            "Pressure": 1015.000005865982,
            "Humidity": 77,
            "Temperature": 14.83,
            "Dewpoint": 10.84,
            "Precipitation": 9.94,
            "retrieved": "2023-09-26T13:10:39.946402"
          },
          {
            "Windspeed": 1.675410408376195,
            "Pressure": 1016.0000058717612,
            "Humidity": 85,
            "Temperature": 12.11,
            "Dewpoint": 9.67,
            "Precipitation": 12.62,
            "retrieved": "2023-09-26T14:10:39.921761"
          },
          {
            "Windspeed": 1.2939553600405433,
            "Pressure": 1015.000005865982,
            "Humidity": 81,
            "Temperature": 12.62,
            "Dewpoint": 9.45,
            "Precipitation": 7.54,
            "retrieved": "2023-09-26T15:10:39.889807"
          },
          {
            "Windspeed": 1.002254440725045,
            "Pressure": 1016.0000058717612,
            "Humidity": 87,
            "Temperature": 12.81,
            "Dewpoint": 10.7,
            "Precipitation": 6.46,
            "retrieved": "2023-09-26T16:10:39.834069"
          }

The OWM API does not seem to provide anything else, although there is a hourly precipitation (hourly.rain.1h and hourly.snow.1h in the API. That could work but only if the user would update every hour, which we don't really control. Default aggregation for precipitation in our integration is "last", which would work if the assumption was true, which it clearly isn't. Average is maybe a better alternative but if you look at the above I am concerned the number of 0s drag it down too much.

Thoughts @gatonero?

[gatonero]

There are accumulated parameters in OWM https://openweathermap.org/api/accumulated-parameters#precip. Does this help?

Just now I saw that this is a feature of professional collections.

Will look for an other solution.

[gatonero]

A call like this is possible https://api.openweathermap.org/data/3.0/onecall/day_summary?lat=50.75&lon=6.08&date=2023-09-2&appid={appid} and gives daily aggregations. See daily aggregations under https://openweathermap.org/api/one-call-3#history

[jeroenterheerdt]

A call like this is possible https://api.openweathermap.org/data/3.0/onecall/day_summary?lat=50.75&lon=6.08&date=2023-09-2&appid={appid} and gives daily aggregations. See daily aggregations under https://openweathermap.org/api/one-call-3#history

I believe that is only after the day finished, while we are calling OWM during the day

[jeroenterheerdt]

There are accumulated parameters in OWM https://openweathermap.org/api/accumulated-parameters#precip. Does this help?

Just now I saw that this is a feature of professional collections.

Will look for an other solution.

Yes, I noticed that as well.

[jeroenterheerdt]

A call like this is possible https://api.openweathermap.org/data/3.0/onecall/day_summary?lat=50.75&lon=6.08&date=2023-09-2&appid={appid} and gives daily aggregations. See daily aggregations under https://openweathermap.org/api/one-call-3#history

I believe that is only after the day finished, while we are calling OWM during the day

ok, I have just tried this and this looks promising:

{ "lat": XXX, "lon": YYY, "tz": "-07:00", "date": "2023-09-27", "units": "metric", "cloud_cover": { "afternoon": 100.0 }, "humidity": { "afternoon": 95.74 }, "precipitation": { "total": 75.47 }, "temperature": { "min": 10.97, "max": 13.52, "afternoon": 11.48, "night": 12.02, "evening": 12.09, "morning": 11.28 }, "pressure": { "afternoon": 1012.7 }, "wind": { "max": { "speed": 3.57, "direction": 89.43 } } } It would require people to have the OneCall 3.0 API subscription (and would force everyone using 2.5 to switch to that), but that's something we can do. (And yes, it's a very rainy day where I live today :)) I'll monitor this and see how it progresses through the day and compare it to my weather station.

[jeroenterheerdt]

well, that sucks. The precipitation did not meet what actually happened. In fact, it went down from 75.47 in the morning to 29.24. The actual number is over 153 (all in mm). So yeah, this is not going to work either. I will check the diffference between my pure OWM config and sensor config to see if our current approach works better than this. Otherwise it's back to the drawing board...

[gatonero]

There's also a field sensor.openweathermap_forecast_precipitation_probability which varies a lot during the day. Maybe we should take it into account.

https://en.wikipedia.org/wiki/Probability_of_precipitation

https://www.weather.gov/media/pah/WeatherEducation/pop.pdf

[jeroenterheerdt]

But that's just the chance of rain, not the actual rain amount

[gatonero]

Yes but it comes together with the amount. So it could be a hint, when to calculate with the amount. But all together it seems to be very vague.

Maybe the precipitation should only taken into account, when the probability is above 80%. Here are some graphs which show how these values might correlate. All from 2023-09-22 / 00:00 to 2023-09-23 / 23:55

1. Forecast Precipitation probability

2. Forecast Precipitation

3. Real measurement

[gatonero]

Another example over a week:

[gatonero]

Here another correlation between rein rate and forecast condition.

[jeroenterheerdt]

interesting.. but would that mean that we take a simple sum of the rain amount? Or are you saying you would only take the rain amount if probability > some threshold (80%?) and then apply an aggregate on it (max/sum/last ??)

[gatonero]

This doesn't seem particularly reliable to me. We should only consider OWM forecasts when the chance of rain is over at least 80% and/or the forecast condition = rainy. It seems to me that the maximum value is always lower than the actual precipitation. So maybe the sum is more reliable. But this has to be observed.

[jeroenterheerdt]

OK, I can try and change the logic just for OWM to see what effects it has.

[jeroenterheerdt]

hey @gatonero are you still monitoring this? does it still look like only taking the daily rain forecast into account when a certain precip chance % has been passed is the right thing to do?

[gatonero]

I'm still observing it but we didn't have rain for three weeks now. I will notify about my surveillance.

[gatonero]

@jeroenterheerdt

New insights. Calculating with OWM forecast is like playing roulette!

1. The day before yesterday at calculating time forecast probability was about 40% but it rained yesterday -> irrigation started (not ok)
2. Yesterday at calculation time forecast probability was about 60% it rained (a lot) today -> no irrigation, which is ok but I guess it's caused by a bucket > 0

Conclusion: OWM isn`t reliable. The only workaround could be, to have a look at current rain rate and/or forecast again at irrigation time???

[jeroenterheerdt]

Crap. I guess this goes to show that you should really only use your own sensors and not rely on OWM. We should update our readme for this.

[gatonero]

No, OWM is not too bad. Just the forecast values are more or less screwed up for our due. This is the nature of weather forecasts especially if they are for small local regions. This is what OWM explains for predictions OWM, accuracy and quality

However, we should keep in mind that the actual difference between the nowcast and the real situation at a specific place and time could be bigger than these average errors.

The actual weather data are quite good as shown in the following graphs. The only big difference is in wind speed for which I will have to proof my station and settings.

[github-actions[bot]]

Stale issue message