REQ: Use actual weather values instead of OWM

posixx commented 4 years ago

Jeroen,

I see you use the following info from OWM:

    rain = data["rain"]
    snow = data["snow"]
    t_day = d["temp"]["day"]
    t_min = d["temp"]["min"]
    t_max = d["temp"]["max"]
    t_dew = float(d["dew_point"])
    pressure = d["pressure"]
    RH_hr = d["humidity"]
    u_2 = d["wind_speed"]

For all of us having our own weatherstation we would like to make use of actual weatherdata instead of forecasted data as actual data will be much more precise, especially outside of the US.

So my request is during integration setup; ask weather to use OWM or hass sensors. When selecting OWM use the functionality as is now, when using hass sensors display 8 textboxes to insert sensor names (weatherstation doesn't know difference between rain and snow so 1 sensor for that):

daily rain current temperature minimum temperature maximum temperature dew point air pressure humidity average windspeed

I hope you can see the value here, also no need to setup OMW accounts and, when using multiple instances, no redundant api calls..

posixx commented 4 years ago

To see the differences here is a copy of OWM current weather and actual weatherdata from my davis weatherstation:

jeroenterheerdt commented 4 years ago

Good idea, I will see what I can do. Cannot test it though since I don't have a weather station myself.

posixx commented 4 years ago

I'm happy to test this for you, as much as you want:-) It will also be with 2 integration instances so we can test both; 1 and multiple instances.

jeroenterheerdt commented 4 years ago

By the way, which weatherstation is this?

posixx commented 4 years ago

Davis Vantage Pro2. I use a WifiLogger card in the console which pushed my info to WOW and publishes all readings as json to my home assistant MQTT instance. Within home assistant i made a MQTT discovery plugin which adds all avalable sensors automatically

jeroenterheerdt commented 4 years ago

OK, got it. I think we will need to allow people to choose which value to get from OWM and which from a weatherstation. I am thinking of building my own station, but that will probably not be able to provide all the values right away.

posixx commented 4 years ago

Ah great idea. As long as it is possible to completely disable OWM api calls as we don't want the extra overhead and creation of api key. So maybe start with aking which sensors to use or OWM; when using sensors for al valuesl skip the OWM setip..

jeroenterheerdt commented 4 years ago

Yes, exactly that was what I was thinking.

Sent from Outlook Mobilehttps://aka.ms/blhgte

From: posixx notifications@github.com Sent: Wednesday, May 27, 2020 8:27:05 AM To: jeroenterheerdt/HAsmartirrigation HAsmartirrigation@noreply.github.com Cc: Jeroen ter Heerdt jeroen_ter_heerdt@hotmail.com; Comment comment@noreply.github.com Subject: Re: [jeroenterheerdt/HAsmartirrigation] REQ: Use actual weather values instead of OWM (#12)

Ah great idea. As long as it is possible to completely disable OWM api calls as we don't want the extra overhead and creation of api key. So maybe start with aking which sensors to use or OWM; when using sensors for al valuesl skip the OWM setip..

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjeroenterheerdt%2FHAsmartirrigation%2Fissues%2F12%23issuecomment-634739575&data=02%7C01%7C%7C77740e72f3a7467c0e6908d8025269c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637261900265318641&sdata=PnZcOuUeDpP41beqyNF2WytPpQmaY2yIpIghd7Z696w%3D&reserved=0, or unsubscribehttps://nam11.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAB6PIPSMMOK6ID6DRRSU7NLRTUWMTANCNFSM4NL6576A&data=02%7C01%7C%7C77740e72f3a7467c0e6908d8025269c9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637261900265328634&sdata=N%2Bqrrgk91D%2BZa7CX2yev9MYXJjPMU0aQ%2BJE6k958BpQ%3D&reserved=0.

ciechompl commented 4 years ago

+1 :) I use Netatmo weather station, so also knows precisely what is the temperature, wind, rain etc in MY location ... so would like to finally make a use of it in a way providing some real value ... not just nice information shown on the tablet in the kitchen :)

jeroenterheerdt commented 4 years ago

Sooo.. Since all these weather values eventually roll up into hourly_adjusted_run_time what is the correct behavior for basing daily_adjusted_run_time on it? Take the minimum of hourly art? Maximum? Average? Last value? Currently I take the last as I assume the most recent prediction for today is best since most of the day has progressed. With a weather station we are not using predictions any longer but actuals..so average or min or max would make more sense.. Thoughts?

posixx commented 4 years ago

I would opt for average: take actual calculations from every hour, and devide through the number of hours. This will take into account sudden weather changes based on the actual readings.

jeroenterheerdt commented 4 years ago

daily art is now the average of the hourly values collected, regardless if you call it yourself or if it's auto-calculated.

jeroenterheerdt commented 4 years ago

first version of this is ready for testing! @posixx @ciechompl

posixx commented 4 years ago

Great. Before i do: can you give us the unit_of_measurements the integration expects? Or label them on the setup step so we now what to expect. For instance, my windspeed sensor uses km/h but i think the integrations expects m/s..

jeroenterheerdt commented 4 years ago

Right, there's that and there is the imperial / metric system to deal with. I'll specify it somehow and document it as well. Stay tuned.

jeroenterheerdt commented 4 years ago

@posixx @ciechompl : v0.0.25 is available which should allow you to test. Units are indicated and documented in the readme as well. Looking forward to your feedback, since I have not been able to test this as I don't have a weather station. I have been using random value sensors...

posixx commented 4 years ago

looks good so far! added integration for both my zones, no errors / warnings in the logs, and hourly sensors display results.. keep you posted!

glowery commented 4 years ago

I've used the updated component as well with an Ambient Weather station integration and so far sensors are displaying and looking good. Thanks for your work on this! Great feature addition!

jeroenterheerdt commented 4 years ago

thanks @posixx and @glowery for letting me know. Please keep an eye on the values, especially the interaction with daily adjusted run time and the bucket. I suspect something is not right, but I cannot put my finger on it yet. More data helps.

glowery commented 4 years ago

Sure thing, will do. Thanks for heads up!

posixx commented 4 years ago

First impressions; run time looks lower then yesterday, when today was hotter then yesterday:

Let's see in about an hour what daily sensor looks like..

jeroenterheerdt commented 4 years ago

let's keep in mind there was a bug where the code passed in hPa where the calculations required kPa, so that was fixed. These calculations should be better. The thing I worry about is how hourly is brought into daily :) basically what I would need help with is: please monitor the netto precipitation (also called bucket_delta) and see if your bucket (attribute of daily_adjusted_run_time is updated correctly with this delta value when the daily value is calculated. Water budgets and adjusted run times are just based on the bucket value, so if that is not updated correctly we have a problem ;).

Of course is the bucket_delta is negative (like in the screenshot above) the bucket should decrease as well.

posixx commented 4 years ago

i'm using latest build so my pressure sensor in mbar should be correct?

jeroenterheerdt commented 4 years ago

@posixx yes mbar=hPa. The code divides it by ten to get kPa when it's needed,because both OWM and most sensors provide hpa / mbar.

posixx commented 4 years ago

Ok 23h past, here are my readings:

Compared to yesterday, when using old component with OWM sensors:

Yesterday: 1052 seconds Today: 1157 seconds.

As i told before today the temperatures were higher this looks promising:-)

posixx commented 4 years ago

And the one from my backyard:

posixx commented 4 years ago

THis is found in the core log of hassio:

2020-06-01 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating for last time today, calculating adjusted run time for next irrigation time! 2020-06-01 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating bucket: 0.0 with netto_precipitation: -5.817398976711855 2020-06-01 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Bucket for today is: -5.817398976711855 mm 2020-06-01 23:00:00 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last): File "/config/custom_components/smart_irrigation/init.py", line 404, in _async_update_last_of_day return data UnboundLocalError: local variable 'data' referenced before assignment 2020-06-01 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating for last time today, calculating adjusted run time for next irrigation time! 2020-06-01 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating bucket: 0.0 with netto_precipitation: -5.817412771934118 2020-06-01 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Bucket for today is: -5.817412771934118 mm 2020-06-01 23:00:00 ERROR (MainThread) [homeassistant.core] Error doing job: Task exception was never retrieved Traceback (most recent call last): File "/config/custom_components/smart_irrigation/init.py", line 404, in _async_update_last_of_day return data UnboundLocalError: local variable 'data' referenced before assignment

So it's not enirely without problems..

jeroenterheerdt commented 4 years ago

ok - this is not good. Bucket looks like it is behaving. Can you confirm you are only using sensors at this point, no OWM?

posixx commented 4 years ago

Yes, never inserted the OWM API key during setup. Alle sensors are from my weatherstation

jeroenterheerdt commented 4 years ago

found the bug and fixed it.

posixx commented 4 years ago

great will update in the morning and let you know in about 24h

jeroenterheerdt commented 4 years ago

a new version is here: https://github.com/jeroenterheerdt/HAsmartirrigation/releases/tag/v0.0.28, there was a bug where I added lead_time twice, both to hourly and daily calculations. oops.

posixx commented 4 years ago

ok installed. Does a HA restart affect the calculation of the daily run time? As the hourly sensor is back to zero when restarting HA..

jeroenterheerdt commented 4 years ago

daily run time and bucket should survive a HA reboot. Hourly is set to 0. To be on the safe side: make a note of the values, do the reboot and if they are not coming back edit the state manually.

posixx commented 4 years ago

ok. daily sensors are 0 most of the time so didn't notice that. Will keep an eye on it when these have values and i need to restart.

posixx commented 4 years ago

On the core log i see:

2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 2.0191281227459372 and adjusted_run_time: 1385 for type: Hourly Adjusted Run Time. Bucket value was: -6.9659920234734845, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] rain: 0.0, snow: 0.0 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 2.0191281227459372 and adjusted_run_time: 1385 for type: Hourly Adjusted Run Time. Bucket value was: -6.9659920234734845, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 0 and adjusted_run_time: 0 for type: Daily Adjusted Run Time. Bucket value was: 0.0, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 0 and adjusted_run_time: 0 for type: Daily Adjusted Run Time. Bucket value was: 0.0, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation] Updating Smart Irrigation Data 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] rain: 0.0, snow: 0.0 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 2.0191281227459372 and adjusted_run_time: 4628 for type: Hourly Adjusted Run Time. Bucket value was: -6.9659920234734845, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] rain: 0.0, snow: 0.0 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 2.0191281227459372 and adjusted_run_time: 4628 for type: Hourly Adjusted Run Time. Bucket value was: -6.9659920234734845, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 0 and adjusted_run_time: 0 for type: Daily Adjusted Run Time. Bucket value was: 0.0, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1 2020-06-02 17:44:45 INFO (MainThread) [custom_components.smart_irrigation.sensor] Calculated water_budget = 0 and adjusted_run_time: 0 for type: Daily Adjusted Run Time. Bucket value was: 0.0, and force mode is: False, force mode duration is: 0, lead_time is: 0, maximum_duration: -1

It looks like the calculations of daily adjusted runtime are ran every routine?

jeroenterheerdt commented 4 years ago

yeah, daily should not be called unless it is time for it or if you call the service.

posixx commented 4 years ago

Ok that looks like a minor issue then as these are called every 5 minutes, when the hourly sensor are calculated.

jeroenterheerdt commented 4 years ago

yeah, I see that. hourly was not even supposed to be updated every 5 minutes, but that is a Development configuration that made it into the release. Figuring out now why daily is updating this frequently as well.

ciechompl commented 4 years ago

one question :) ... let's say yesterday at 23:00 daily sensor got value 19 min, but today at 3:00 irrigation hasn't happened as hourly got value 0 in a meantime (we had a rain between 23:00 and 3:00 :)) => reset_bucked not called ... so what will be next daily value today at 23:00 ? average from last 24 h ignoring previous daily ? or average from 48 h as reset bucket haven't been called ?

jeroenterheerdt commented 4 years ago

@ciechompl daily is an average of hourly values. Right now daily is updated way to often. That is a bug. It is supposed to be updated once per day due to auto refresh or calling the service. Then daily takes the accumulated hourly values since the last call, and takes the average to update itself. Afterwards the accumulated hourly values are emptied. This happens regardless of irrigation automation triggering or not since that should be based based on the daily value itself, not on hourly. So unless you use hourly values (which you should not) to make the decision to irrigation this should not happen once the bug is fixed.

ciechompl commented 4 years ago

ok understood ... I use hourly only as a condition for automation: daily>0 AND hourly>0 => start irrigation. as per your suggestion to eliminate over irrigation in case if next day/hours will be rainy :)

jeroenterheerdt commented 4 years ago

turns out I was wrong - updating daily is fine every hour. The bucket is the one that needs to be updated just daily and that is happening correctly (once a day for auto refresh or when called manually). Fixed a bug with the updates happening every five minutes though (that is what I was using during development...). New release is 0.0.29

@ciechompl if hourly dropped below zero just at the moment you run your automation it will not run while daily indicates that it needs to run. regardless, as long as you don't call reset_bucket the values will still persist and your daily value will keep increasing, which is what you want.

posixx commented 4 years ago

Ok past 23h now. COre log:

2020-06-02 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating for last time today, calculating adjusted run time for next irrigation time! 2020-06-02 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating bucket: 0.0 with netto_precipitation: -3.922710742963342, which should be average of: [0.0, 0.0, -5.787587216971071, -5.787587216971071, -5.2869868742166215, -5.2869868742166215, -4.616268880665675, -4.616268880665675] 2020-06-02 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Bucket for today is: -3.922710742963342 mm 2020-06-02 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating for last time today, calculating adjusted run time for next irrigation time! 2020-06-02 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Updating bucket: 0.0 with netto_precipitation: -3.922710742963342, which should be average of: [0.0, 0.0, -5.787587216971071, -5.787587216971071, -5.2869868742166215, -5.2869868742166215, -4.616268880665675, -4.616268880665675] 2020-06-02 23:00:00 INFO (MainThread) [custom_components.smart_irrigation] Bucket for today is: -3.922710742963342 mm

Installed latest version ar about 19:30 as you can see here:

So hourly sensors seem forget past hours after restart. Don't know if this can be persisted to prevent lower runtimes as a result of restarting HA?

posixx commented 4 years ago

Also my Davis sensors are null / empty right after restart so it would be nice to have the integration starting hourly calculations after 5 or 10 minutes

jeroenterheerdt commented 4 years ago

@posixx restoring hourly is not easy. But I should be able to make the hourly update quicker after a reboot. I will try to get a option setting for it and default it to 10 minutes.

posixx commented 4 years ago

Great, that will help a lot

jeroenterheerdt commented 4 years ago

https://github.com/jeroenterheerdt/HAsmartirrigation/releases/tag/v0.0.30 implements this initial delay setting. Also, @ciechompl I have added a % increase setting so you do not have to do that on your own any more.

posixx commented 4 years ago

Delay works, after 300 seconds the daily sensor was updated. A small cosmetic bug:

initial_update_delay displays milliseconds instead of seconds.

ciechompl commented 4 years ago

Also, @ciechompl I have added a % increase setting so you do not have to do that on your own any more.

cool :) thank you :) just only to be 100% sure :) .. I looked into the code:

 CONF_INCREASE_PERCENT * 100, DEFAULT_INCREASE_PERCENT

are you sure you should multiply by 100 not divide by 100 ? :) I might be wrong byt by my simple thinking :) when I put e.g. 50 in the config (my goal would be to get result only 50% of the calculated time) it will be 5000 as a factor for further adjustments. In general, my idea was to allow to tune up and tune down in meaning that users can decide to irrigate only for 50% of calculated time. I might be wrong :) but looking at this part I think you allow only to tune up (1.0 + ) in meaning that whatever we put into config it will be added to the time .. .e,g, 50% -> run time*1.5:

adjusted_run_time = adjusted_run_time * float(1.0 +
   self.coordinator.increase_percent

so if you would put into the config as a default 100% (not 0) and later you would just multiply adjusted run time by this % , users would be able to do whatever they want :) -> from 0% to infinity % :) adjust up and down ... full flexibility :)

jeroenterheerdt / HAsmartirrigation

REQ: Use actual weather values instead of OWM #12