jeroenterheerdt / HAsmartirrigation

Smart Irrigation custom component for Home Assistant
MIT License
308 stars 48 forks source link

Large variance between OWM and local sensors #174

Closed dlindnegm closed 9 months ago

dlindnegm commented 9 months ago

What happened?

I've set up 2 identical zones with the only difference that one uses OWM as the source and the other uses my local weather station for everything except evapotranspiration (no sensor). The OWM calculation produces a run time of 60050 and using my own sensors produces a run time of 6730. That's a difference of 53320 seconds, or almost 15 hours!

How to reproduce

Create 2 zones, one using OWM for all sensors and one using local weather station sensors. Update and calculate each zone.

Relevant log output

Using OWM for all sensors:
Module returned Evapotranspiration deficiency of -11.6. Bucket was 0.0.
New bucket value is [old_bucket]+[delta]=0.0+-11.6=-11.6.
Since bucket < 0, irrigation is necessary.
To calculate the exact duration, the following steps were taken:
The precipitation rate is defined as[throughput]*60/[size]=1.669*60/144.0=0.7
The duration is calculated as abs([bucket])/[precipitation_rate]*3600=11.6/0.7*3600=60050
Now, the multiplier is applied. The multiplier is 1.0, hence the duration is 60050
Finally, the lead time is applied. The lead time is 0.0, hence the final duration is 60050

Using my local sensors:
Module returned Evapotranspiration deficiency of -1.3. Bucket was 0.0.
New bucket value is [old_bucket]+[delta]=0.0+-1.3=-1.3.
Since bucket < 0, irrigation is necessary.
To calculate the exact duration, the following steps were taken:
The precipitation rate is defined as[throughput]*60/[size]=1.669*60/144.0=0.7
The duration is calculated as abs([bucket])/[precipitation_rate]*3600=1.3/0.7*3600=6730
Now, the multiplier is applied. The multiplier is 1.0, hence the duration is 6730
Finally, the lead time is applied. The lead time is 0.0, hence the final duration is 6730

Which version are you running?

v2023.8.0-beta19

Diagnostics file

Additional information

No response

jeroenterheerdt commented 9 months ago

Closing this as there is no diagnostics file so no way for me to troubleshoot.

dlindnegm commented 9 months ago

Not sure what sort of log you need, but here is the one from enabling debug logging from the integration. I didn't see much in there other than several "no data available" entries as I was repeatedly updating.

home-assistant_smart_irrigation_2023-08-21T22-45-53.952Z.log

jeroenterheerdt commented 9 months ago

from the bug template you filled out: " Diagnostics file You are required to attach a diagnostics file. To download a diagnostics file, in Home Assistant, go to Settings>Devices&Services>Integrations>Smart Irrigation, or use this link. Click the 'three vertical dots' menu and select 'download diagnostics'."

You checked the box, but didn't read the instruction.

dlindnegm commented 9 months ago

Duh! Diagnostics file attached. config_entry-smart_irrigation-70e7f1274d03ec5638c8056566ed75ee.json (1).txt

jeroenterheerdt commented 9 months ago

thank you, reopening issue

jeroenterheerdt commented 9 months ago

interesting. this is a big difference indeed. Not sure what is causing this. Can you send a diagnostics file after you updated the first mapping and before you calculate? I can use those values to run the logic and see if that output is correct.

dlindnegm commented 9 months ago

First file is after bucket was reset to zero. Second file is after the calculation.

config_entry-smart_irrigation-70e7f1274d03ec5638c8056566ed75ee.json (5).txt

config_entry-smart_irrigation-70e7f1274d03ec5638c8056566ed75ee.json (6).txt

dlindnegm commented 9 months ago

Not sure if it's related, but I don't understand the results for a direct API 3.0 call to OWM for my GPS coordinates:

"timezone":"America/New_York","timezone_offset":-14400,"current": {"dt":1692667946,"sunrise":1692615613,"sunset":1692663641,"temp":299.19,"feels_like":299.19,"pressure":1022,"humidity":84,"dew_point":296.27,"uvi":0,"clouds":0,"visibility":10000,"wind_speed":0,"wind_deg":0,"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]}}

Humidity, windspeed, and uvi all look normal, but what measurement system are they using for temp, feels like, pressure and dew point? My actual local readings are: temp: 77.9 feels like: 80.3 pressure: 29.005 inHg dew point: 75.8

Update: Maybe not an issue - once I added the units=imperial option: "timezone":"America/New_York","timezone_offset":-14400,"current":{"dt":1692668635,"sunrise":1692615613,"sunset":1692663641,"temp":78.62,"feels_like":80.19,"pressure":1022,"humidity":85,"dew_point":73.72,"uvi":0,"clouds":0,"visibility":10000,"wind_speed":0,"wind_deg":0,"weather":[{"id":800,"main":"Clear","description":"clear sky","icon":"01n"}]}}

jeroenterheerdt commented 9 months ago

Holy cow! They are using SI, so temperature in Celsius and pressure in mbar. Now pressure is fine in the result above, but the temperature... That's like we are on a different planet. What's the URL you called (removing your api key from the call of course)? The code specifies metric as the unit, see https://github.com/jeroenterheerdt/HAsmartirrigation/blob/2cf03a0dd5d46d2b966ace7cea6dfdabf959e45f/custom_components/smart_irrigation/OWMClient.py#L46. I am wondering what the results are if you do it like that, are those values very different compared to your sensors (after conversion of course)?

dlindnegm commented 9 months ago

https://api.openweathermap.org/data/3.0/onecall?lat=36.0067&lon=-84.2688&exclude=hourly,daily,minutely,alerts&units=imperial&appid=

Not sure if you noticed my updated edit - that I needed to add the units=imperial to get fahrenheit.

jeroenterheerdt commented 9 months ago

Yeah, I saw it. Still not sure what is happening here though. Internally the component uses metric values, so I am wondering there is a conversion error with the imperial values from your sensors instead. The OWM call asks for metric values as well. But that shouldn't explain why the OWM is so fsr off unless somewhere a double conversion is happening... Weird.

jeroenterheerdt commented 9 months ago

so I think a part of what is happening here is in the forecast days logic. When you call the update with the OWM mapping it gets called twice and the delta is negative for both (around -7 for both days). That explains why the bucket is set to -14, which I don't think is right, as it's the sum. I think it makes more sense to take an average of the delta instead, agreed?

Now, the other bug is that there is no actual forecast taking ino account when you did the calculate on the module that had the sensors. I actually thought about this last night so I added another bug (#178). I think the right behavior here is: use the sensor values to calculate for today and for any forecast days use OWM. Agreed?

dlindnegm commented 9 months ago

The average of the delta makes more sense. Is the reason why the OWM update get's called twice because it's called once for the forecast and once for the other sensor data as a group?

I agree with you on issue #178. Sounds like an API key will be required for anyone that only uses their local sensors but wants to use the forecast?

jeroenterheerdt commented 9 months ago

yeah, the forecast needs to come from somewhere and it should be a weather forecasting service like OWM as I don't think it makes a lot of sensor to just "copy" todays temperature/humidity/pressure/etc to tomorrow. Maybe in the future we can add support for other forecasting services? Right now OWM is getting called twice, I haven't done the work to "cache" the results".

jeroenterheerdt commented 9 months ago

also, I notice you have a "gap" in your zone IDs. How did you get in this state? I think because your second zone is ID 5, the panel doesn't allow you to change anything about that zone in the UI. This is not supposed to happen. I'll add another bug (#179) to remind myself to investigate this, but it would help if you could let me know what you remember about how you got into this state. Please try beta20 to see if this is better.

dlindnegm commented 9 months ago

I was repeatedly adding/deleting and renaming zones trying to identify why I had such odd results with the OWM run times. In a few cases I would rename the default zones, in others I would delete them and create new ones. I think the issue is related to those default zones somehow. Maybe just drop the 2 defaults altogether and make folks create their own?

jeroenterheerdt commented 9 months ago

i'll look into it in bug #179

jeroenterheerdt commented 9 months ago

I was repeatedly adding/deleting and renaming zones trying to identify why I had such odd results with the OWM run times. In a few cases I would rename the default zones, in others I would delete them and create new ones. I think the issue is related to those default zones somehow. Maybe just drop the 2 defaults altogether and make folks create their own?

found the bug with this, fixed it in beta 21. Please check if the beta 20/21 is behaving better in your scenario.

dlindnegm commented 9 months ago

This error originated from a custom integration.

Logger: aiohttp.server Source: custom_components/smart_irrigation/init.py:382 Integration: Smart Irrigation (documentation, issues) First occurred: 9:17:31 PM (1 occurrences) Last logged: 9:17:31 PM

Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle resp = await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle result = await handler(request, *request.match_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper result = await method(view, request, data, args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/smart_irrigation/websockets.py", line 174, in post await coordinator.async_update_zone_config(zone, data) File "/config/custom_components/smart_irrigation/init.py", line 680, in async_update_zone_config await self._async_calculate_all() File "/config/custom_components/smart_irrigation/init.py", line 382, in _async_calculate_all if modinst and modinst.name=="PyETO" and modinst.forecast>0: ^^^^^^^^^^^^^^^^ AttributeError: 'PyETO' object has no attribute 'forecast'

config_entry-smart_irrigation-4c013ffa62e4a130fd84688a1714162a.json.txt

dlindnegm commented 9 months ago

Still seeing big differences between using OWM sensors vs my own in beta 21, but not as dramatic as before!

With a zero day forecast: My sensors - bucket = -4.6 --- seconds = 23813 OWM sensors - bucket = -5.9 seconds = 30543 30543 - 23813 = 6730 sec = 112 minutes x 1.669 gpm = 187 gallons difference

With a 1 day forecast: My sensors - bucket = -5.1 --- seconds = 26401 OWM sensors - bucket = -5.8 seconds = 30025 30025 - 26401 = 3624 sec = 60 minutes x 1.669 gpm = 100 gallons difference

dlindnegm commented 9 months ago

Maybe I'm too hard on it, but with beta 21 after a while the ability to calculate even one zone starts failing with this error. Diag file attached. I'll try restarting the integration and/or HA to see what it takes to get it working again.

Logger: custom_components.smart_irrigation Source: custom_components/smart_irrigation/init.py:675 Integration: Smart Irrigation (documentation, issues) First occurred: 9:56:17 PM (3 occurrences) Last logged: 9:57:23 PM

Calculate zone Deck Plants failed: no data available. Calculate zone Lower Veggie Garden failed: no data available.

config_entry-smart_irrigation-4c013ffa62e4a130fd84688a1714162a.json (1).txt

jeroenterheerdt commented 9 months ago
  • Not sure I can re-create the zone gap issue but I'll watch for it.
  • With beta 21 I'm seeing an issue where I can update and calculate 6 sensors individually, but when I use the "Calculate all zones" option nothing is updated and I get the following error in my HA logs. I've attached the diag log file also.

This error originated from a custom integration.

Logger: aiohttp.server Source: custom_components/smart_irrigation/init.py:382 Integration: Smart Irrigation (documentation, issues) First occurred: 9:17:31 PM (1 occurrences) Last logged: 9:17:31 PM

Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle resp = await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle result = await handler(request, *request.match_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper result = await method(view, request, data, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/smart_irrigation/websockets.py", line 174, in post await coordinator.async_update_zone_config(zone, data) File "/config/custom_components/smart_irrigation/init.py", line 680, in async_update_zone_config await self._async_calculate_all() File "/config/custom_components/smart_irrigation/init**.py", line 382, in _async_calculate_all if modinst and modinst.name=="PyETO" and modinst.forecast>0: ^^^^^^^^^^^^^^^^ AttributeError: 'PyETO' object has no attribute 'forecast'

config_entry-smart_irrigation-4c013ffa62e4a130fd84688a1714162a.json.txt

What you mention here is a total bug, one that I can easily fix.

jeroenterheerdt commented 9 months ago

Still seeing big differences between using OWM sensors vs my own in beta 21, but not as dramatic as before!

With a zero day forecast: My sensors - bucket = -4.6 --- seconds = 23813 OWM sensors - bucket = -5.9 seconds = 30543 30543 - 23813 = 6730 sec = 112 minutes x 1.669 gpm = 187 gallons difference

With a 1 day forecast: My sensors - bucket = -5.1 --- seconds = 26401 OWM sensors - bucket = -5.8 seconds = 30025 30025 - 26401 = 3624 sec = 60 minutes x 1.669 gpm = 100 gallons difference

Still a lot of difference but to be expected. OWM is not very accurate.

jeroenterheerdt commented 9 months ago

Maybe I'm too hard on it, but with beta 21 after a while the ability to calculate even one zone starts failing with this error. Diag file attached. I'll try restarting the integration and/or HA to see what it takes to get it working again.

Logger: custom_components.smart_irrigation Source: custom_components/smart_irrigation/init.py:675 Integration: Smart Irrigation (documentation, issues) First occurred: 9:56:17 PM (3 occurrences) Last logged: 9:57:23 PM

Calculate zone Deck Plants failed: no data available. Calculate zone Lower Veggie Garden failed: no data available.

config_entry-smart_irrigation-4c013ffa62e4a130fd84688a1714162a.json (1).txt

Isn't this expected? You just calculated so all collected weather data is wiped. You need to update before you can calculate again otherwise there is nothing to calculate with (ie no sensor data has been collected).

dlindnegm commented 9 months ago

The problem seems to start after I click the Calculate all zones button. After that, single zone updates start to fail, and at some point none will calculate. I've tried logging out of HA, closing and re-launching the browser, reloading the integration. After I restarted HA I was able to calculate one zone, but it seized again when I try a second, third, etc. zone and keeps generating the same 'no data available' errors. I think this was the same situation I had before and the only fix was to uninstall and revert to a prior build, followed by an update to the latest.

dlindnegm commented 9 months ago

I've been updating before every calculate, whether singly or with the calculate all button.

Edit: I reset the bucket to zero, update, and calculate for every test.

dlindnegm commented 9 months ago

Is there a way to show a progress dialog or hourglass cursor whenever an update or calculate is in-process? There's no indication if either of those is complete or successful, and I wonder if attempting another too quickly fails because the prior one isn't complete. And could this be related to the failure of the Calculate all zones button - if it's somehow trying to process multiple zones (I currently have 6), and something is timing out or locking a file?

dlindnegm commented 9 months ago

More testing:

jeroenterheerdt commented 9 months ago

Please generate a diagnostic file, thanks

dlindnegm commented 9 months ago

Latest test result:

HA error and diagnostics file below.

Logger: homeassistant Source: custom_components/smart_irrigation/init.py:382 Integration: Smart Irrigation (documentation, issues) First occurred: 9:44:00 AM (1 occurrences) Last logged: 9:44:00 AM

Error doing job: Task exception was never retrieved Traceback (most recent call last): File "/config/custom_components/smart_irrigation/init.py", line 382, in _async_calculate_all if modinst and modinst.name=="PyETO" and modinst.forecast>0: ^^^^^^^^^^^^^^^^ AttributeError: 'PyETO' object has no attribute 'forecast'

config_entry-smart_irrigation-b243bc1b533c4e97f3a27859a84bd373.json.txt

dlindnegm commented 9 months ago

I've never changed the default 'Automatic weather data update' settings:

Automatically update weather data: Yes Update sensor data every: 1 days (First) Update at: 0:00

In my latest test I set that first update to 9:54 and the 'Automatically calculate zone durations' to 9:55, assuming that it wasn't getting OWM sensor data before it tried to perform the calculation. But it still fails with the same AttributeError: 'PyETO' object has no attribute 'forecast'.

Is there a reason why the Update and Calculate operations are separate? Trying to understand why I would ever want to perform an update of the sensor data without a follow-up calculate. And if I'm trying to perform a calculate, either scheduled or manually, shouldn't an update always occur just moments before?

jeroenterheerdt commented 9 months ago
  • Not sure I can re-create the zone gap issue but I'll watch for it.
  • With beta 21 I'm seeing an issue where I can update and calculate 6 sensors individually, but when I use the "Calculate all zones" option nothing is updated and I get the following error in my HA logs. I've attached the diag log file also.

This error originated from a custom integration. Logger: aiohttp.server Source: custom_components/smart_irrigation/init.py:382 Integration: Smart Irrigation (documentation, issues) First occurred: 9:17:31 PM (1 occurrences) Last logged: 9:17:31 PM Error handling request Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/aiohttp/web_protocol.py", line 433, in _handle_request resp = await request_handler(request) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_app.py", line 504, in _handle resp = await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/aiohttp/web_middlewares.py", line 117, in impl return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/security_filter.py", line 85, in security_filter_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/forwarded.py", line 100, in forwarded_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/request_context.py", line 28, in request_context_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/auth.py", line 236, in auth_middleware return await handler(request) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/view.py", line 148, in handle result = await handler(request, *request.match_info) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/src/homeassistant/homeassistant/components/http/data_validator.py", line 72, in wrapper result = await method(view, request, data, args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/config/custom_components/smart_irrigation/websockets.py", line 174, in post await coordinator.async_update_zone_config(zone, data) File "/config/custom_components/smart_irrigation/init.py", line 680, in async_update_zone_config await self._async_calculate_all() File "/config/custom_components/smart_irrigation/init**.py", line 382, in _async_calculate_all if modinst and modinst.name=="PyETO" and modinst.forecast>0: ^^^^^^^^^^^^^^^^ AttributeError: 'PyETO' object has no attribute 'forecast' config_entry-smart_irrigation-4c013ffa62e4a130fd84688a1714162a.json.txt

What you mention here is a total bug, one that I can easily fix.

added a but #182 for this one.

jeroenterheerdt commented 9 months ago

Latest test result:

  • Uninstalled both HA and HACS integrations & restarted
  • Installed beta 21 & restarted
  • Configured an Automatic duration calculation for 09:44
  • Did not create any new zones or modify any default zones or settings (Using OWM for all sensors).
  • Did not perform an 'Update all zones' or 'Calculate all zones'.
  • Waited for a few minutes after 09:44 and no zones calculations were performed and the same 'no data available' error was in the HA log.

HA error and diagnostics file below.

Logger: homeassistant Source: custom_components/smart_irrigation/init.py:382 Integration: Smart Irrigation (documentation, issues) First occurred: 9:44:00 AM (1 occurrences) Last logged: 9:44:00 AM

Error doing job: Task exception was never retrieved Traceback (most recent call last): File "/config/custom_components/smart_irrigation/init.py", line 382, in _async_calculate_all if modinst and modinst.name=="PyETO" and modinst.forecast>0: ^^^^^^^^^^^^^^^^ AttributeError: 'PyETO' object has no attribute 'forecast'

config_entry-smart_irrigation-b243bc1b533c4e97f3a27859a84bd373.json.txt

same bug as #182, let's fix that one first.

jeroenterheerdt commented 9 months ago

I've never changed the default 'Automatic weather data update' settings:

Automatically update weather data: Yes Update sensor data every: 1 days (First) Update at: 0:00

In my latest test I set that first update to 9:54 and the 'Automatically calculate zone durations' to 9:55, assuming that it wasn't getting OWM sensor data before it tried to perform the calculation. But it still fails with the same AttributeError: 'PyETO' object has no attribute 'forecast'.

Is there a reason why the Update and Calculate operations are separate? Trying to understand why I would ever want to perform an update of the sensor data without a follow-up calculate. And if I'm trying to perform a calculate, either scheduled or manually, shouldn't an update always occur just moments before?

you're hitting the same bug over and over again :). But you raise a good point, I think we should make calculate update the data in case nothing is available instead of throwing an warning out. I'll add a REQ for it (#183). I don't think we should have calculate update the data all the time. The reason for this separation is to maximum flexibility: you can now decide if you want to collect weather info up to the minute and have it all aggregated at calculate time. It's kind of replicating hourly run time and daily run time that we used to have in v1 but this is way more flexible.

dlindnegm commented 9 months ago

I see, so I can update the weather sensor data at 23:00 and perform the calculations the next morning based on those values. I have noticed that I get quite different run times as I update and calculate at different times during the day/night. Makes sense!

dlindnegm commented 9 months ago

Working now.