Grab min max from OWM daily like precip?

[jeroenterheerdt]

What happened?

See title

How to reproduce

See title

Relevant log output

No response

Which version are you running?

Beta

Diagnostics file

• [X] I have attached a diagnostics file

Additional information

No response

[dlindnegm]

When selecting OWM as the source for a sensor there is no option for using average, first, last, maximum, median, minimum, sum. If the auto-update is collecting these values every hour thru the day, which of those options are you using behind the scenes to calculate the bucket size?

It seems odd that there are sensors for Temperature, Maximum Temperature and Minimum Temperature. I'm not clear why there are also no sensor selectors for Dew Point Max and Min, Humidity Max and Min, Solar Rad Max and Min, etc...

[jeroenterheerdt]

I see OWM as the set it and forget it option. So no aggregates needed, we handle it. Averages for everything except precip which is last and max and min temp. Makes sense, right? Max temp and min temp exist as separate sensors because it's needs for the calc. Other max min values are not used in calcs. Hope this helps.

Regarding what this issue is for: I will not change this unless I get feedback from you in the other issue that what it does now is not right.

[dlindnegm]

That does make sense. I'll let mine sit for another day to look at the comparison between the 2. I think my issue has been that I've been uninstalling, reinstalling, updating to new betas, changing sensor options, emptying buckets, updating and calculating so many times that it never has had a chance to actually collect a full days worth of values. So at 23:00 it just grabs a current set of values, and at that time of day the average doesn't mean much!

[jeroenterheerdt]

I really appreciate all your efforts. I am looking forward to what it looks like on your end. We can do the experiment outlined in this issue in case we're not happy with what we're seeing. I don't think it changes much if your collect data more than once a day. However, if you set it to collecting data once a day, then it does make a difference if we collect max/min temperature from "current" or from "daily". in "current" there is no max/min really - just temperature, so if you collect only once these will be the same. In "daily" there is a max / min temperature, but historical they have not been very accurate compared to collecting multiple "current" values and calculating max/min temperature from there. Anyway, let's see.

[dlindnegm]

It's not a full days worth of data, but this is what I see this morning. I'm using the "Daily" Sensor for my min and max temps, and since that first value is from 22:55, the max temp is from the previous day and is much higher than the next 8 hours reported. Same thing for the min when using the Daily sensor as a source.

I'm wondering if instead of using the Daily min/max as a sensor I should just use my local temp sensor (using the last value). That would more closely match the hourly intervals logged by OWM and make a better comparison for the average?

Reset Buckets: 22:50 Auto Update: 22:55 Auto Calculate: 23:00

OWM Max: My Sensors Max: 21.9 31.88 20.63 22.38 20.5 22.38 19.78 22.38 19.47 22.38 19.29 22.38 18.89 22.38 18.69 22.38 18.78 22.38 20.02 22.38

---

OWM Min: My Sensors Min: 21.9 18.27 20.63 21.77 20.5 21.72 19.78 21.11 19.47 20.72 19.29 19.77 18.89 19.77 18.69 19.77 18.78 19.77 20.02 19.72

[jeroenterheerdt]

It's not a full days worth of data, but this is what I see this morning. I'm using the "Daily" Sensor for my min and max temps, and since that first value is from 22:55, the max temp is from the previous day and is much higher than the next 8 hours reported. Same thing for the min when using the Daily sensor as a source.

I'm wondering if instead of using the Daily min/max as a sensor I should just use my local temp sensor (using the last value). That would more closely match the hourly intervals logged by OWM and make a better comparison for the average?

Reset Buckets: 22:50 Auto Update: 22:55 Auto Calculate: 23:00

OWM Max: My Sensors Max: 21.9 31.88 20.63 22.38 20.5 22.38 19.78 22.38 19.47 22.38 19.29 22.38 18.89 22.38 18.69 22.38 18.78 22.38 20.02 22.38

OWM Min: My Sensors Min: 21.9 18.27 20.63 21.77 20.5 21.72 19.78 21.11 19.47 20.72 19.29 19.77 18.89 19.77 18.69 19.77 18.78 19.77 20.02 19.72

I think this is fair comparison. I think we should try with OWM getting min/max from daily data and do compile the same comparisons. If that's closer I think we should adopt that approach.

[jeroenterheerdt]

so I switched the behavior around in beta11: min and max temp are now retrieved from daily OWM data instead of current. This is better for the situation that you're only updating once a day and should do us no harm for multiple updates a day, but that remains to be seen. Please wait for a day and do the same comparison. Curious to the result!

[andreid1303]

This is not correct approach. For example, if I want to update after midnight, or not on daily basis, calculations will use only current day OW?

In case you using own aggregates you should care they are in sync with zone updates, i.e. reset them after each calculation via automations (for DailySensor uncheck automatic reset at 0:00, call daily_sensor_max_temp_reset at 23:05, or after smart_irrigation calculation event? ).

On other hand, it would be easier to debug with built in aggregates if zone sensors have attributes for weather sensors.

[andreid1303]

This should work:

automation:
 - alias: "Reset Daily Sensors for Zone MySensors"
    trigger:
      - platform: state
        entity_id: sensor.zone_mysensors
        not_from: 
          - "unknown"
          - "unavailable"
        not_to:
          - null # avoid only attributes changes
          - "0" # not reset when bucket resets
    action:
      - service: daily.daily_max_my_temperature_reset
      - service: daily.daily_min_my_temperature_reset

[jeroenterheerdt]

I am not sure what this has to do with the experiment of getting max temp data somewhere. Let me clarify what I am doing here. OWM provides current and daily weather data. Current is the current weather when the API is called and does not provide a MAX or MIN temperature, because, well, it's current. The daily weather contains the weatherdata for that day up until the point the API is called. So if you call it at 23:00 it will contain the MAX/MIN temperature for that day. The V1 of this integration used the daily info exclusively and that's why it updated only daily, at 23:00. Made sense at the time.

However, in V2 there is more flexibility. People can update once a day, at a time they choose or more often. And here is the problem. Here are some scenarios to explain the issue and why I want to see if getting max/min temp from daily is better:

Update frequency	(First) update	Temp from current	Temp from Daily	Winner
1/day	2300	Max temp = Min temp = Temp @ 2300	Max temp = max temp for day until 2300, min temp = min temp for day until 2300	Daily
1/day	1000	Max temp = Min temp = Temp @ 1000	Max temp = max temp for day until 1000, min temp = min temp for day until 1000	Daily
1/hour	0100	Max temp = max(temp @ 0100, 0200...). Min temp = min(temp @ 0100,0200,...)	Max temp = max(max temp for day until @0100,0200...), Min temp = min(temp for day until 0100,0200...)	Daily

See, in all cases getting it from daily is better.

[jeroenterheerdt]

I am not sure what this has to do with the experiment of getting max temp data somewhere. Let me clarify what I am doing here. OWM provides current and daily weather data. Current is the current weather when the API is called and does not provide a MAX or MIN temperature, because, well, it's current. The daily weather contains the weatherdata for that day up until the point the API is called. So if you call it at 23:00 it will contain the MAX/MIN temperature for that day. The V1 of this integration used the daily info exclusively and that's why it updated only daily, at 23:00. Made sense at the time.

However, in V2 there is more flexibility. People can update once a day, at a time they choose or more often. And here is the problem. Here are some scenarios to explain the issue and why I want to see if getting max/min temp from daily is better:

Update frequency	(First) update	Temp from current	Temp from Daily	Winner
1/day	2300	Max temp = Min temp = Temp at 2300	Max temp = max temp for day until 2300, min temp = min temp for day until 2300	Daily
1/day	1000	Max temp = Min temp = Temp at 1000	Max temp = max temp for day until 1000, min temp = min temp for day until 1000	Daily
1/hour	0100	Max temp = max(temp at 0100, 0200...). Min temp = min(temp at 0100,0200,...)	Max temp = max(max temp for day up to 0100,0200...), Min temp = min(temp for day until to 0100,0200...)	Daily

See, in all cases getting it from daily is arguably better.

[andreid1303]

Lets choose your second example: daily update at 10:00 with following weather conditions by OW:

• previous day: sunny afternoon, cloudy evening with minimum temp 15C at 05:00, max temp 28C at 14:00, and 24C at 10:00
• today till 10:00: cloudy, minimum temp 20C at 05:00, max temp 24C at 10:00

I assume when using "OW current", calculations will be based on min temp of 20C and max temp of 28C. And I guess that using "OW daily min, max": 20C (or 15C?) and 24C correspondently.

So it difers, and what is correct?

[jeroenterheerdt]

Lets choose your second example: daily update at 10:00 with following weather conditions by OW:

• previous day: sunny afternoon, cloudy evening with minimum temp 15C at 05:00, max temp 28C at 14:00, and 24C at 10:00
• today till 10:00: cloudy, minimum temp 20C at 05:00, max temp 24C at 10:00

I assume when using "OW current", calculations will be based on min temp of 20C and max temp of 28C. And I guess that using "OW daily min, max": 20C (or 15C?) and 24C correspondently.

So it difers, and what is correct?

This is not correct. Current will only have a concept of the current temperature, not "up until this point in time". So, in this example the max temperature by OWM current will be whatever the temperature is at 10 AM, and the min will be set to the same. Let's say it's 21 degrees C at 10AM, that is what max and min will be. Current only provides the current temperature, no max/min / up to this point in the day temperatures at all.

OWM Daily would provide min= 20C, max = 24C in your example. In summary: If I update once a day at 10:00 and the temperature is 21C at that time but varied between 20C and 24C these will be the results of the two behaviors: OWM Current: Min temp = Max temp = 21C OWM Daily: Min temp = 20C, Max temp = 24C

Hence, Daily is more correct. See the problem now?

[dlindnegm]

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value. I've attached my smart_irrigation.storage. Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely. The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion. But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9. That results in an approx 15 degree Fahrenheit difference. The min values all look normal, so just an issue with the max.

smart_irrigation.storage.txt

[andreid1303]

So behind latest beta the implementation OW sensors is static, no history was recorded? I've not get it.

[jeroenterheerdt]

So behind latest beta the implementation OW sensors is static, no history was recorded? I've not get it.

I don't think I understand what you mean and I don't know to explain this in another way than what I have already said. Id suggest: do as @dlindnegm is doing: test it and report back on comparisons with your own sensors. I am doing the same and I see better results with the updated behavior than before but the differences for me were not so profound as for @dlindnegm.

[jeroenterheerdt]

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value. I've attached my smart_irrigation.storage. Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely. The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion. But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9. That results in an approx 15 degree Fahrenheit difference. The min values all look normal, so just an issue with the max.

smart_irrigation.storage.txt

So your saying that your max temperature is not stored correctly? Just that one? All others are fine? That's weird but maybe somewhere the max temp gets a double conversion or so, not sure.

[jeroenterheerdt]

So behind latest beta the implementation OW sensors is static, no history was recorded? I've not get it.

I don't think I understand what you mean and I don't know to explain this in another way than what I have already said. Id suggest: do as @dlindnegm is doing: test it and report back on comparisons with your own sensors. I am doing the same and I see better results with the updated behavior than before but the differences for me were not so profound as for @dlindnegm.

If you collect data only once then the latest beta provides better temperature estimations that the previous ones. Why? Because max and min temp now come from daily and will provide different results where previously we got them from current, which if you collect data only once will result in min and max being the same, because current only provides the current temperature.

[andreid1303]

What about custom sensors? Their data is recorded to calculate aggregates? Why not make OW API requests at same intervals, record received current temperature then calculate max, min, mean same way?

[dlindnegm]

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value. I've attached my smart_irrigation.storage. Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely. The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion. But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9. That results in an approx 15 degree Fahrenheit difference. The min values all look normal, so just an issue with the max. smart_irrigation.storage.txt

So your saying that your max temperature is not stored correctly? Just that one? All others are fine? That's weird but maybe somewhere the max temp gets a double conversion or so, not sure.

Not sure, but it seems there's a major difference between the max values from midnight up until about 8:00 am. Maybe something to do with when each of those sensors rolls over or resets?

[jeroenterheerdt]

What about custom sensors? Their data is recorded to calculate aggregates? Why not make OW API requests at same intervals, record received current temperature then calculate max, min, mean same way?

that is exactly what is happening here... my point is that if there was only 1 call to the OWM API per day then the min and max would be be same unless we pull it from Daily. Now, if you do multiple calls both Daily and Current will return multiple values, both of which are aggregated anyway, so in that sense there is no difference.

[jeroenterheerdt]

I think there's a problem with your calculations when converting Fahrenheit to Celsius using the Daily Sensor for the Temperature Max value. I've attached my smart_irrigation.storage. Our daily high has been around 85-87 degrees, and my Daily Sensor matches that closely. The OWM reported max temp values are mostly in the range of 29.6 - 30.8, which also matches the Daily Sensor Max after conversion. But all of the max temp values reported in the .storage file for My Sensors using the Daily Sensor Max are in the range of 18.3 - 21.9. That results in an approx 15 degree Fahrenheit difference. The min values all look normal, so just an issue with the max. smart_irrigation.storage.txt

So your saying that your max temperature is not stored correctly? Just that one? All others are fine? That's weird but maybe somewhere the max temp gets a double conversion or so, not sure.

Not sure, but it seems there's a major difference between the max values from midnight up until about 8:00 am. Maybe something to do with when each of those sensors rolls over or resets?

so there seems to be a bug here. the max should never go down and the min should never go up. Not sure what is happening at 10:00 for MAX and 7:00 for MIN. What timezone are you in again? Maybe the OWM API resets what "daily" means at one of those points? That could potentially explain one of them. What did those charts look like before, in the previous beta?

[andreid1303]

As I understood data collection for aggregates starts from last calculate. Then what for is setting in General "First Update at ..."?

[dlindnegm]

Eastern Time Zone. I never knew where to look for this data before, so this is the first I've dug into it. Here's my Daily Sensor Max from HA. Since it resets at midnight it makes sense, but the OWM max doesn't seem to reset? I can change my Daily Sensors to not auto reset if you think that helps isolate this.

[jeroenterheerdt]

As I understood data collection for aggregates starts from last calculate. Then what for is setting in General "First Update at ..."?

I don't know where you got that but, no. The schedule dictates when to collect data. The first update is just when the schedule starts. So if you say: collect data every hour, you need some ability to influence if you wanted to collect at the hour and start at midnight for example. That's what that is for.

[jeroenterheerdt]

Eastern Time Zone. I never knew where to look for this data before, so this is the first I've dug into it. Here's my Daily Sensor Max from HA. Since it resets at midnight it makes sense, but the OWM max doesn't seem to reset? I can change my Daily Sensors to not auto reset if you think that helps isolate this.

no I think the Daily thing does what it is supposed to do - after all, I wrote it ;). I am just thinking that OWM daily data follows some weird 'reset' schedule that we can't control.

[dlindnegm]

https://openweather.my.site.com/s/article/min-and-max-temperature-in-current-weather-api-and-forecast-api-2019-10-24-21-47-24

Are you grabbing from the current/historical api or the daily weather forecast api? That article indicates that the current/historical api is a snapshot in time, whereas the forecast api is a daily number. I also thought that the current weather api was free, but the forecast data was only available from a paid subscription? It seems like the OWM min and max are using a sliding 24 hour window with no resets. I'll take another look at this tonight after I have a full 24 hours of data.

[jeroenterheerdt]

so we're calling the OneCall API: https://openweathermap.org/api/one-call-api, so the "current" and "daily" are both from there. I take the the first "daily" to calculate the max/min and precip because that's the expected / forecasted values for today. So it's not a true representation of what happened today, but it's a good approximation. Calling the daily forecast multiple times a day has shown the for precip/max and min temperature to change so it's not static. That's what you're seeing here. I still cannot really explain why the min / max change for you and why that happens at two different moments in time.

[andreid1303]

As I understood data collection for aggregates starts from last calculate. Then what for is setting in General "First Update at ..."?

I don't know where you got that but, no. The schedule dictates when to collect data. The first update is just when the schedule starts. So if you say: collect data every hour, you need some ability to influence if you wanted to collect at the hour and start at midnight for example. That's what that is for.

I got it from your words: https://github.com/jeroenterheerdt/HAsmartirrigation/issues/235#issuecomment-1706984974

[jeroenterheerdt]

Nowhere I said that data is automatically collected after calculate. That is your interpretation of my words, which, well eh, is wrong. The calculation and updates are only related in the sense that weatherdata collected by updates is used by calculate to do it's work and once it is successful (ie calculated a duration) it will wipe that data. Nowhere in these sentences is it implied that calculate does somehow influence the collection of data or start it again, because, well it doesn't.

[andreid1303]

Ok, collecting weather data is a continuous process. New update calculates aggregates in a time slice since previous update till the new update. Is this correct? Sorry, I mean new Calculate Zone Call manual or automatic (not update). It uses only data with timestamp after previous Calculate?

[andreid1303]

Daily OW means daily, from 0:00 till 23:59 o'clock in your time zone, they are not like start: "{{ now() - timedelta( hours = 24 ) }}" end: "{{ now() }}"

OneCall subscription is free up to 1000 calls per hour.

[jeroenterheerdt]

Daily OW means daily, from 0:00 till 23:59 o'clock in your time zone, they are not like start: "{{ now() - timedelta( hours = 24 ) }}" end: "{{ now() }}"

OneCall subscription is free up to 1000 calls per hour.

Right, so that is perfect then, right? You want the values for the day. The more I think about it it sounds like you are trying to go for a scenario that I don't see / understand and have not designed the integration for.

[jeroenterheerdt]

Ok, collecting weather data is a continuous process. New update calculates aggregates in a time slice since previous update till the new update. Is this correct? Sorry, I mean new Calculate Zone Call manual or automatic (not update). It uses only data with timestamp after previous Calculate?

You are over complicating things. I have explained multiple times how this works. The timestamp is something you made up. We don't do anything with timestamping. It is as simple as I have explained it multiple times now. Nothing more, nothing less.

[dlindnegm]

Speaking of timestamps...

How much trouble would it be to add a timestamp into the .storage file for each update? That would make it easier to put those values into a spreadsheet to see what's happening. Something like this:

      },
      {
        **"Time": 9/1/2023 23:55,**
        "Dewpoint": 21.222222222222225,
        "Humidity": 99.0,
        "Maximum Temperature": 21.722222222222218,
        "Minimum Temperature": 20.500000000000004,
        "Precipitation": 0.0,
        "Pressure": 974.7384976,
        "Solar Radiation": 3.8741760000000003,
        "Temperature": 21.5,
        "Windspeed": 0.0
      },

[andreid1303]

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!"

I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

[dlindnegm]

I set up a new Daily Sensor Test that doesn't reset at midnight and the screenshots show how the daily max can be quite different depending on what time of day you grab it. You have a much better understanding of how this all works than I, but it seems to me that the OWM max temp is a sliding 24 hour average, but the default Daily Sensor max is not, since it resets at midnight. By removing the midnight reset it looks more like the OWM max.

And in my mind, that could account for why when I have 2 zones that are set up exactly the same, with the exception of one using OWM max and one using the Daily Sensor max, the OWM bucket can be much larger depending on when the calculation was performed. Sometimes the bucket using OWM max is -4.x and the bucket using the Daily Sensor max is -.1. The OWM max seems to always be the actual maximum temp for the day, but the Daily Sensor Max (with a midnight reset) can show the maximum temp as much as 10 degrees F less that the actual high temp, and thus it's calculating a much smaller bucket/runtime.

[jeroenterheerdt]

Speaking of timestamps...

How much trouble would it be to add a timestamp into the .storage file for each update? That would make it easier to put those values into a spreadsheet to see what's happening. Something like this:

      },
      {
        **"Time": 9/1/2023 23:55,**
        "Dewpoint": 21.222222222222225,
        "Humidity": 99.0,
        "Maximum Temperature": 21.722222222222218,
        "Minimum Temperature": 20.500000000000004,
        "Precipitation": 0.0,
        "Pressure": 974.7384976,
        "Solar Radiation": 3.8741760000000003,
        "Temperature": 21.5,
        "Windspeed": 0.0
      },

I might be able to do that.

[jeroenterheerdt]

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!"

I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

[jeroenterheerdt]

I set up a new Daily Sensor Test that doesn't reset at midnight and the screenshots show how the daily max can be quite different depending on what time of day you grab it. You have a much better understanding of how this all works than I, but it seems to me that the OWM max temp is a sliding 24 hour average, but the default Daily Sensor max is not, since it resets at midnight. By removing the midnight reset it looks more like the OWM max.

And in my mind, that could account for why when I have 2 zones that are set up exactly the same, with the exception of one using OWM max and one using the Daily Sensor max, the OWM bucket can be much larger depending on when the calculation was performed. Sometimes the bucket using OWM max is -4.x and the bucket using the Daily Sensor max is -.1. The OWM max seems to always be the actual maximum temp for the day, but the Daily Sensor Max (with a midnight reset) can show the maximum temp as much as 10 degrees F less that the actual high temp, and thus it's calculating a much smaller bucket/runtime.

I think you might be on to something here, but I don't know a good solution yet. We can't chadnge what OWM daily provides. So if we wanted to go back to OWM current how do we account for the situation that someone does only one update a day and gets wonky results? Remove that option for the integration and force an update at least twice a day? Or just leave it like this and document that you should not do this? Ideas?

[andreid1303]

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!" I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

Ye, I do not understand what is behind the GUI. If I call Update - Calculate X times my bucket decreases X times like: -5; - 10; - 15; -20 etc. Is it normal?

[jeroenterheerdt]

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!" I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

Ye, I do not understand what is behind the GUI. If I call Update - Calculate X times my bucket decreases X times like: -5; - 10; - 15; -20 etc. Is it normal?

Yes, of course. You asked us to calculate multiple times, so that's what we do. You yourself are responsible for the calculate, irrigate, reset bucket cycle, but you only need to reset the bucket if you irrigated. That way the integration is agnostic to what you use to run your irrigation.

[dlindnegm]

I think you might be on to something here, but I don't know a good solution yet. We can't chadnge what OWM daily provides. So if we wanted to go back to OWM current how do we account for the situation that someone does only one update a day and gets wonky results? Remove that option for the integration and force an update at least twice a day? Or just leave it like this and document that you should not do this? Ideas?

Can you provide the syntax for the API call that includes the OWM min and max? I'd like to compare those results to the Daily Sensor numbers.

I'm thinking that the best approach might be to go back to grabbing the OWM current temps every hour, and drop any reference to min and max temp sensors altogether. Even the Daily Sensor Max without a midnight reset can have dramatically different results depending on when it's calculated.

So folks would only need to provide a current temp sensor of their own, or use the OWM current temp. You continue to log those values every minute, hour, day (user selectable), and at every calculate you grab whatever is logged into the .storage file and select the max and min values for each. I don't see any issue with the number of API calls, since the free tier provides 60/minute and 1 million calls per month.

Leave the default like it is - update every hour. If someone wants to use only one update per day that's their choice, and you can document the issues with frequency of updates and their effect on the results. I think you could drop the option to update for less than one hour, as I don't see that improving accuracy more than once per hour. So drop the option to update every x minutes. And if someone only wanted to update every 5 days! they would have 120 current temp entries logged, and you just grab the max and min value from those. Would that make everybody happy?

[jeroenterheerdt]

that is certainly an option. I can make a new version with this proposal. It will take me a bit of rewiring, but it sounds like that's a good middle ground. thanks!

[andreid1303]

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!" I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

Ye, I do not understand what is behind the GUI. If I call Update - Calculate X times my bucket decreases X times like: -5; - 10; - 15; -20 etc. Is it normal?

Yes, of course. You asked us to calculate multiple times, so that's what we do. You yourself are responsible for the calculate, irrigate, reset bucket cycle, but you only need to reset the bucket if you irrigated. That way the integration is agnostic to what you use to run your irrigation.

Ok, I got more:

1. At 18:00 I called Update All Zones, Calculate All Zones and got buckets -5,0 mm for OW zone and -5,1 mm for MySensors. Then "Irrigate" 5 mm and Reset All Buckets.
2. Now I wish "Irrigate" again but only ETO past two hours, did same Update, Calculate and got - 2,7 mm for OW and 5,0 mm for MySensors. First, evidently this is not past 2 hours ETO, and second why such difference between OW and MySensors this time?

[jeroenterheerdt]

Sorry, I missed were I can read how zone bucket calculates in timeline? Just as simple as "When you call Calculate service, the collected sensor data with interval X from time A to time B is analysed to get predefined aggregates ..." and "for OW sensors it is different (or same)?" If it is different why we compare results? "Just choose one and trust it!" I give up, really can't help you. Better spend time to write own python script for pyETO and automations.

And here I was thinking I was helping you. I really don't understand what you want to hear. I tried. It's very simple. There is no relation between calculate and any interval. You call it and all data that is on the sensor group is used and after a successful calculate that is wiped. How that data got there is irrelevant.

Ye, I do not understand what is behind the GUI. If I call Update - Calculate X times my bucket decreases X times like: -5; - 10; - 15; -20 etc. Is it normal?

Yes, of course. You asked us to calculate multiple times, so that's what we do. You yourself are responsible for the calculate, irrigate, reset bucket cycle, but you only need to reset the bucket if you irrigated. That way the integration is agnostic to what you use to run your irrigation.

Ok, I got more:

1. At 18:00 I called Update All Zones, Calculate All Zones and got buckets -5,0 mm for OW zone and -5,1 mm for MySensors. Then "Irrigate" 5 mm and Reset All Buckets.
2. Now I wish "Irrigate" again but only ETO past two hours, did same Update, Calculate and got - 2,7 mm for OW and 5,0 mm for MySensors. First, evidently this is not past 2 hours ETO, and second why such difference between OW and MySensors this time?

that is a good question. However, please open a separate issue and provide your diagnostic file as requested. without it is impossible to explain.

[andreid1303]

I think you might be on to something here, but I don't know a good solution yet. We can't chadnge what OWM daily provides. So if we wanted to go back to OWM current how do we account for the situation that someone does only one update a day and gets wonky results? Remove that option for the integration and force an update at least twice a day? Or just leave it like this and document that you should not do this? Ideas?

Can you provide the syntax for the API call that includes the OWM min and max? I'd like to compare those results to the Daily Sensor numbers.

I'm thinking that the best approach might be to go back to grabbing the OWM current temps every hour, and drop any reference to min and max temp sensors altogether. Even the Daily Sensor Max without a midnight reset can have dramatically different results depending on when it's calculated.

So folks would only need to provide a current temp sensor of their own, or use the OWM current temp. You continue to log those values every minute, hour, day (user selectable), and at every calculate you grab whatever is logged into the .storage file and select the max and min values for each. I don't see any issue with the number of API calls, since the free tier provides 60/minute and 1 million calls per month.

Leave the default like it is - update every hour. If someone wants to use only one update per day that's their choice, and you can document the issues with frequency of updates and their effect on the results. I think you could drop the option to update for less than one hour, as I don't see that improving accuracy more than once per hour. So drop the option to update every x minutes. And if someone only wanted to update every 5 days! they would have 120 current temp entries logged, and you just grab the max and min value from those. Would that make everybody happy?

If you have own solar radiance sensor (that is more accurate than estimate from temp), you will not get accurate mean with 1 hour update because clouds can cover the sun for minutes. May be better to move Update' and Auto Calculate constants into Sensor Group objects.

[andreid1303]

config_entry-smart_irrigation-6e18a9ad38f8ac00ef4285672ad17839.json-6.txt

[jeroenterheerdt]

that is certainly an option. I can make a new version with this proposal. It will take me a bit of rewiring, but it sounds like that's a good middle ground. thanks!

implemented this in beta12! Also added the timestamps as you suggested: retrieval time on weatherdata in the storage file, last updated data time on the sensor group and last calculation time is stored on zone.

Let me know if you see improvements now!

[dlindnegm]

Installed beta12 and will watch it tomorrow. One thing I just noticed with beta11 - Before I upgraded to beta12 I manually copied off the smart_irrigation.storage file and noticed that there were only 11 entries for weather data. My setting for Automatic weather data update is set to 23:50, but only 11 entries means that something cleared all of the weather data sometime before or at 11:50??? There's no calculation data in the file, so it doesn't seem like that's what reset it. Adding the time stamps will show if it continues tomorrow.