search

custom-components / sensor.owlintuition

A set of sensors to integrate the OWL Intuition devices network
MIT License
11 stars 9 forks source link

Not seeing all packets #18

Closed shortbloke closed 3 years ago

shortbloke commented 3 years ago

Performing a network capture of traffic from the Network Owl to my Home Assistant instance, I can see 50 packets were sent in less than 10 minutes, yet in the same time period (with debug logging) I can see the owl intuition component only received around 11 packets, all of which were of type electricity. Yet my packet capture showed 3 solar packets during this time which don't appear to have been received/processed.

shortbloke commented 3 years ago

It appears that the component only reports one message per minute, which may only be the last packet received. (TBC)

When the Solar is changing rapidly it may only process solar messages, in evening/night solar messages are less frequent compared to electricity and so more electricity packets are processed.

For example in the last 15mins (during a sunny spell) HA has processed 15 solar messages and zero electricity ones. This causes sensors to be inaccurate, or stay as NaN for some time until they are lucky enough to be the last packet in the 1 min period.

I understand the need for rate limiting, but I think this needs to be a max of 1 sensor update per min for each main type.

shortbloke commented 3 years ago

Ah, just found threads discussing this very problem: https://community.home-assistant.io/t/owl-intuition-pv-home-assistant/18157/93

Seems to be an issue with the current implementation, but without a final resolution on how to make this component reliable.

glpatcern commented 3 years ago

Indeed, this was discussed at length and it's a well known issue with the listening logic for UDP traffic in this integration. A few years ago I gave it a try to use the async loop (see https://community.home-assistant.io/t/async-update-guidelines/51283) but I could not get a stable solution.

I should possibly retry now as HA has evolved a great deal, or go for what was discussed in the thread you mentioned, that is a proxy component, as used in another integration: despite it makes the deployment more complex - you'd have another (dockerized) process running in your system - it would ensure capturing 100% of the UDP traffic. What do you think?

Almost needless to say, nothing has happened yet because I had really zero time to invest on this...

shortbloke commented 3 years ago

I looked at the possibility of using a proxy, but agree that seems overly complex for this specific integration. I'm just now experimenting with using the Home Assistant Node-Red integration as it provides a UDP listener which appears to work more reliably. See: https://github.com/shortbloke/Node-Red-Owl-Intuition-HomeAssistant

I need to finish testing and write up some basic setup instructions. I'd prefer a proper working component/integration, but Node-Red seems a practical alternative.

glpatcern commented 3 years ago

Indeed, to the minimum I'd add a proxy as an optional feature defaulting to the normal listener, but eventually I've been hesitant (with the minimal time available...) to implement all that.

I did not know about Node-Red, interesting. But yeah maintainability of the JSON "code" is pretty bad compared to a dedicated native integration.

glpatcern commented 3 years ago

Anyway, trying to get into the issue itself, from your comment above it seems you never get more than one message per minute; but this code:

https://github.com/custom-components/sensor.owlintuition/blob/master/custom_components/owlintuition/sensor.py#L168

is supposed to ensure a less aggressive rate limiting when multiple entities are defined. One test you could do is to set secs = 1 to essentially drop any rate limiting. But IIRC from the thread in the discourse forum, it seems that solar messages are sent "right after" electricity messages (or viceversa), thus there's always a chance to miss one of them. Would you confirm that?

shortbloke commented 3 years ago

Removing the rate-limiting (setting the time to 1) shows my packets time-stamped 31 seconds apart. 

2021-09-09 15:29:12 DEBUG (SyncWorker_0) [custom_components.owlintuition.sensor] Packet: <electricity id='443719100B48' ver='2.0'><timestamp>1631197752</timestamp><signal rssi='-65' lqi='32'/><battery level='100%'/><channels><chan id='0'><curr units='w'>5040.00</curr><day units='wh'>36963.88</day></chan><chan id='1'><curr units='w'>538.00</curr><day units='wh'>5204.40</day></chan><chan id='2'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='3'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='4'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='5'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan></channels><property><current><watts>5040.00</watts><cost>56.77</cost></current><day><wh>36963.88</wh><cost>424.04</cost></day><tariff time='1631201352'><start>1631145600</start><curr_price>0.13</curr_price><block_limit>4294967295</block_limit><block_usage>30203</block_usage></tariff></property></electricity>
2021-09-09 15:29:12 DEBUG (SyncWorker_0) [custom_components.owlintuition.sensor] Datagram received for type electricity
2021-09-09 15:29:19 DEBUG (SyncWorker_8) [custom_components.owlintuition.sensor] Packet: <solar id='443719100B48'><timestamp>1631197759</timestamp><current><generating units='w'>521.00</generating><exporting units='w'>0.00</exporting></current><day><generated units='wh'>5204.40</generated><exported units='wh'>154.00</exported></day></solar>
2021-09-09 15:29:19 DEBUG (SyncWorker_8) [custom_components.owlintuition.sensor] Datagram received for type solar
2021-09-09 15:29:55 DEBUG (SyncWorker_6) [custom_components.owlintuition.sensor] Packet: <solar id='443719100B48'><timestamp>1631197795</timestamp><current><generating units='w'>488.00</generating><exporting units='w'>0.00</exporting></current><day><generated units='wh'>5213.96</generated><exported units='wh'>154.00</exported></day></solar>
2021-09-09 15:29:55 DEBUG (SyncWorker_6) [custom_components.owlintuition.sensor] Datagram received for type solar
2021-09-09 15:30:24 DEBUG (SyncWorker_4) [custom_components.owlintuition.sensor] Packet: <electricity id='443719100B48' ver='2.0'><timestamp>1631197824</timestamp><signal rssi='-65' lqi='39'/><battery level='100%'/><channels><chan id='0'><curr units='w'>5695.00</curr><day units='wh'>37037.32</day></chan><chan id='1'><curr units='w'>471.00</curr><day units='wh'>5213.96</day></chan><chan id='2'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='3'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='4'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='5'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan></channels><property><current><watts>5695.00</watts><cost>65.87</cost></current><day><wh>37037.32</wh><cost>425.13</cost></day><tariff time='1631201424'><start>1631145600</start><curr_price>0.13</curr_price><block_limit>4294967295</block_limit><block_usage>30286</block_usage></tariff></property></electricity>
2021-09-09 15:30:24 DEBUG (SyncWorker_4) [custom_components.owlintuition.sensor] Datagram received for type electricity
2021-09-09 15:30:55 DEBUG (SyncWorker_14) [custom_components.owlintuition.sensor] Packet: <solar id='443719100B48'><timestamp>1631197855</timestamp><current><generating units='w'>454.00</generating><exporting units='w'>0.00</exporting></current><day><generated units='wh'>5222.14</generated><exported units='wh'>154.00</exported></day></solar>
2021-09-09 15:30:55 DEBUG (SyncWorker_14) [custom_components.owlintuition.sensor] Datagram received for type solar
2021-09-09 15:31:24 DEBUG (SyncWorker_5) [custom_components.owlintuition.sensor] Packet: <electricity id='443719100B48' ver='2.0'><timestamp>1631197884</timestamp><signal rssi='-65' lqi='40'/><battery level='100%'/><channels><chan id='0'><curr units='w'>4989.00</curr><day units='wh'>37126.56</day></chan><chan id='1'><curr units='w'>471.00</curr><day units='wh'>5222.14</day></chan><chan id='2'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='3'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='4'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='5'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan></channels><property><current><watts>4989.00</watts><cost>56.97</cost></current><day><wh>37126.56</wh><cost>426.10</cost></day><tariff time='1631201484'><start>1631145600</start><curr_price>0.13</curr_price><block_limit>4294967295</block_limit><block_usage>30361</block_usage></tariff></property></electricity>
2021-09-09 15:31:24 DEBUG (SyncWorker_5) [custom_components.owlintuition.sensor] Datagram received for type electricity
2021-09-09 15:31:55 DEBUG (SyncWorker_5) [custom_components.owlintuition.sensor] Packet: <solar id='443719100B48'><timestamp>1631197915</timestamp><current><generating units='w'>471.00</generating><exporting units='w'>0.00</exporting></current><day><generated units='wh'>5228.48</generated><exported units='wh'>154.00</exported></day></solar>
2021-09-09 15:31:55 DEBUG (SyncWorker_5) [custom_components.owlintuition.sensor] Datagram received for type solar
2021-09-09 15:32:24 DEBUG (SyncWorker_9) [custom_components.owlintuition.sensor] Packet: <electricity id='443719100B48' ver='2.0'><timestamp>1631197944</timestamp><signal rssi='-64' lqi='32'/><battery level='100%'/><channels><chan id='0'><curr units='w'>4956.00</curr><day units='wh'>37209.71</day></chan><chan id='1'><curr units='w'>488.00</curr><day units='wh'>5228.48</day></chan><chan id='2'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='3'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='4'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='5'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan></channels><property><current><watts>4956.00</watts><cost>56.34</cost></current><day><wh>37209.71</wh><cost>427.12</cost></day><tariff time='1631201544'><start>1631145600</start><curr_price>0.13</curr_price><block_limit>4294967295</block_limit><block_usage>30439</block_usage></tariff></property></electricity>
2021-09-09 15:32:24 DEBUG (SyncWorker_9) [custom_components.owlintuition.sensor] Datagram received for type electricity
2021-09-09 15:32:55 DEBUG (SyncWorker_14) [custom_components.owlintuition.sensor] Packet: <solar id='443719100B48'><timestamp>1631197975</timestamp><current><generating units='w'>504.00</generating><exporting units='w'>0.00</exporting></current><day><generated units='wh'>5236.96</generated><exported units='wh'>154.00</exported></day></solar>
2021-09-09 15:32:55 DEBUG (SyncWorker_14) [custom_components.owlintuition.sensor] Datagram received for type solar
2021-09-09 15:33:24 DEBUG (SyncWorker_1) [custom_components.owlintuition.sensor] Packet: <electricity id='443719100B48' ver='2.0'><timestamp>1631198004</timestamp><signal rssi='-65' lqi='33'/><battery level='100%'/><channels><chan id='0'><curr units='w'>4552.00</curr><day units='wh'>37294.33</day></chan><chan id='1'><curr units='w'>504.00</curr><day units='wh'>5236.96</day></chan><chan id='2'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='3'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='4'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='5'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan></channels><property><current><watts>4552.00</watts><cost>51.05</cost></current><day><wh>37294.33</wh><cost>427.96</cost></day><tariff time='1631201604'><start>1631145600</start><curr_price>0.13</curr_price><block_limit>4294967295</block_limit><block_usage>30505</block_usage></tariff></property></electricity>
2021-09-09 15:33:24 DEBUG (SyncWorker_1) [custom_components.owlintuition.sensor] Datagram received for type electricity
2021-09-09 15:33:55 DEBUG (SyncWorker_7) [custom_components.owlintuition.sensor] Packet: <solar id='443719100B48'><timestamp>1631198035</timestamp><current><generating units='w'>488.00</generating><exporting units='w'>0.00</exporting></current><day><generated units='wh'>5246.51</generated><exported units='wh'>154.00</exported></day></solar>
2021-09-09 15:33:55 DEBUG (SyncWorker_7) [custom_components.owlintuition.sensor] Datagram received for type solar
2021-09-09 15:34:24 DEBUG (SyncWorker_0) [custom_components.owlintuition.sensor] Packet: <electricity id='443719100B48' ver='2.0'><timestamp>1631198064</timestamp><signal rssi='-63' lqi='16'/><battery level='100%'/><channels><chan id='0'><curr units='w'>3124.00</curr><day units='wh'>37355.55</day></chan><chan id='1'><curr units='w'>471.00</curr><day units='wh'>5246.51</day></chan><chan id='2'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='3'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='4'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan><chan id='5'><curr units='w'>0.00</curr><day units='wh'>0.00</day></chan></channels><property><current><watts>3124.00</watts><cost>33.45</cost></current><day><wh>37355.55</wh><cost>428.58</cost></day><tariff time='1631201664'><start>1631145600</start><curr_price>0.13</curr_price><block_limit>4294967295</block_limit><block_usage>30550</block_usage></tariff></property></electricity>
2021-09-09 15:34:24 DEBUG (SyncWorker_0) [custom_components.owlintuition.sensor] Datagram received for type electricity

Should have tried that before spending time with Node-Red (which I agree isn't friendly from a code maintenance point of view). I just assumed it wouldn't help, like it didn't help the other person in the thread.

glpatcern commented 3 years ago

Thanks, I didn't think about trying again as it didn't help in the past, but yeah at this point this is something easier to work on. I'm actually considering to just hard-code the throttling to the minimum as anyway HA is running this in a thread (and eventually the async loop must work, too).

psl3 commented 3 years ago

@shortbloke - have installed your latest Energy Component code and will test. I have been delayed due to struggling to get Owl electricity power and solar power both working at the same time. If I reboot then I seem to lose one or the other. I hope this is the 'missing packets' issue that is being covered in this thread - would be great to have a solution.

shortbloke commented 3 years ago

@psl3 To clarify the component is the work of @glpatcern and it's his update to remove throttling that should help with the packet loss issue. Would be good to know if it works better for you like it did for me.

Hopefully you saw my revision to the template sensor to provide Grid energy.

psl3 commented 3 years ago

Thanks @shortbloke and @glpatcern. I did reboot yesterday and again got electricity power but 'unknown' for solar power and solar export. Then installed the sensor.py code provided by @glpatcern and instantly I have all three owl readings. With @shortbloke's latest grid code too I have had my first realistic Energy component read out. (the increase at 12.00 is real as we switched on the hot tub, otherwise looks like I would have even exported power for a few hours). image

glpatcern commented 3 years ago

Glad this worked! I think this closes the issue for now, and @shortbloke feel free to contribute your templated sensor with a PR, maybe depending how quickly there's a reaction in the issue you raised with HA core.

shortbloke commented 3 years ago

@glpatcern Have submitted a PR, but there remains an issue when the solar and elec reset their total usage at different times. I'm struggling to see how to handle this. Created issue #20 as somewhere to discuss how this might be tackled if you can offer any ideas. (hopefully, you don't mind me using issues in this way).