scottyphillips / pychonet

A simple to use library for interfacing with the ECHONETlite protocol
GNU General Public License v3.0
19 stars 17 forks source link

Listen to UDP port 3610 #40

Closed nao-pon closed 2 years ago

nao-pon commented 2 years ago

ECHONET Lite can be notified of device changes by listening to UDP port 3610 at the multicast address 224.0.23.0.

For example, turning 0x90 to on (0x41) / off (0x42) on a device with 0x02 - 0x72, it will get following data.

on:  10 81 00 28 02 72 01 0e f0 01 73 01 90 01 41
off: 10 81 00 29 02 72 01 0e f0 01 73 01 90 01 42

I think that you can use this to update the device changes immediately. But I'm not sure how to implement as server functionality in Python.

Do we have a chance to implement this?

scottyphillips commented 2 years ago

In the old days all the UDP functionality was very inefficient, and did not play nicely with Asyncio event loops at all. Basically I would fire a packet, listen for a response (which involved waiting for 3 seconds and pausing the entire Home Assistant) and then process the packet. It was fine when it was just me and my solo air conditioner but as more people started using the code with multiple devices, what was happening was that it was bogging Home Assistant down listening for messages.

As more people started using the Home Assistant component; and with multiple devices, I needed a better way of processing the packets in a manner that was asyncio friendly, and would work with Home Assistants event loop. To that end I used https://github.com/bashkirtsevich-llc/aioudp which provided a ready to use UDP service that was asyncio friendly. I rewrote the code in about a day.

What aio-udp-server allowed me to do was decouple the transmit and receive functionality, the receive is basically a listener that keeps the socket open at all times and can process any incoming packet that lands on port 3610. The transmit functionality works as UDP is expected to work, which is fire the message and forget. I use the TID to try to have some idea of whether or not packets are getting lost or are being received.

The drama we have is two fold:

  1. The aio-udp-server library is pretty basic and doesn't really have any coded functionality for listening to multicast.
  2. The listener (echonetMessageReceived) doesn't have much in the way of logic to process status change announcements.

To fix the first problem, we would probably have to rewrite aio-udp-server to enable a multicast listener (which should be possible, but would take some time). It would probably make sense to move aio-udp-serverinto pychonet and rewrite it as its not a particularly large library...

To fix the second problem we would have to update the listener logic to process any status change property announcements that are received on the multicast address.

Additionally, Home Assistant may or may not be able to listen to multicast; and that will really come down to the installation - if it's running in a Docker container for example it might not be possible to receive multicast packets (not that I am 100% sure).

There is some bare metal python3 socket code here for listening to multicast that I stole from stackoverflow:

import socket
import struct

sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM, socket.IPPROTO_UDP)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind(('', 4242))
mreq = struct.pack("=4sl", socket.inet_aton("224.51.105.104"), socket.INADDR_ANY)

sock.setsockopt(socket.IPPROTO_IP, socket.IP_ADD_MEMBERSHIP, mreq) 
while True:
   print sock.recv(10240)
scottyphillips commented 2 years ago

Okay I think I have the listener subscribed to the multicast address. Ive incorporated the aio-udp-server code directly into the library.

Download latest and run in interactive mode:

python3 -m asyncio

from pychonet.lib.udpserver import UDPServer
from pychonet import Factory
from pychonet import ECHONETAPIClient as api
from pychonet import HomeAirConditioner
from pychonet import EchonetInstance

udp = UDPServer()
loop = asyncio.get_event_loop()
udp.run("0.0.0.0", 3610, loop)
server = api(udp)
server._debug_flag = True

Now, in Linux with Wireshark I could definitely see multicast packets being processed but there is a bit of work to do around the logic. MACOS didn't work because multicast is blocked.

You could try to add your device eg: await server.discover('192.168.1.6')

and then see if you can see multicast packets arriving from that device.

nao-pon commented 2 years ago

@scottyphillips Great! Very fast work!

I went into the shell of the homeassistant docker instance of the HomeAssistant OS and tried it.

I got the following message! (Some excerpts)

Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
b'\x10\x81\t\x9b\x02r\x01\x05\xff\x01r\x03\xd5\x01\x0c\xee\x02\x00\xc8\xef\x01C'
('192.168.1.156', 2524)
Echonet Message Received - Processed data is {'EHD1': 16, 'EHD2': 129, 'TID': 2459, 'SEOJGC': 2, 'SEOJCC': 114, 'SEOJCI': 1, 'DEOJGC': 5, 'DEOJCC': 255, 'DEOJCI': 1, 'ESV': 114, 'OPC': [{'EPC': 213, 'PDC': 1, 'EDT': b'\x0c'}, {'EPC': 238, 'PDC': 2, 'EDT': b'\x00\xc8'}, {'EPC': 239, 'PDC': 1, 'EDT': b'C'}]}
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
b'\x10\x81\t\x9c\x02r\x01\x05\xff\x01r\n\x80\x010\x90\x01B\xd0\x01B\x91\x02\x15\x1e\xd1\x01+\xe1\x01)\xe2\x01B\xe3\x01A\xd4\x01\x06\xe4\x01B'
('192.168.1.156', 2525)
Echonet Message Received - Processed data is {'EHD1': 16, 'EHD2': 129, 'TID': 2460, 'SEOJGC': 2, 'SEOJCC': 114, 'SEOJCI': 1, 'DEOJGC': 5, 'DEOJCC': 255, 'DEOJCI': 1, 'ESV': 114, 'OPC': [{'EPC': 128, 'PDC': 1, 'EDT': b'0'}, {'EPC': 144, 'PDC': 1, 'EDT': b'B'}, {'EPC': 208, 'PDC': 1, 'EDT': b'B'}, {'EPC': 145, 'PDC': 2, 'EDT': b'\x15\x1e'}, {'EPC': 209, 'PDC': 1, 'EDT': b'+'}, {'EPC': 225, 'PDC': 1, 'EDT': b')'}, {'EPC': 226, 'PDC': 1, 'EDT': b'B'}, {'EPC': 227, 'PDC': 1, 'EDT': b'A'}, {'EPC': 212, 'PDC': 1, 'EDT': b'\x06'}, {'EPC': 228, 'PDC': 1, 'EDT': b'B'}]}
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
b'\x10\x81\t\x9d\x02r\x01\x05\xff\x01r\x03\xd5\x01\x0c\xee\x02\x00\xc8\xef\x01C'
('192.168.1.156', 2526)
Echonet Message Received - Processed data is {'EHD1': 16, 'EHD2': 129, 'TID': 2461, 'SEOJGC': 2, 'SEOJCC': 114, 'SEOJCI': 1, 'DEOJGC': 5, 'DEOJCC': 255, 'DEOJCI': 1, 'ESV': 114, 'OPC': [{'EPC': 213, 'PDC': 1, 'EDT': b'\x0c'}, {'EPC': 238, 'PDC': 2, 'EDT': b'\x00\xc8'}, {'EPC': 239, 'PDC': 1, 'EDT': b'C'}]}
Unknown host - probably multicast message
Unknown host - probably multicast message
Unknown host - probably multicast message
nao-pon commented 2 years ago

P.S. One thing I was curious about was that the ECHONET Lite entity in the HA UI was disabled while I was experimenting with running python3 in the shell.

When I stopped python3 in the shell, it returned to normal again.

If this condition is intended or expected, then there is no problem.

scottyphillips commented 2 years ago

It’s likely that the multicast listener is working then. All the repeated ‘unknown host’ messages are just the EPC processing code iterating over multiple EPCs (was tired last night and just wanted to get a multicast proof of concept out the door)

the disabling of the echonetlite node is (I speculate) probably due to hijacking the UDP port

scottyphillips commented 2 years ago

The code in question if you want to play around is: echonetMessageReceived(self, raw_data, addr) in echonetapiclient.py

We can probably just treat the packet no differently to any other received echonet message but just populate the missing host details in the main state dict.

scottyphillips commented 2 years ago

Im really limited in my ability to test multicast echonet lite outside of firing a manually crafted discovery packet - MOEKADEN room doesn't fire them - so ill have to let you take the lead on this one...

nao-pon commented 2 years ago

I tried the latest version of the test.

  1. Replace the pychonet library in the homeassistant core docker container of the HomeAsssistant OS with the latest version of master.
  2. Rewrite "from aioudp import UDPServer" in echonetlite to "from pychonet.lib.udpserver import UDPServer".
  3. I restarted the Home Assistant core.

However, the echonetlite integration did not boot up properly.

Is there anything else I need to fix?

EDITED:

Reverting to https://github.com/scottyphillips/pychonet/commit/9300dc5ce4575f32902d0caba3435e1d3529b2e2 and back to "from aioudp import UDPServer" works fine.

scottyphillips commented 2 years ago

Probably. Home assistant log should tell you what’s broken. I wouldn’t be using the latest pychonet with home assistant yet. Its not configured as a pip release. I would be running independently where you can monitor echonet packets in real time.

Get Outlook for iOShttps://aka.ms/o0ukef


From: Naoki Sawada @.> Sent: Sunday, June 19, 2022 5:36:40 PM To: scottyphillips/pychonet @.> Cc: Scott Phillips @.>; Mention @.> Subject: Re: [scottyphillips/pychonet] Listen to UDP port 3610 (Issue #40)

I tried the latest version of the test.

  1. Replace the pychonet library in the homeassistant core docker container of the HomeAsssistant OS with the latest version of master.
  2. Rewrite "from aioudp import UDPServer" in echonetlite to "from pychonet.lib.udpserver import UDPServer".
  3. I restarted the Home Assistant core.

However, the echonetlite integration did not boot up properly.

Is there anything else I need to fix?

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fscottyphillips%2Fpychonet%2Fissues%2F40%23issuecomment-1159641592&data=05%7C01%7C%7C9706be35cec4490ede9508da51c67348%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637912210040723775%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=fAs88rQTHhOjbjf3C2DI4EdxZbnNeNcWBI0ALife3NM%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGHQNDGGZSSN5AHEHPWSKK3VP3EYRANCNFSM5ZBPRULA&data=05%7C01%7C%7C9706be35cec4490ede9508da51c67348%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637912210040723775%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=OrqpDQBNRFRtMPx%2BqNeCo1zMd2abusqWVsqso1DkIAw%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

scottyphillips commented 2 years ago

Its probably easy enough to fix - change line 90 in _init__.py in echonetlite_homeassistant From this: server = ECHONETAPIClient(server=udp, loop=loop) To this: server = ECHONETAPIClient(udp)

When I was reviewing the code I got rid of some dud code on the ECHONETAPIClient.

You won't be able to use aioudp as its not configured for multicast. You have to use the modified version I added to master.

nao-pon commented 2 years ago

From this: server = ECHONETAPIClient(server=udp, loop=loop) To this: server = ECHONETAPIClient(udp)

Now it works for the time being. I will start debug after tomorrow.

nao-pon commented 2 years ago

@scottyphillips I was able to update the HA entities immediately by subscribing to the data update.

I will brush up the code and send you a PR. 😄

scottyphillips commented 2 years ago

I’ll have to release a new version of Pychonet then with the revised multicast code.

Scott

Get Outlook for iOShttps://aka.ms/o0ukef


From: Naoki Sawada @.> Sent: Monday, June 20, 2022 7:05:14 PM To: scottyphillips/pychonet @.> Cc: Scott Phillips @.>; Mention @.> Subject: Re: [scottyphillips/pychonet] Listen to UDP port 3610 (Issue #40)

@scottyphillipshttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fscottyphillips&data=05%7C01%7C%7C87a94650be424cbbc44d08da529bfd27%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637913127170805188%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=PTH8T1DYk18sVon9NGEZED1V8%2FDv7Qf0W1bY%2FY1LN2U%3D&reserved=0 I was able to update the HA entities immediately by subscribing to the data update.

I will brush up the code and send you a PR. 😄

— Reply to this email directly, view it on GitHubhttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fscottyphillips%2Fpychonet%2Fissues%2F40%23issuecomment-1160176738&data=05%7C01%7C%7C87a94650be424cbbc44d08da529bfd27%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637913127170961454%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=KHy%2FgHxVCBX0rYRnkI5%2FLxm%2FnrAVgp8%2Bh9n9oY5muTI%3D&reserved=0, or unsubscribehttps://nam12.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAGHQNDH32GLERMTLVSM7MO3VQAX4VANCNFSM5ZBPRULA&data=05%7C01%7C%7C87a94650be424cbbc44d08da529bfd27%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637913127170961454%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=mTadLUOEGG0GmVEQtv5D5mSd5PM8G67GkKj1B5SltGM%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

nao-pon commented 2 years ago

I'll let you know my branch I'm working on.

nao-pon commented 2 years ago

I updated climate.py but, it has not been verified.

nao-pon commented 2 years ago

@scottyphillips Is there a way to tell if the data received in echonetMessageReceived() is due to push or polling?

Before we stop polling, we need to make sure we get the response in push.

nao-pon commented 2 years ago

It was possible by checking the TID.

scottyphillips commented 2 years ago

This async_callback functionality is causing all kinds of issues.

For Example in home assistant, when I try to set a fan speed or configure a HVAC mode, it causes chaos. It keeps polluting the update data with "invalid setting" and I am struggling to figure out why.

If we cant resolve this, then I am going to revert all these updates because this push functionality is not really fit for purpose at the moment and is causing more problems then its solving.

See example below when I try to set the fan speed to 'low' inside the climate entity:

2022-07-09 10:38:26 DEBUG (MainThread) [custom_components.echonetlite.climate] Updated fan mode is: low
2022-07-09 10:38:26 DEBUG (MainThread) [custom_components.echonetlite]
ECHONETlite polling update data:
- Operation status (0x80): Off
- Operation status (0x80): Off
- Air flow rate setting (0xa0): Invalid setting
- Operation mode setting (0xb0): invalid_setting
- Set temperature value (0xb3): 19
- Measured value of room temperature (0xbb): 18
- Measured outdoor air temperature (0xbe): 10
2022-07-09 10:38:26 DEBUG (MainThread) [custom_components.echonetlite.climate] Called async_update_callback on boof.
Changed: True
Update data: {128: 'Off', 160: 'Invalid setting', 176: 'invalid_setting', 179: 19, 187: 18, 190: 10}
Old data: {128: 'Off', 160: 'medium-high', 176: 'invalid_setting', 179: 19, 187: 18, 190: 10}
2022-07-09 10:38:26 DEBUG (MainThread) [custom_components.echonetlite.climate] Updated fan mode is:
2022-07-09 10:38:26 ERROR (MainThread) [homeassistant.components.websocket_api.http.connection] [140027908565472] ''
Traceback (most recent call last):
File "/usr/src/homeassistant/homeassistant/components/websocket_api/commands.py", line 193, in handle_call_service
await hass.services.async_call(
File "/usr/src/homeassistant/homeassistant/core.py", line 1713, in async_call
task.result()
File "/usr/src/homeassistant/homeassistant/core.py", line 1750, in _execute_service
await cast(Callable[[ServiceCall], Awaitable[None]], handler.job.target)(
File "/usr/src/homeassistant/homeassistant/helpers/entity_component.py", line 204, in handle_service
await service.entity_service_call(
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 680, in entity_service_call
future.result() # pop exception if have
File "/usr/src/homeassistant/homeassistant/helpers/entity.py", line 930, in async_request_call
await coro
File "/usr/src/homeassistant/homeassistant/helpers/service.py", line 717, in _handle_entity_call
await result
File "/config/custom_components/echonetlite/climate.py", line 240, in async_set_fan_mode
await self._connector._instance.setFanSpeed(fan_mode)
File "/usr/local/lib/python3.10/site-packages/pychonet/HomeAirConditioner.py", line 318, in setFanSpeed
return self.setMessage(ENL_FANSPEED, FAN_SPEED[fan_speed])
KeyError: ''
nao-pon commented 2 years ago

I got it. Let's move to #98@echonetlite_homeassistant.