Shelly Gen1 Door/Window Sensor not working over CoIoT

uuf6429 commented 1 month ago

The problem

I was trying to set up a shelly sensor and followed the linked documentation to the letter. I tried other approaches (e.g. mqtt, different settings, hosts, ports and so on).

In the end I turned off the HA container and "replaced" it with a UDP listener container:

  ha-udp-test:
    image: mendhak/udp-listener
    privileged: true
    network_mode: host
    environment:
      - UDPPORT=5683

This actually worked:

So this test means:

connection is passing through firewall just fine
connection to the host is working fine
the device seemed to be properly configured (networking-wise)

I checked that port 5683 is allocated for both containers (the host is Windows, by the way) with powershell Get-NetUDPEndpoint and the listing was identical for both containers.

Next I opened a terminal in the HA container and ran netstat -ulp, which also looks fine:

Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
udp        0      0 0.0.0.0:1900            0.0.0.0:*                           68/python3          
udp        0      0 0.0.0.0:1900            0.0.0.0:*                           68/python3          
udp        0      0 0.0.0.0:39172           0.0.0.0:*                           68/python3          
udp        0      0 0.0.0.0:48224           0.0.0.0:*                           68/python3          
udp        0      0 0.0.0.0:53156           0.0.0.0:*                           -                   
udp        0      0 0.0.0.0:111             0.0.0.0:*                           -                   
udp        0      0 localhost:716           0.0.0.0:*                           -                   
udp        0      0 192.168.65.3:5353       0.0.0.0:*                           68/python3          
udp        0      0 0.0.0.0:5683            0.0.0.0:*                           68/python3          <<<<
udp        0      0 0.0.0.0:38501           0.0.0.0:*                           68/python3          
udp6       0      0 [::]:1900               [::]:*                              68/python3          
udp6       0      0 [::]:1900               [::]:*                              68/python3          
udp6       0      0 [::]:51179              [::]:*                              68/python3          
udp6       0      0 [::]:35430              [::]:*                              68/python3          
udp6       0      0 [::]:111                [::]:*                              -                   
udp6       0      0 [::]:38066              [::]:*                              -                   
udp6       0      0 [::]:5353               [::]:*                              68/python3          
udp6       0      0 fdc4:f303:9324::3:5353  [::]:*                              68/python3          
udp6       0      0 [::]:38132              [::]:*                              68/python3

It's interesting to note that if I run "Reconfigure" in the HA interface (while the device is "actively configurable"), sensor values are picked up correctly. I am guessing this is because of some sort of API polling, completely unrelated to CoIoT.

At the moment, my very rough guess is that something is wrong with parsing the data, although I do not see any logs to (dis)prove that.

Sidenote: I also dug into the ha-core source, but couldn't find obvious references to the shelly integration with port 5683, then I found this repository - is that the actual implementation for handling CoIoT? Should this issue be (re/cross)posted there?

What version of Home Assistant Core has the issue?

core-2024.10.2

What was the last working version of Home Assistant Core?

core-2024.10.2

What type of installation are you running?

Home Assistant Container

Integration causing the issue

shelly

Link to integration documentation on our website

https://www.home-assistant.io/integrations/shelly

Diagnostics information

{
  "home_assistant": {
    "installation_type": "Home Assistant Container",
    "version": "2024.10.2",
    "dev": false,
    "hassio": false,
    "virtualenv": false,
    "python_version": "3.12.4",
    "docker": true,
    "arch": "x86_64",
    "timezone": "Europe/Berlin",
    "os_name": "Linux",
    "os_version": "5.15.153.1-microsoft-standard-WSL2",
    "run_as_root": true
  },
  "custom_components": {},
  "integration_manifest": {
    "domain": "shelly",
    "name": "Shelly",
    "codeowners": [
      "balloob",
      "bieniu",
      "thecode",
      "chemelli74",
      "bdraco"
    ],
    "config_flow": true,
    "dependencies": [
      "bluetooth",
      "http",
      "network"
    ],
    "documentation": "https://www.home-assistant.io/integrations/shelly",
    "integration_type": "device",
    "iot_class": "local_push",
    "loggers": [
      "aioshelly"
    ],
    "quality_scale": "platinum",
    "requirements": [
      "aioshelly==11.4.2"
    ],
    "zeroconf": [
      {
        "type": "_http._tcp.local.",
        "name": "shelly*"
      }
    ],
    "is_built_in": true
  },
  "setup_times": {
    "null": {
      "setup": 3.854499664157629e-05
    },
    "01JA2QW4TWGQNP0RW3HEH4GMBE": {
      "wait_import_platforms": -2.4835014600103023,
      "wait_base_component": -0.00021679700876120478,
      "config_entry_setup": 2.6169942100095795
    },
    "01JATNDQQJ35G089C4B36FEPAJ": {
      "wait_import_platforms": -2.6144183689903002,
      "config_entry_setup": 2.618026595999254
    }
  },
  "data": {
    "entry": {
      "created_at": "2024-10-22T17:47:02.258142+00:00",
      "data": {
        "host": "<shelly sensor host>",
        "port": 80,
        "sleep_period": 43200,
        "model": "SHDW-2",
        "gen": 1
      },
      "discovery_keys": {},
      "disabled_by": null,
      "domain": "shelly",
      "entry_id": "01JATNDQQJ35G089C4B36FEPAJ",
      "minor_version": 2,
      "modified_at": "2024-10-22T20:07:28.416620+00:00",
      "options": {},
      "pref_disable_new_entities": false,
      "pref_disable_polling": false,
      "source": "user",
      "title": "Shelly Door/Window 2 (SHDW-2)",
      "unique_id": "C8C9A318E2F0",
      "version": 1
    },
    "device_info": {
      "name": "Shelly Door/Window 2 (SHDW-2)",
      "model": "SHDW-2",
      "sw_version": ""
    },
    "device_settings": "not initialized",
    "device_status": "not initialized",
    "last_error": "not initialized",
    "bluetooth": "not initialized"
  }
}

Example YAML snippet

No response

Anything in the logs that might be useful for us?

2024-10-23 08:38:58.278 DEBUG (MainThread) [aioshelly.block_device.coap] CoapMessage: ip=127.0.0.1, type=CoapType.PERIODIC(30), options={11: b's', 3332: b'SHDW-2#18E2F0#2', 3412: b'\x96\x00', 3420: b'\x01\x00'}, payload={'G': [[0, 9103, 0], [0, 3108, 1], [0, 3119, 0], [0, 3109, -1], [0, 6110, -1], [0, 3106, 31], [0, 3110, 'dark'], [0, 3101, 26.4], [0, 3102, 79.52], [0, 3115, 0], [0, 3111, 100], [0, 9102, ['button']]]}
2024-10-23 08:38:58.279 DEBUG (MainThread) [aioshelly.block_device.coap] Calling CoAP message update for device id 18E2F0
2024-10-23 08:38:58.279 DEBUG (MainThread) [homeassistant.components.shelly] Shelly Shelly Door/Window 2 (SHDW-2) handle update, type: BlockUpdateType.ONLINE
2024-10-23 08:38:58.279 DEBUG (MainThread) [homeassistant.components.shelly] Connecting to Shelly Device - Shelly Door/Window 2 (SHDW-2)
2024-10-23 08:38:58.279 DEBUG (MainThread) [aioshelly.block_device.device] host <shelly sensor host>: block device initialize
2024-10-23 08:38:58.281 DEBUG (MainThread) [homeassistant.components.shelly] Manually updated Shelly Door/Window 2 (SHDW-2) data
2024-10-23 08:38:58.300 DEBUG (MainThread) [aioshelly.block_device.device] host <shelly sensor host>: http request: /settings (params=None)
2024-10-23 08:38:58.403 DEBUG (MainThread) [aioshelly.block_device.device] aiohttp response: {'device': {'type': 'SHDW-2', 'mac': 'C8C9A318E2F0', 'hostname': 'shellydw2-18E2F0', 'sleep_mode': True}, 'wifi_ap': {'enabled': False, 'ssid': 'shellydw2-18E2F0', 'key': ''}, 'wifi_sta': {'enabled': True, 'ssid': '<wifi 2.4G ssid>', 'ipv4_method': 'static', 'ip': '<shelly sensor host>', 'gw': None, 'mask': '255.255.255.0', 'dns': '8.8.8.8'}, 'wifi_sta1': {'enabled': False, 'ssid': None, 'ipv4_method': 'dhcp', 'ip': None, 'gw': None, 'mask': None, 'dns': None}, 'mqtt': {'enable': False, 'server': '<ha host>:1883', 'user': '', 'id': 'shellydw2-18E2F0', 'reconnect_timeout_max': 60.0, 'reconnect_timeout_min': 2.0, 'clean_session': True, 'keep_alive': 60, 'max_qos': 0, 'retain': True, 'update_period': 30}, 'coiot': {'enabled': True, 'update_period': 15, 'peer': '<ha host>:5683'}, 'sntp': {'server': 'at.pool.ntp.org', 'enabled': True}, 'login': {'enabled': False, 'unprotected': False, 'username': 'admin'}, 'pin_code': '', 'name': 'Closet Door', 'fw': '20231107-163707/v1.14.1-rc1-g0617c15', 'pon_wifi_reset': False, 'discoverable': True, 'build_info': {'build_id': '20231107-163707/v1.14.1-rc1-g0617c15', 'build_timestamp': '2023-11-07T16:37:07Z', 'build_version': '1.0'}, 'cloud': {'enabled': False, 'connected': False}, 'timezone': None, 'lat': 500.0, 'lng': 500.0, 'tzautodetect': True, 'tz_utc_offset': 0, 'tz_dst': False, 'tz_dst_auto': True, 'time': '', 'unixtime': 0, 'led_status_disable': False, 'debug_enable': False, 'allow_cross_origin': False, 'actions': {'active': False, 'names': ['report_url', 'dark_url', 'twilight_url', 'open_url', 'close_url', 'vibration_url', 'temp_over_url', 'temp_under_url']}, 'dark_threshold': 100, 'twilight_threshold': 300, 'sleep_mode': {'period': 12, 'unit': 'h'}, 'tilt_enabled': False, 'tilt_calibrated': False, 'vibration_enabled': False, 'vibration_sensitivity': 50, 'reverse_open_close': False, 'lux_wakeup_enable': True, 'sensors': {'temperature_threshold': 1.0, 'temperature_unit': 'C'}, 'temperature_offset': 0.0}
2024-10-23 08:38:58.403 DEBUG (MainThread) [aioshelly.block_device.device] host <shelly sensor host>: http request: /status (params=None)
2024-10-23 08:38:58.435 DEBUG (MainThread) [aioshelly.block_device.device] aiohttp response: {'wifi_sta': {'connected': True, 'ssid': '<wifi 2.4G ssid>', 'ip': '<shelly sensor host>', 'rssi': -46}, 'cloud': {'enabled': False, 'connected': False}, 'mqtt': {'connected': False}, 'time': '', 'unixtime': 0, 'serial': 1, 'has_update': False, 'mac': 'C8C9A318E2F0', 'cfg_changed_cnt': 0, 'actions_stats': {'skipped': 0}, 'is_valid': True, 'tmp': {'value': 26.4, 'units': 'C', 'tC': 26.4, 'tF': 79.52, 'is_valid': True}, 'lux': {'value': 31, 'illumination': 'dark', 'is_valid': True}, 'accel': {'tilt': -1, 'vibration': -1}, 'sensor': {'state': 'open', 'is_valid': True}, 'bat': {'value': 100, 'voltage': 5.97}, 'act_reasons': ['button'], 'sensor_error': 0, 'update': {'status': 'unknown', 'has_update': False, 'new_version': '', 'old_version': '20231107-163707/v1.14.1-rc1-g0617c15'}, 'ram_total': 52216, 'ram_free': 41708, 'fs_size': 233681, 'fs_free': 128010, 'uptime': 2}
2024-10-23 08:38:58.435 DEBUG (MainThread) [aioshelly.block_device.coap] Sending request 'cit/d' to device <shelly sensor host>
2024-10-23 08:39:02.859 DEBUG (MainThread) [aioshelly.block_device.coap] CoapMessage: ip=127.0.0.1, type=CoapType.PERIODIC(30), options={11: b's', 3332: b'SHDW-2#18E2F0#2', 3412: b'\x96\x00', 3420: b'\x01\x00'}, payload={'G': [[0, 9103, 0], [0, 3108, 1], [0, 3119, 0], [0, 3109, -1], [0, 6110, -1], [0, 3106, 121], [0, 3110, 'twilight'], [0, 3101, 26.6], [0, 3102, 79.88], [0, 3115, 0], [0, 3111, 100], [0, 9102, ['button']]]}
2024-10-23 08:39:02.859 DEBUG (MainThread) [aioshelly.block_device.coap] Calling CoAP message update for device id 18E2F0
2024-10-23 08:39:08.174 DEBUG (MainThread) [aioshelly.block_device.coap] CoapMessage: ip=127.0.0.1, type=CoapType.PERIODIC(30), options={11: b's', 3332: b'SHDW-2#18E2F0#2', 3412: b'\x96\x00', 3420: b'\x01\x00'}, payload={'G': [[0, 9103, 0], [0, 3108, 1], [0, 3119, 0], [0, 3109, -1], [0, 6110, -1], [0, 3106, 69], [0, 3110, 'dark'], [0, 3101, 26.6], [0, 3102, 79.88], [0, 3115, 0], [0, 3111, 100], [0, 9102, ['button']]]}
2024-10-23 08:39:08.175 DEBUG (MainThread) [aioshelly.block_device.coap] Calling CoAP message update for device id 18E2F0
2024-10-23 08:39:08.302 DEBUG (MainThread) [aioshelly.block_device.device] host <shelly sensor host>: timeout error: DeviceConnectionTimeoutError(TimeoutError())
2024-10-23 08:39:08.302 DEBUG (MainThread) [homeassistant.components.shelly] Error connecting to Shelly device Shelly Door/Window 2 (SHDW-2), error: DeviceConnectionTimeoutError(TimeoutError())
2024-10-23 08:39:13.489 DEBUG (MainThread) [aioshelly.block_device.coap] CoapMessage: ip=127.0.0.1, type=CoapType.PERIODIC(30), options={11: b's', 3332: b'SHDW-2#18E2F0#2', 3412: b'\x96\x00', 3420: b'\x01\x00'}, payload={'G': [[0, 9103, 0], [0, 3108, 1], [0, 3119, 0], [0, 3109, -1], [0, 6110, -1], [0, 3106, 100], [0, 3110, 'dark'], [0, 3101, 26.7], [0, 3102, 80.06], [0, 3115, 0], [0, 3111, 100], [0, 9102, ['button']]]}
2024-10-23 08:39:13.489 DEBUG (MainThread) [aioshelly.block_device.coap] Calling CoAP message update for device id 18E2F0
2024-10-23 08:39:13.489 DEBUG (MainThread) [homeassistant.components.shelly] Shelly Shelly Door/Window 2 (SHDW-2) handle update, type: BlockUpdateType.ONLINE
2024-10-23 08:39:13.489 DEBUG (MainThread) [homeassistant.components.shelly] Connecting to Shelly Device - Shelly Door/Window 2 (SHDW-2)
2024-10-23 08:39:13.489 DEBUG (MainThread) [aioshelly.block_device.device] host <shelly sensor host>: block device initialize
2024-10-23 08:39:13.490 DEBUG (MainThread) [homeassistant.components.shelly] Manually updated Shelly Door/Window 2 (SHDW-2) data
2024-10-23 08:39:13.506 DEBUG (MainThread) [aioshelly.block_device.device] host <shelly sensor host>: http request: /settings (params=None)
2024-10-23 08:39:13.598 DEBUG (MainThread) [aioshelly.block_device.device] aiohttp response: {'device': {'type': 'SHDW-2', 'mac': 'C8C9A318E2F0', 'hostname': 'shellydw2-18E2F0', 'sleep_mode': True}, 'wifi_ap': {'enabled': False, 'ssid': 'shellydw2-18E2F0', 'key': ''}, 'wifi_sta': {'enabled': True, 'ssid': '<wifi 2.4G ssid>', 'ipv4_method': 'static', 'ip': '<shelly sensor host>', 'gw': None, 'mask': '255.255.255.0', 'dns': '8.8.8.8'}, 'wifi_sta1': {'enabled': False, 'ssid': None, 'ipv4_method': 'dhcp', 'ip': None, 'gw': None, 'mask': None, 'dns': None}, 'mqtt': {'enable': False, 'server': '<ha host>:1883', 'user': '', 'id': 'shellydw2-18E2F0', 'reconnect_timeout_max': 60.0, 'reconnect_timeout_min': 2.0, 'clean_session': True, 'keep_alive': 60, 'max_qos': 0, 'retain': True, 'update_period': 30}, 'coiot': {'enabled': True, 'update_period': 15, 'peer': '<ha host>:5683'}, 'sntp': {'server': 'at.pool.ntp.org', 'enabled': True}, 'login': {'enabled': False, 'unprotected': False, 'username': 'admin'}, 'pin_code': '', 'name': 'Closet Door', 'fw': '20231107-163707/v1.14.1-rc1-g0617c15', 'pon_wifi_reset': False, 'discoverable': True, 'build_info': {'build_id': '20231107-163707/v1.14.1-rc1-g0617c15', 'build_timestamp': '2023-11-07T16:37:07Z', 'build_version': '1.0'}, 'cloud': {'enabled': False, 'connected': False}, 'timezone': None, 'lat': 500.0, 'lng': 500.0, 'tzautodetect': True, 'tz_utc_offset': 0, 'tz_dst': False, 'tz_dst_auto': True, 'time': '', 'unixtime': 0, 'led_status_disable': False, 'debug_enable': False, 'allow_cross_origin': False, 'actions': {'active': False, 'names': ['report_url', 'dark_url', 'twilight_url', 'open_url', 'close_url', 'vibration_url', 'temp_over_url', 'temp_under_url']}, 'dark_threshold': 100, 'twilight_threshold': 300, 'sleep_mode': {'period': 12, 'unit': 'h'}, 'tilt_enabled': False, 'tilt_calibrated': False, 'vibration_enabled': False, 'vibration_sensitivity': 50, 'reverse_open_close': False, 'lux_wakeup_enable': True, 'sensors': {'temperature_threshold': 1.0, 'temperature_unit': 'C'}, 'temperature_offset': 0.0}
2024-10-23 08:39:13.599 DEBUG (MainThread) [aioshelly.block_device.device] host <shelly sensor host>: http request: /status (params=None)
2024-10-23 08:39:13.626 DEBUG (MainThread) [aioshelly.block_device.device] aiohttp response: {'wifi_sta': {'connected': True, 'ssid': '<wifi 2.4G ssid>', 'ip': '<shelly sensor host>', 'rssi': -40}, 'cloud': {'enabled': False, 'connected': False}, 'mqtt': {'connected': False}, 'time': '', 'unixtime': 0, 'serial': 1, 'has_update': False, 'mac': 'C8C9A318E2F0', 'cfg_changed_cnt': 0, 'actions_stats': {'skipped': 0}, 'is_valid': True, 'tmp': {'value': 26.7, 'units': 'C', 'tC': 26.7, 'tF': 80.06, 'is_valid': True}, 'lux': {'value': 100, 'illumination': 'dark', 'is_valid': True}, 'accel': {'tilt': -1, 'vibration': -1}, 'sensor': {'state': 'open', 'is_valid': True}, 'bat': {'value': 100, 'voltage': 5.97}, 'act_reasons': ['button'], 'sensor_error': 0, 'update': {'status': 'unknown', 'has_update': False, 'new_version': '', 'old_version': '20231107-163707/v1.14.1-rc1-g0617c15'}, 'ram_total': 52216, 'ram_free': 38596, 'fs_size': 233681, 'fs_free': 128010, 'uptime': 17}
2024-10-23 08:39:13.626 DEBUG (MainThread) [aioshelly.block_device.coap] Sending request 'cit/d' to device <shelly sensor host>
2024-10-23 08:39:18.799 DEBUG (MainThread) [aioshelly.block_device.coap] CoapMessage: ip=127.0.0.1, type=CoapType.PERIODIC(30), options={11: b's', 3332: b'SHDW-2#18E2F0#2', 3412: b'\x96\x00', 3420: b'\x01\x00'}, payload={'G': [[0, 9103, 0], [0, 3108, 1], [0, 3119, 0], [0, 3109, -1], [0, 6110, -1], [0, 3106, 77], [0, 3110, 'dark'], [0, 3101, 26.7], [0, 3102, 80.06], [0, 3115, 0], [0, 3111, 100], [0, 9102, ['button']]]}
2024-10-23 08:39:18.799 DEBUG (MainThread) [aioshelly.block_device.coap] Calling CoAP message update for device id 18E2F0

Additional information

Example UDP traffic

P�γcits��SHDW-2#18E2F0#2�C� � �{"G":[[0,9103,0],[0,3108,1],[0,3119,0],[0,3109,-1],[0,6110,-1],[0,3106,15],[0,3110,"dark"],[0,3101,24.90],[0,3102,76.82],[0,3115,0],[0,3111,100],[0,9102,["button"]]]}
P�ϳcits��SHDW-2#18E2F0#2�C� � �{"G":[[0,9103,0],[0,3108,1],[0,3119,0],[0,3109,-1],[0,6110,-1],[0,3106,0],[0,3110,"dark"],[0,3101,25.30],[0,3102,77.54],[0,3115,0],[0,3111,100],[0,9102,["button"]]]}
P�гcits��SHDW-2#18E2F0#2�C� � �{"G":[[0,9103,0],[0,3108,1],[0,3119,0],[0,3109,-1],[0,6110,-1],[0,3106,5],[0,3110,"dark"],[0,3101,26.00],[0,3102,78.80],[0,3115,0],[0,3111,100],[0,9102,["button"]]]}

uuf6429 commented 1 month ago

Off-topic sidenote, using a URL with a fragment/hash for the integration documentation field (I used https://www.home-assistant.io/integrations/shelly#shelly-device-configuration-generation-1 originally) seems to confuse/break @homeassistant, my bad.

So I'm pinging the codeowners manually: @balloob @bieniu @thecode @chemelli74 @bdraco

chemelli74 commented 1 month ago

Hi @uuf6429, do you mind describing your network structure ? For example, does HA have more than 1 network interface ?

One thing I noticed in your log is: 2024-10-23 08:38:58.278 DEBUG (MainThread) [aioshelly.block_device.coap] CoapMessage: ip=127.0.0.1, type=CoapType.PERIODIC(30), options={11: b's', 3332: b'SHDW-2#18E2F0#2', 3412: b'\x96\x00', 3420: b'\x01\x00'}, payload={'G': [[0, 9103, 0], [0, 3108, 1], [0, 3119, 0], [0, 3109, -1], [0, 6110, -1], [0, 3106, 31], [0, 3110, 'dark'], [0, 3101, 26.4], [0, 3102, 79.52], [0, 3115, 0], [0, 3111, 100], [0, 9102, ['button']]]}

I bet you have a reverse proxy somewhere in the game.

uuf6429 commented 1 month ago

Hi! I'm not overly familiar with how everything works when it comes to docker on windows - it is possible there is some reverse proxying somewhere... perhaps that's how "Host Networking" in "Docker Desktop" on Windows works.

the shelly sensor is connected to a router via 2.4ghz ssid
HA is running as a container in Docker Desktop on Windows 11, with full privileges and host networking.
this machine, that HA is running on, is connected to the same router via cable (eth)
they're all on the same subnet

I managed to add the debug log (that you saw) rather late in the troubleshooting, but seeing as there is some logging in HA triggered by the shelly sensor, I'm wondering what else HA would need and/or why it would need to connect back to the device. I don't know much about CoIoT to answer that.

home-assistant / core