search

mdeweerd / zha-toolkit

🧰 Zigbee Home Assistant Toolkit - service for "rare" Zigbee operations using ZHA on Home Assistant
GNU General Public License v3.0
171 stars 17 forks source link

Enable Array writes #210

Closed tomasbedrich closed 8 months ago

tomasbedrich commented 8 months ago

This PR adds support for ZCL Array type attributes to be written using zha_toolkit.attr_write.

tomasbedrich commented 8 months ago

Do you have more information about how the generic method fails?

service: zha_toolkit.attr_write
data:
  ieee: button.ubisys_c4_5504_identify
  endpoint: 232
  cluster: 64512
  attribute: 1
  attr_type: 0x48
  attr_val: "41040006000d0106000206010d0206000206020d0306000206030d04060002"
  read_before_write: true
  read_after_write: false
  use_cache: false
$ tail -f /config/home-assistant.log | grep "custom_components.zha_toolkit"
2023-10-17 13:11:26.669 INFO (MainThread) [custom_components.zha_toolkit] Running ZHA Toolkit service: <ServiceCall zha_toolkit.attr_write (c:01HCYNAQEB8VBEP9BNR4F4CTCA): ieee=button.ubisys_c4_5504_identify, endpoint=232, cluster=64512, attribute=1, attr_type=72, attr_val=41040006000d0106000206010d0206000206020d0306000206030d04060002, read_before_write=True, read_after_write=False, use_cache=False>
2023-10-17 13:11:26.670 DEBUG (MainThread) [custom_components.zha_toolkit] Got hass.data['zha']/gateway <custom_components.zha.core.gateway.ZHAGateway object at 0xffff9ada28d0>
2023-10-17 13:11:26.694 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_toolkit' from '/config/custom_components/zha_toolkit/__init__.py'>
2023-10-17 13:11:26.700 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Parameters '{'ieee': 'button.ubisys_c4_5504_identify', 'endpoint': 232, 'cluster': 64512, 'attribute': 1, 'attr_type': 72, 'attr_val': '41040006000d0106000206010d0206000206020d0306000206030d04060002', 'read_before_write': True, 'read_after_write': False, 'use_cache': False}'
2023-10-17 13:11:26.701 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Final manf 'None'
2023-10-17 13:11:26.702 DEBUG (MainThread) [custom_components.zha_toolkit.utils] registry_entity RegistryEntry(entity_id='button.ubisys_c4_5504_identify', unique_id='00:1f:ee:00:00:00:9a:e3-1-3', platform='zha', aliases=set(), area_id=None, capabilities=None, config_entry_id='5e33ebc13289867554f7364db82af185', device_class=None, device_id='c2520a704c74c0b5a187208e684b55fc', disabled_by=None, entity_category=<EntityCategory.DIAGNOSTIC: 'diagnostic'>, hidden_by=None, icon=None, id='80e47e521f856efd43f9a429ec9a1715', has_entity_name=True, name=None, options={'conversation': {'should_expose': False}}, original_device_class='identify', original_icon=None, original_name='Identify', supported_features=0, translation_key=None, unit_of_measurement=None)
2023-10-17 13:11:26.702 DEBUG (MainThread) [custom_components.zha_toolkit.utils] registry_device DeviceEntry(area_id=None, config_entries={'5e33ebc13289867554f7364db82af185'}, configuration_url=None, connections={('zigbee', '00:1f:ee:00:00:00:9a:e3')}, disabled_by=None, entry_type=None, hw_version=None, id='c2520a704c74c0b5a187208e684b55fc', identifiers={('zha', '00:1f:ee:00:00:00:9a:e3')}, manufacturer='ubisys', model='C4 (5504)', name_by_user=None, name='ubisys C4 (5504)', suggested_area=None, sw_version='0x02120404', via_device_id='34cca87150cffeda1beaafec3e4619c1', is_new=False)
2023-10-17 13:11:26.705 DEBUG (MainThread) [custom_components.zha_toolkit] 'ieee' parameter: 'button.ubisys_c4_5504_identify' -> IEEE Addr: '00:1f:ee:00:00:00:9a:e3'
2023-10-17 13:11:26.706 DEBUG (MainThread) [custom_components.zha_toolkit] Default handler for attr_write
2023-10-17 13:11:26.706 DEBUG (MainThread) [custom_components.zha_toolkit] Handler: <function command_handler_default at 0xffff9853c900>
2023-10-17 13:11:26.707 DEBUG (MainThread) [custom_components.zha_toolkit] running default command: <ServiceCall zha_toolkit.attr_write (c:01HCYNAQEB8VBEP9BNR4F4CTCA): ieee=button.ubisys_c4_5504_identify, endpoint=232, cluster=64512, attribute=1, attr_type=72, attr_val=41040006000d0106000206010d0206000206020d0306000206030d04060002, read_before_write=True, read_after_write=False, use_cache=False>
2023-10-17 13:11:26.708 DEBUG (MainThread) [custom_components.zha_toolkit.default] Trying to import custom_components.zha_toolkit.zcl_attr to call attr_write
2023-10-17 13:11:26.715 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Request attr read [1]
2023-10-17 13:11:26.760 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Reading attr result (attrs, status): ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])}, {})
2023-10-17 13:11:26.760 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Type determined from read: 0x48
2023-10-17 13:11:26.761 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Data type '<class 'zigpy.zcl.foundation.Array'>' for attr type 72
2023-10-17 13:11:26.761 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Converted 41040006000d0106000206010d0206000206020d0306000206030d04060002 to TypeValue(type=Array, value=Array(type=NoneType, value=None)) - will compare to Array(type=NoneType, value=None) - Type: 0x48
2023-10-17 13:11:26.761 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Request attr write [Attribute(attrid=0x0001, value=TypeValue(type=Array, value=Array(type=NoneType, value=None)))]
2023-10-17 13:11:26.762 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'zha_toolkit_version': 'v1.1.2', 'zigpy_version': '0.58.0.post2+git.80f9f8db', 'zigpy_rf_version': '0.35.9', 'ieee_org': 'button.ubisys_c4_5504_identify', 'ieee': '00:1f:ee:00:00:00:9a:e3', 'command': 'attr_write', 'command_data': None, 'start_time': '2023-10-17T11:11:26.705662+00:00', 'errors': ['AttributeError("\'str\' object has no attribute \'to_bytes\'")'], 'params': {'endpoint_id': 232, 'cluster_id': 64512, 'attr_id': 1, 'attr_type': 72, 'attr_val': '41040006000d0106000206010d0206000206020d0306000206030d04060002', 'dir': 0, 'tries': 1, 'expect_reply': True, 'args': [], 'read_before_write': True}, 'compare_val': Array(type=NoneType, value=None), 'attr_type': '0x48', 'write_is_equal': False, 'read_before': ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])}, {}), 'success': False}
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 468, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 704, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 666, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2012, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2049, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/__init__.py", line 816, in toolkit_service
    raise handler_exception
  File "/config/custom_components/zha_toolkit/__init__.py", line 780, in toolkit_service
    handler_result = await handler(
                     ^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/__init__.py", line 882, in command_handler_default
    return await default.default(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/default.py", line 40, in default
    await handler(app, listener, ieee, cmd, data, service, params, event_data)
  File "/config/custom_components/zha_toolkit/zcl_attr.py", line 462, in attr_write
    result_write = await u.cluster__write_attributes(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/utils.py", line 964, in cluster__write_attributes
    return await cluster._write_attributes(attrs, manufacturer=manufacturer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 357, in request
    hdr, request = self._create_request(
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 320, in _create_request
    request.serialize()  # Throw an error before generating a new TSN
    ^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/struct.py", line 250, in serialize
    chunks.append(value.serialize())
                  ^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/basic.py", line 862, in serialize
    return b"".join([self._item_type(i).serialize() for i in self])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/basic.py", line 862, in <listcomp>
    return b"".join([self._item_type(i).serialize() for i in self])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/struct.py", line 250, in serialize
    chunks.append(value.serialize())
                  ^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/foundation.py", line 102, in serialize
    return self.type.to_bytes(1, "little") + self.value.serialize()
                                             ^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/foundation.py", line 102, in serialize
    return self.type.to_bytes(1, "little") + self.value.serialize()
           ^^^^^^^^^^^^^^^^^^
AttributeError: 'str' object has no attribute 'to_bytes'

Or another way:

service: zha_toolkit.attr_write
data:
  ieee: button.ubisys_c4_5504_identify
  endpoint: 232
  cluster: 64512
  attribute: 1
  attr_type: 0x48
  attr_val: [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2]
  read_before_write: true
  read_after_write: false
  use_cache: false
2023-10-17 13:14:53.801 INFO (MainThread) [custom_components.zha_toolkit] Running ZHA Toolkit service: <ServiceCall zha_toolkit.attr_write (c:01HCYNH1Q6W2EYASQ7CC2ERVA8): ieee=button.ubisys_c4_5504_identify, endpoint=232, cluster=64512, attribute=1, attr_type=72, attr_val=[65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], read_before_write=True, read_after_write=False, use_cache=False>
2023-10-17 13:14:53.801 DEBUG (MainThread) [custom_components.zha_toolkit] Got hass.data['zha']/gateway <custom_components.zha.core.gateway.ZHAGateway object at 0xffff9ada28d0>
2023-10-17 13:14:53.826 DEBUG (MainThread) [custom_components.zha_toolkit] module is <module 'custom_components.zha_toolkit' from '/config/custom_components/zha_toolkit/__init__.py'>
2023-10-17 13:14:53.832 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Parameters '{'ieee': 'button.ubisys_c4_5504_identify', 'endpoint': 232, 'cluster': 64512, 'attribute': 1, 'attr_type': 72, 'attr_val': [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], 'read_before_write': True, 'read_after_write': False, 'use_cache': False}'
2023-10-17 13:14:53.832 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Final manf 'None'
2023-10-17 13:14:53.833 DEBUG (MainThread) [custom_components.zha_toolkit.utils] registry_entity RegistryEntry(entity_id='button.ubisys_c4_5504_identify', unique_id='00:1f:ee:00:00:00:9a:e3-1-3', platform='zha', aliases=set(), area_id=None, capabilities=None, config_entry_id='5e33ebc13289867554f7364db82af185', device_class=None, device_id='c2520a704c74c0b5a187208e684b55fc', disabled_by=None, entity_category=<EntityCategory.DIAGNOSTIC: 'diagnostic'>, hidden_by=None, icon=None, id='80e47e521f856efd43f9a429ec9a1715', has_entity_name=True, name=None, options={'conversation': {'should_expose': False}}, original_device_class='identify', original_icon=None, original_name='Identify', supported_features=0, translation_key=None, unit_of_measurement=None)
2023-10-17 13:14:53.833 DEBUG (MainThread) [custom_components.zha_toolkit.utils] registry_device DeviceEntry(area_id=None, config_entries={'5e33ebc13289867554f7364db82af185'}, configuration_url=None, connections={('zigbee', '00:1f:ee:00:00:00:9a:e3')}, disabled_by=None, entry_type=None, hw_version=None, id='c2520a704c74c0b5a187208e684b55fc', identifiers={('zha', '00:1f:ee:00:00:00:9a:e3')}, manufacturer='ubisys', model='C4 (5504)', name_by_user=None, name='ubisys C4 (5504)', suggested_area=None, sw_version='0x02120404', via_device_id='34cca87150cffeda1beaafec3e4619c1', is_new=False)
2023-10-17 13:14:53.835 DEBUG (MainThread) [custom_components.zha_toolkit] 'ieee' parameter: 'button.ubisys_c4_5504_identify' -> IEEE Addr: '00:1f:ee:00:00:00:9a:e3'
2023-10-17 13:14:53.835 DEBUG (MainThread) [custom_components.zha_toolkit] Default handler for attr_write
2023-10-17 13:14:53.836 DEBUG (MainThread) [custom_components.zha_toolkit] Handler: <function command_handler_default at 0xffff9362cf40>
2023-10-17 13:14:53.836 DEBUG (MainThread) [custom_components.zha_toolkit] running default command: <ServiceCall zha_toolkit.attr_write (c:01HCYNH1Q6W2EYASQ7CC2ERVA8): ieee=button.ubisys_c4_5504_identify, endpoint=232, cluster=64512, attribute=1, attr_type=72, attr_val=[65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], read_before_write=True, read_after_write=False, use_cache=False>
2023-10-17 13:14:53.838 DEBUG (MainThread) [custom_components.zha_toolkit.default] Trying to import custom_components.zha_toolkit.zcl_attr to call attr_write
2023-10-17 13:14:53.842 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Request attr read [1]
2023-10-17 13:14:53.884 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Reading attr result (attrs, status): ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])}, {})
2023-10-17 13:14:53.884 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Type determined from read: 0x48
2023-10-17 13:14:53.884 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Data type '<class 'zigpy.zcl.foundation.Array'>' for attr type 72
2023-10-17 13:14:53.884 DEBUG (MainThread) [custom_components.zha_toolkit.utils] Converted [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2] to TypeValue(type=Array, value=Array(type=NoneType, value=None)) - will compare to Array(type=NoneType, value=None) - Type: 0x48
2023-10-17 13:14:53.885 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Request attr write [Attribute(attrid=0x0001, value=TypeValue(type=Array, value=Array(type=NoneType, value=None)))]
2023-10-17 13:14:53.886 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'zha_toolkit_version': 'v1.1.2', 'zigpy_version': '0.58.0.post2+git.80f9f8db', 'zigpy_rf_version': '0.35.9', 'ieee_org': 'button.ubisys_c4_5504_identify', 'ieee': '00:1f:ee:00:00:00:9a:e3', 'command': 'attr_write', 'command_data': None, 'start_time': '2023-10-17T11:14:53.835470+00:00', 'errors': ['AttributeError("\'list\' object has no attribute \'to_bytes\'")'], 'params': {'endpoint_id': 232, 'cluster_id': 64512, 'attr_id': 1, 'attr_type': 72, 'attr_val': [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], 'dir': 0, 'tries': 1, 'expect_reply': True, 'args': [], 'read_before_write': True}, 'compare_val': Array(type=NoneType, value=None), 'attr_type': '0x48', 'write_is_equal': False, 'read_before': ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])}, {}), 'success': False}
Traceback (most recent call last):
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 468, in _async_step
    await getattr(self, handler)()
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 704, in _async_call_service_step
    response_data = await self._async_run_long_action(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/helpers/script.py", line 666, in _async_run_long_action
    return long_task.result()
           ^^^^^^^^^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2012, in async_call
    response_data = await coro
                    ^^^^^^^^^^
  File "/usr/src/homeassistant/homeassistant/core.py", line 2049, in _execute_service
    return await target(service_call)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/__init__.py", line 816, in toolkit_service
    raise handler_exception
  File "/config/custom_components/zha_toolkit/__init__.py", line 780, in toolkit_service
    handler_result = await handler(
                     ^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/__init__.py", line 882, in command_handler_default
    return await default.default(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/default.py", line 40, in default
    await handler(app, listener, ieee, cmd, data, service, params, event_data)
  File "/config/custom_components/zha_toolkit/zcl_attr.py", line 462, in attr_write
    result_write = await u.cluster__write_attributes(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/custom_components/zha_toolkit/utils.py", line 964, in cluster__write_attributes
    return await cluster._write_attributes(attrs, manufacturer=manufacturer)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 357, in request
    hdr, request = self._create_request(
                   ^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/__init__.py", line 320, in _create_request
    request.serialize()  # Throw an error before generating a new TSN
    ^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/struct.py", line 250, in serialize
    chunks.append(value.serialize())
                  ^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/basic.py", line 862, in serialize
    return b"".join([self._item_type(i).serialize() for i in self])
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/basic.py", line 862, in <listcomp>
    return b"".join([self._item_type(i).serialize() for i in self])
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/types/struct.py", line 250, in serialize
    chunks.append(value.serialize())
                  ^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/foundation.py", line 102, in serialize
    return self.type.to_bytes(1, "little") + self.value.serialize()
                                             ^^^^^^^^^^^^^^^^^^^^^^
  File "/config/deps/lib/python3.11/site-packages/zigpy/zcl/foundation.py", line 102, in serialize
    return self.type.to_bytes(1, "little") + self.value.serialize()
           ^^^^^^^^^^^^^^^^^^
AttributeError: 'list' object has no attribute 'to_bytes'

Maybe I am just passing attr_value in a wrong way?

tomasbedrich commented 8 months ago

I believe the most important thing is that ZCL Arrays are typed collections. The generic encoding logic only creates an Array object with a single parameter, while Array needs two parameters to be constructed - item type, contents. This can be observed in logs, search for: "Converted "

mdeweerd commented 8 months ago

Maybe I am just passing attr_value in a wrong way?

I realized that I can make writes to attributes that do not exist, so I updated the code to fix it so that it works for arrays. I "disabled" the code you added for 0x48 specifically by comparing to 0xFF48 so that it is not lost at this time.

When I tried my code using the following:.

  attr_val: [4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2]

I got:

    Attribute Field
        Attribute: Unknown (0xfde8)
        Data Type: Array (0x48)
        Elements Type: 56-Bit Bitmap (0x1e)
        Elements Number: 4
        Element #1, Bitmap: 020006010d0006
            Bitmap56: 0x00020006010d0006
        Element #2, Bitmap: 020006020d0106
            Bitmap56: 0x00020006020d0106
        Element #3, Bitmap: 020006030d0206
            Bitmap56: 0x00020006030d0206
        Element #4, Bitmap: 020006040d0306
            Bitmap56: 0x00020006040d0306

Frame (82 bytes):
0000  61 88 11 0a 88 d4 21 00 00 48 02 d4 21 00 00 1e   a.....!..H..!...
0010  f9 28 20 3d df 03 0c 41 6a 01 00 4b 12 00 00 60   .( =...Aj..K...`
0020  bf 63 ac 11 66 0e 6b 43 91 ca 88 be 76 36 33 b8   .c..f.kC....v63.
0030  28 15 91 ca 01 58 85 5c 89 db 75 a4 d8 c0 87 dd   (....X.\..u.....
0040  cd 9e 9e 75 37 9a 7c 3f 53 53 57 53 a6 ca d1 c0   ...u7.|?SSWS....
0050  0a eb                                             ..
Decrypted ZigBee Payload (45 bytes):
0000  40 01 00 00 04 01 01 ad 00 51 02 e8 fd 48 1e 04   @........Q...H..
0010  00 06 00 0d 01 06 00 02 06 01 0d 02 06 00 02 06   ................
0020  02 0d 03 06 00 02 06 03 0d 04 06 00 02            .............

However, when using your code, I got:

ZigBee Cluster Library Frame, Command: Write Attributes, Seq: 40
    Frame Control Field: Profile-wide (0x00)
        .... ..00 = Frame Type: Profile-wide (0x0)
        .... .0.. = Manufacturer Specific: False
        .... 0... = Direction: Client to Server
        ...0 .... = Disable Default Response: False
    Sequence Number: 40
    Command: Write Attributes (0x02)
    Attribute Field
        Attribute: Unknown (0xfde8)
        Data Type: Array (0x48)
        Elements Type: Octet String (0x41)
        Elements Number: 4
        Element #1, Octets: 00:0d:01:06:00:02
            Octet String: 00:0d:01:06:00:02
        Element #2, Octets: 01:0d:02:06:00:02
            Octet String: 01:0d:02:06:00:02
        Element #3, Octets: 02:0d:03:06:00:02
            Octet String: 02:0d:03:06:00:02
        Element #4, Octets: 03:0d:04:06:00:02
            Octet String: 03:0d:04:06:00:02

Decrypted ZigBee Payload (45 bytes):
0000  40 01 00 00 04 01 01 ff 00 28 02 e8 fd 48 41 04   @........(...HA.
0010  00 06 00 0d 01 06 00 02 06 01 0d 02 06 00 02 06   ................
0020  02 0d 03 06 00 02 06 03 0d 04 06 00 02            .............

So the latter is better for arrays. However, the questions is:

So the generic method is now better at handling lists, but Arrays still need something specific.

I invite you to verify the "compare_val" as I can try it on a device.

tomasbedrich commented 8 months ago

Anything else I can help w.r.t. this PR?

mdeweerd commented 8 months ago

Anything else I can help w.r.t. this PR?

Yes, I invited you to verify the "compare_val" as I can not try it myself on a device.

tomasbedrich commented 8 months ago

With version https://github.com/mdeweerd/zha-toolkit/pull/210/commits/561e95e3bbd813530c14f4c9c29678ef844b4cd0, I have this result:

service: zha_toolkit.attr_write
data:
  ieee: button.ubisys_c4_5504_identify
  endpoint: 232
  cluster: 64512
  attribute: 1
  attr_type: 0x48
  attr_val: [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2]
  read_before_write: false
  read_after_write: true
  use_cache: false

res:

zha_toolkit_version: v1.1.2
zigpy_version: 0.58.0.post2+git.80f9f8db
zigpy_rf_version: 0.35.9
ieee_org: button.ubisys_c4_5504_identify
ieee: 00:1f:ee:00:00:00:9a:e3
command: attr_write
command_data: null
start_time: "2023-10-18T09:19:18.987042+00:00"
errors: []
params:
  endpoint_id: 232
  cluster_id: 64512
  attr_id: 1
  attr_type: 72
  attr_val:
    - 65
    - 4
    - 0
    - 6
    - 0
    - 13
    - 1
    - 6
    - 0
    - 2
    - 6
    - 1
    - 13
    - 2
    - 6
    - 0
    - 2
    - 6
    - 2
    - 13
    - 3
    - 6
    - 0
    - 2
    - 6
    - 3
    - 13
    - 4
    - 6
    - 0
    - 2
  dir: 0
  tries: 1
  expect_reply: true
  args: []
  read_after_write: true
compare_val:
  - 65
  - 4
  - 0
  - 6
  - 0
  - 13
  - 1
  - 6
  - 0
  - 2
  - 6
  - 1
  - 13
  - 2
  - 6
  - 0
  - 2
  - 6
  - 2
  - 13
  - 3
  - 6
  - 0
  - 2
  - 6
  - 3
  - 13
  - 4
  - 6
  - 0
  - 2
attr_type: "0x48"
write_is_equal: false
result_write:
  - - status: 0
      attrid: null
result_read: >-
  ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02',
  b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02',
  b'\x03\r\x04\x06\x00\x02'])}, {})
success: false

While the compare_val equals to what was written, the result read is different because zigpy parses the Array structure. Based on that, zha_toolkit returns write_is_equal: false, which is a bit misleading. This is why I originally returned compare_val = None for this raw array write.

mdeweerd commented 8 months ago

Ok, zigpy returns the list but I'ld expecte the serialised value is compared, except that the structure is reported.

I've updated the code to report the serialized values when the match fails. So based on that we can update the "compare_val" to what would actually match when the read succeeds. (That code change is untested, so might need some adjustments).

https://github.com/mdeweerd/zha-toolkit/pull/210/commits/28e92d4c31b9b9235030c622ed5cb1c1b667de4b#diff-3cff7aaa1d7e2385a2d66097c78d84cf4bd88f08e74e77ddcddb1158b6f88376R498-R508

tomasbedrich commented 8 months ago

I rebased the branch to incorporate another fixes from main and ran another series of tests. Now it looks like:

service: zha_toolkit.attr_write
data:
  ieee: button.ubisys_c4_5504_identify
  endpoint: 232
  cluster: 64512
  attribute: 1
  attr_type: 0x48
  attr_val: [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2]
  read_before_write: false
  read_after_write: true
  use_cache: false

res:

zha_toolkit_version: v1.1.2
zigpy_version: 0.58.0.post2+git.80f9f8db
zigpy_rf_version: 0.35.9
ieee_org: button.ubisys_c4_5504_identify
ieee: 00:1f:ee:00:00:00:9a:e3
command: attr_write
command_data: null
start_time: "2023-10-19T08:52:32.396584+00:00"
errors: []
params:
  endpoint_id: 232
  cluster_id: 64512
  attr_id: 1
  attr_type: 72
  attr_val:
    - 65
    - 4
    - 0
    - 6
    - 0
    - 13
    - 1
    - 6
    - 0
    - 2
    - 6
    - 1
    - 13
    - 2
    - 6
    - 0
    - 2
    - 6
    - 2
    - 13
    - 3
    - 6
    - 0
    - 2
    - 6
    - 3
    - 13
    - 4
    - 6
    - 0
    - 2
  dir: 0
  tries: 1
  expect_reply: true
  args: []
  read_after_write: true
compare_val:
  - 65
  - 4
  - 0
  - 6
  - 0
  - 13
  - 1
  - 6
  - 0
  - 2
  - 6
  - 1
  - 13
  - 2
  - 6
  - 0
  - 2
  - 6
  - 2
  - 13
  - 3
  - 6
  - 0
  - 2
  - 6
  - 3
  - 13
  - 4
  - 6
  - 0
  - 2
attr_type: "0x48"
write_is_equal: false
result_write:
  - - status: 0
      attrid: null
warnings:
  - >-
    Read does not match expected:
    b'A\x04\x00\x06\x00\r\x01\x06\x00\x02\x06\x01\r\x02\x06\x00\x02\x06\x02\r\x03\x06\x00\x02\x06\x03\r\x04\x06\x00\x02'
    <>
    b'A\x04\x00\x06\x00\r\x01\x06\x00\x02\x06\x01\r\x02\x06\x00\x02\x06\x02\r\x03\x06\x00\x02\x06\x03\r\x04\x06\x00\x02'
result_read: >-
  ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02',
  b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02',
  b'\x03\r\x04\x06\x00\x02'])}, {})
success: false
mdeweerd commented 8 months ago

Thank you. I see that the serialized data matches, so this is what needs to be compared - currently that is not the case. You can update the code for that and test or I'll do it later (but I can not test). The serialisation is done when generating the "Read does not match expected" message (added to see if that would match).

mdeweerd commented 8 months ago

I've updated the code to compare serialised values before and after the write.

I also enabled reading Arrays in scan_device - I think I disabled it because I had issues with, now fixed in zigpy.

Can you test this on your device. scan_device might not result in an array read if your device does not support "scanning" that attribute.

tomasbedrich commented 8 months ago

As for the scanning, I cannot see any attributes, not even for standard clusters, so I cannot evaluate if this improved the behaviour for Arrays or not:

service: zha_toolkit.scan_device
data:
  ieee: button.ubisys_c4_5504_identify
  endpoint: [1, 232]

the output is following:

zha_toolkit_version: v1.1.2
zigpy_version: 0.58.0.post2+git.80f9f8db
zigpy_rf_version: 0.35.9
ieee_org: button.ubisys_c4_5504_identify
ieee: 00:1f:ee:00:00:00:9a:e3
command: scan_device
command_data: null
start_time: "2023-10-24T21:49:17.305585+00:00"
errors: []
params:
  endpoint_id:
    - 1
    - 232
  dir: 0
  tries: 1
  expect_reply: true
  args: []
  read_before_write: true
  read_after_write: true
scan:
  ieee: 00:1f:ee:00:00:00:9a:e3
  nwk: "0xcb79"
  model: C4 (5504)
  manufacturer: ubisys
  manufacturer_id: "0x4338"
  endpoints:
    - id: 1
      device_type: "0x0104"
      profile: "0x0104"
      in_clusters:
        "0x0000":
          cluster_id: "0x0000"
          title: Basic
          name: basic
          attributes: {}
          commands_received: {}
          commands_generated: {}
        "0x0003":
          cluster_id: "0x0003"
          title: Identify
          name: identify
          attributes: {}
          commands_received: {}
          commands_generated: {}
      out_clusters:
        "0x0005":
          cluster_id: "0x0005"
          title: Scenes
          name: scenes
          attributes: {}
          commands_received: {}
          commands_generated: {}
        "0x0006":
          cluster_id: "0x0006"
          title: On/Off
          name: on_off
          attributes: {}
          commands_received: {}
          commands_generated: {}
        "0x0008":
          cluster_id: "0x0008"
          title: Level control
          name: level
          attributes: {}
          commands_received: {}
          commands_generated: {}
        "0x0300":
          cluster_id: "0x0300"
          title: Color Control
          name: light_color
          attributes: {}
          commands_received: {}
          commands_generated: {}
    - id: 232
      device_type: "0x0507"
      profile: "0x0104"
      in_clusters:
        "0x0000":
          cluster_id: "0x0000"
          title: Basic
          name: basic
          attributes: {}
          commands_received: {}
          commands_generated: {}
        "0xfc00":
          cluster_id: "0xfc00"
          title: Manufacturer Specific
          name: manufacturer_specific
          attributes: {}
          commands_received: {}
          commands_generated: {}
      out_clusters:
        "0x0003":
          cluster_id: "0x0003"
          title: Identify
          name: identify
          attributes: {}
          commands_received: {}
          commands_generated: {}
        "0x0019":
          cluster_id: "0x0019"
          title: Ota
          name: ota
          attributes: {}
          commands_received: {}
          commands_generated: {}
success: true
tomasbedrich commented 8 months ago

As for the write:

service: zha_toolkit.attr_write
data:
  ieee: button.ubisys_c4_5504_identify
  endpoint: 232
  cluster: 64512
  attribute: 1
  attr_type: 0x48
  attr_val: [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2]

This works okay as long as attr_val is the same as the actual value read:

zha_toolkit_version: v1.1.2
zigpy_version: 0.58.0.post2+git.80f9f8db
zigpy_rf_version: 0.35.9
ieee_org: button.ubisys_c4_5504_identify
ieee: 00:1f:ee:00:00:00:9a:e3
command: attr_write
command_data: null
start_time: "2023-10-24T21:56:22.879429+00:00"
errors: []
params:
  endpoint_id: 232
  cluster_id: 64512
  attr_id: 1
  attr_type: 72
  attr_val:
    - 65
    - 4
    - 0
    - 6
    - 0
    - 13
    - 1
    - 6
    - 0
    - 2
    - 6
    - 1
    - 13
    - 2
    - 6
    - 0
    - 2
    - 6
    - 2
    - 13
    - 3
    - 6
    - 0
    - 2
    - 6
    - 3
    - 13
    - 4
    - 6
    - 0
    - 2
  dir: 0
  tries: 1
  expect_reply: true
  args: []
  read_before_write: true
  read_after_write: true
compare_val:
  - 65
  - 4
  - 0
  - 6
  - 0
  - 13
  - 1
  - 6
  - 0
  - 2
  - 6
  - 1
  - 13
  - 2
  - 6
  - 0
  - 2
  - 6
  - 2
  - 13
  - 3
  - 6
  - 0
  - 2
  - 6
  - 3
  - 13
  - 4
  - 6
  - 0
  - 2
attr_type: "0x48"
write_is_equal: true
info: Data read is equal to data to write
result_read: >-
  ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02',
  b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02',
  b'\x03\r\x04\x06\x00\x02'])}, {})
success: true

But, when I change the contents of attr_val (only a single byte), I get the following error back in HA logs:

2023-10-24 23:54:52.823 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Reading attr result (attrs, status): ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x02\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])}, {})
2023-10-24 23:54:52.831 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'zha_toolkit_version': 'v1.1.2', 'zigpy_version': '0.58.0.post2+git.80f9f8db', 'zigpy_rf_version': '0.35.9', 'ieee_org': 'button.ubisys_c4_5504_identify', 'ieee': '00:1f:ee:00:00:00:9a:e3', 'command': 'attr_write', 'command_data': None, 'start_time': '2023-10-24T21:54:52.659660+00:00', 'errors': [], 'params': {'endpoint_id': 232, 'cluster_id': 64512, 'attr_id': 1, 'attr_type': 72, 'attr_val': [65, 4, 0, 6, 0, 13, 2, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], 'dir': 0, 'tries': 1, 'expect_reply': True, 'args': [], 'read_before_write': True, 'read_after_write': True}, 'compare_val': [65, 4, 0, 6, 0, 13, 2, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], 'attr_type': '0x48', 'write_is_equal': False, 'read_before': ({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])}, {}), 'result_write': Write_Attributes_rsp(status_records=[WriteAttributesStatusRecord(status=<Status.SUCCESS: 0>)]), 'result_read': "({1: Array(type=AnonymousLVList, value=[b'\\x00\\r\\x02\\x06\\x00\\x02', b'\\x01\\r\\x02\\x06\\x00\\x02', b'\\x02\\r\\x03\\x06\\x00\\x02', b'\\x03\\r\\x04\\x06\\x00\\x02'])}, {})", 'success': True}
2023-10-24 23:54:52.834 ERROR (MainThread) [homeassistant.components.websocket_api.messages] Unable to serialize to JSON. Bad data found at $.result.response.read_before=({1: Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])}, {})(<class 'tuple'>
mdeweerd commented 8 months ago

There was some code to ensure that "result_read" could be converted to json, but not "read_before". I fixed that as well as some regressions for scalar values.

tomasbedrich commented 8 months ago

Not there yet:

2023-10-25 09:03:37.801 DEBUG (MainThread) [custom_components.zha_toolkit.zcl_attr] Type determined from read: 0x48
2023-10-25 09:03:37.802 ERROR (MainThread) [custom_components.zha_toolkit.utils] Can't convert to JSON Array(type=AnonymousLVList, value=[b'\x00\r\x01\x06\x00\x02', b'\x01\r\x02\x06\x00\x02', b'\x02\r\x03\x06\x00\x02', b'\x03\r\x04\x06\x00\x02'])
2023-10-25 09:03:37.803 DEBUG (MainThread) [custom_components.zha_toolkit] event_data {'zha_toolkit_version': 'v1.1.2', 'zigpy_version': '0.58.0.post2+git.80f9f8db', 'zigpy_rf_version': '0.35.9', 'ieee_org': 'button.ubisys_c4_5504_identify', 'ieee': '00:1f:ee:00:00:00:9a:e3', 'command': 'attr_write', 'command_data': None, 'start_time': '2023-10-25T07:03:37.717141+00:00', 'errors': [], 'params': {'endpoint_id': 232, 'cluster_id': 64512, 'attr_id': 1, 'attr_type': 72, 'attr_val': [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], 'dir': 0, 'tries': 1, 'expect_reply': True, 'args': [], 'read_before_write': True, 'read_after_write': True}, 'compare_val': [65, 4, 0, 6, 0, 13, 1, 6, 0, 2, 6, 1, 13, 2, 6, 0, 2, 6, 2, 13, 3, 6, 0, 2, 6, 3, 13, 4, 6, 0, 2], 'attr_type': '0x48', 'write_is_equal': True, 'info': 'Data read is equal to data to write', 'result_read': ({1: b'A\x04\x00\x06\x00\r\x01\x06\x00\x02\x06\x01\r\x02\x06\x00\x02\x06\x02\r\x03\x06\x00\x02\x06\x03\r\x04\x06\x00\x02'}, {}), 'success': True}
2023-10-25 09:03:37.805 ERROR (MainThread) [homeassistant.components.websocket_api.messages] Unable to serialize to JSON. Bad data found at $.result.response.result_read=({1: b'A\x04\x00\x06\x00\r\x01\x06\x00\x02\x06\x01\r\x02\x06\x00\x02\x06\x02\r\x03\x06\x00\x02\x06\x03\r\x04\x06\x00\x02'}, {})(<class 'tuple'>
mdeweerd commented 8 months ago

I didn't expect bytes to not be json encodable. I've updated it.

tomasbedrich commented 8 months ago

The current state now works on my side:

zha_toolkit_version: v1.1.2
zigpy_version: 0.58.0.post2+git.80f9f8db
zigpy_rf_version: 0.35.9
ieee_org: button.ubisys_c4_5504_identify
ieee: 00:1f:ee:00:00:00:9a:e3
command: attr_write
command_data: null
start_time: "2023-10-26T23:19:39.655990+00:00"
errors: []
params:
  endpoint_id: 232
  cluster_id: 64512
  attr_id: 1
  attr_type: 72
  attr_val:
    - 65
    - 4
    - 0
    - 6
    - 0
    - 13
    - 1
    - 6
    - 0
    - 2
    - 6
    - 1
    - 13
    - 2
    - 6
    - 0
    - 2
    - 6
    - 2
    - 13
    - 3
    - 6
    - 0
    - 2
    - 6
    - 3
    - 13
    - 4
    - 6
    - 0
    - 2
  dir: 0
  tries: 1
  expect_reply: true
  args: []
  read_before_write: true
  read_after_write: true
compare_val:
  - 65
  - 4
  - 0
  - 6
  - 0
  - 13
  - 1
  - 6
  - 0
  - 2
  - 6
  - 1
  - 13
  - 2
  - 6
  - 0
  - 2
  - 6
  - 2
  - 13
  - 3
  - 6
  - 0
  - 2
  - 6
  - 3
  - 13
  - 4
  - 6
  - 0
  - 2
attr_type: "0x48"
write_is_equal: false
read_before:
  - "1": 41040006000d0206000206010d0206000206020d0306000206030d04060002
  - {}
result_write:
  - - status: 0
      attrid: null
result_read:
  - "1": 41040006000d0106000206010d0206000206020d0306000206030d04060002
  - {}
success: true
mdeweerd commented 8 months ago

Good news! I can see why the order has to change, but I'ld like to keep the reported values as strings rather than conversions to hexadecimal values. The string value can be reused as the input for another write.

tomasbedrich commented 8 months ago

The string value can be reused as the input for another write.

That's exactly why I added 5ba046e just now :)

tomasbedrich commented 8 months ago

Up to you, if you think it needs more changes, please go ahead.

mdeweerd commented 8 months ago

Well, providing the data as "hex" was not really the logic up to now, so I'm changing that back to the binary string type (or whatever that is called).

mdeweerd commented 8 months ago

Merged & pre-released.

tomasbedrich commented 8 months ago

I can confirm the pre-release works. Thanks.