zigpy / zha-device-handlers

ZHA device handlers bridge the functionality gap created when manufacturers deviate from the ZCL specification, handling deviations and exceptions by parsing custom messages to and from Zigbee devices.
Apache License 2.0
684 stars 636 forks source link

[Device Support Request] - TS0601 by _TZE200_v1jqz5cy - Chlorine Meter PH ORP EC TDS Salinity Temp CL Tester Swimming Pool Water Quality Analyzer #2565

Open KirkKirk opened 10 months ago

KirkKirk commented 10 months ago

Problem description

Not supported device in ZHA

Smart WiFi Zigbee Chlorine Meter PH ORP EC TDS Salinity Temp CL Tester Swimming Pool Water Quality Analyzer

https://www.aliexpress.com/item/1005005575336871.html?spm=a2g0o.order_list.order_list_main.11.28e31802Wxn6Wl

Solution description

Add it to the supported device list in ZHA Please.

Screenshots/Video

Screenshots/Video [![image](https://github.com/zigpy/zha-device-handlers/assets/30799136/1d0bfbdc-8897-4c2f-bce7-a6ed9012923e)]

Device signature

Device signature ```json [{ "node_descriptor": "NodeDescriptor(logical_type=, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=, mac_capability_flags=, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=True, *is_full_function_device=False, *is_mains_powered=False, *is_receiver_on_when_idle=False, *is_router=False, *is_security_capable=False)", "endpoints": { "1": { "profile_id": "0x0104", "device_type": "0x0051", "input_clusters": [ "0x0000", "0x0004", "0x0005", "0xef00" ], "output_clusters": [ "0x000a", "0x0019" ] } }, "manufacturer": "_TZE200_v1jqz5cy", "model": "TS0601", "class": "zigpy.device.Device" }] ```

Diagnostic information

Diagnostic information ```json [{ "home_assistant": { "installation_type": "Home Assistant OS", "version": "2023.8.4", "dev": false, "hassio": true, "virtualenv": false, "python_version": "3.11.4", "docker": true, "arch": "aarch64", "timezone": "Australia/Sydney", "os_name": "Linux", "os_version": "6.1.21-v8", "supervisor": "2023.08.1", "host_os": "Home Assistant OS 10.5", "docker_version": "23.0.6", "chassis": "embedded", "run_as_root": true }, "custom_components": { "nodered": { "version": "2.2.0", "requirements": [] }, "smartthings": { "version": "1.0", "requirements": [ "pysmartapp==0.3.3", "pysmartthings==0.7.6" ] }, "dohome": { "version": "0.2.0", "requirements": [] }, "braviatv_psk": { "version": "0.4.2", "requirements": [ "pySonyBraviaPSK==0.2.4" ] }, "hacs": { "version": "1.32.1", "requirements": [ "aiogithubapi>=22.10.1" ] }, "truenas": { "version": "0.0.0", "requirements": [] }, "tesla_custom": { "version": "3.16.1", "requirements": [ "teslajsonpy==3.9.3" ] }, "sonoff": { "version": "3.5.2", "requirements": [ "pycryptodome>=3.6.6" ] }, "localtuya": { "version": "5.2.1", "requirements": [] }, "roborock": { "version": "1.0.11", "requirements": [ "python-roborock==0.32.3", "ical==5.0.0" ] } }, "integration_manifest": { "domain": "zha", "name": "Zigbee Home Automation", "after_dependencies": [ "onboarding", "usb" ], "codeowners": [ "@dmulcahey", "@adminiuga", "@puddly" ], "config_flow": true, "dependencies": [ "file_upload" ], "documentation": "https://www.home-assistant.io/integrations/zha", "iot_class": "local_polling", "loggers": [ "aiosqlite", "bellows", "crccheck", "pure_pcapy3", "zhaquirks", "zigpy", "zigpy_deconz", "zigpy_xbee", "zigpy_zigate", "zigpy_znp" ], "requirements": [ "bellows==0.35.9", "pyserial==3.5", "pyserial-asyncio==0.6", "zha-quirks==0.0.102", "zigpy-deconz==0.21.0", "zigpy==0.56.4", "zigpy-xbee==0.18.1", "zigpy-zigate==0.11.0", "zigpy-znp==0.11.4" ], "usb": [ { "vid": "10C4", "pid": "EA60", "description": "*2652*", "known_devices": [ "slae.sh cc2652rb stick" ] }, { "vid": "1A86", "pid": "55D4", "description": "*sonoff*plus*", "known_devices": [ "sonoff zigbee dongle plus v2" ] }, { "vid": "10C4", "pid": "EA60", "description": "*sonoff*plus*", "known_devices": [ "sonoff zigbee dongle plus" ] }, { "vid": "10C4", "pid": "EA60", "description": "*tubeszb*", "known_devices": [ "TubesZB Coordinator" ] }, { "vid": "1A86", "pid": "7523", "description": "*tubeszb*", "known_devices": [ "TubesZB Coordinator" ] }, { "vid": "1A86", "pid": "7523", "description": "*zigstar*", "known_devices": [ "ZigStar Coordinators" ] }, { "vid": "1CF1", "pid": "0030", "description": "*conbee*", "known_devices": [ "Conbee II" ] }, { "vid": "10C4", "pid": "8A2A", "description": "*zigbee*", "known_devices": [ "Nortek HUSBZB-1" ] }, { "vid": "0403", "pid": "6015", "description": "*zigate*", "known_devices": [ "ZiGate+" ] }, { "vid": "10C4", "pid": "EA60", "description": "*zigate*", "known_devices": [ "ZiGate" ] }, { "vid": "10C4", "pid": "8B34", "description": "*bv 2010/10*", "known_devices": [ "Bitron Video AV2010/10" ] } ], "zeroconf": [ { "type": "_esphomelib._tcp.local.", "name": "tube*" }, { "type": "_zigate-zigbee-gateway._tcp.local.", "name": "*zigate*" }, { "type": "_zigstar_gw._tcp.local.", "name": "*zigstar*" }, { "type": "_slzb-06._tcp.local.", "name": "slzb-06*" } ], "is_built_in": true }, "data": { "ieee": "**REDACTED**", "nwk": 7411, "manufacturer": "_TZE200_v1jqz5cy", "model": "TS0601", "name": "_TZE200_v1jqz5cy TS0601", "quirk_applied": false, "quirk_class": "zigpy.device.Device", "manufacturer_code": 4098, "power_source": "Battery or Unknown", "lqi": null, "rssi": null, "last_seen": "2023-09-04T18:58:59", "available": true, "device_type": "EndDevice", "signature": { "node_descriptor": "NodeDescriptor(logical_type=, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=, mac_capability_flags=, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=True, *is_full_function_device=False, *is_mains_powered=False, *is_receiver_on_when_idle=False, *is_router=False, *is_security_capable=False)", "endpoints": { "1": { "profile_id": "0x0104", "device_type": "0x0051", "input_clusters": [ "0x0000", "0x0004", "0x0005", "0xef00" ], "output_clusters": [ "0x000a", "0x0019" ] } }, "manufacturer": "_TZE200_v1jqz5cy", "model": "TS0601" }, "active_coordinator": false, "entities": [], "neighbors": [], "routes": [], "endpoint_names": [ { "name": "SMART_PLUG" } ], "user_given_name": "Pool Sensor", "device_reg_id": "9de124bce188ac54b74bf7f2d0ac920a", "area_id": "backyard", "cluster_details": { "1": { "device_type": { "name": "SMART_PLUG", "id": 81 }, "profile_id": 260, "in_clusters": { "0x0000": { "endpoint_attribute": "basic", "attributes": { "0x0001": { "attribute_name": "app_version", "value": 65 }, "0x0004": { "attribute_name": "manufacturer", "value": "_TZE200_v1jqz5cy" }, "0x0005": { "attribute_name": "model", "value": "TS0601" } }, "unsupported_attributes": {} }, "0x0004": { "endpoint_attribute": "groups", "attributes": {}, "unsupported_attributes": {} }, "0x0005": { "endpoint_attribute": "scenes", "attributes": {}, "unsupported_attributes": {} }, "0xef00": { "endpoint_attribute": null, "attributes": {}, "unsupported_attributes": {} } }, "out_clusters": { "0x0019": { "endpoint_attribute": "ota", "attributes": {}, "unsupported_attributes": {} }, "0x000a": { "endpoint_attribute": "time", "attributes": {}, "unsupported_attributes": {} } } } } } }] ```

Logs

Logs ```python [Paste the logs here] ```

Custom quirk

Custom quirk ```python [Paste your custom quirk here] ```

Additional information

No response

tschiex commented 10 months ago

I would be happy to have the device supported too, but I cannot find it in the list of all the Zigbee2mqtt devices: looking for "_TZE200_v1jqz5cy" in the zigbee-herdsman-converters project returned nothing.

Instead, I found the very same request for the device to be supported in zigbee2mqtt here:

https://github.com/Koenkk/zigbee2mqtt/issues/18704

Do you have the Tuya ZigBee gateway? It is apparently needed to at least identify the Tuya Data Points and more. I don't have one...

KirkKirk commented 10 months ago

I find it strange that I can't seem to locate the source where I read about zigbee2mqtt supporting it. The threat you mentioned is the only information I came across this time. It's quite puzzling. I do have the Zigbee coordinator USB dongle, and I have successfully paired it with several other sensors.I didn't expect this expensive sensor to be unsupported. I hope someone can help as it looks promising for the purpose but it is now useless for me.

tschiex commented 10 months ago

The device is expensive, but rare (not many people bought it, compared to a connected plug :-). So less people interested.

I have contacted YIERYI, asking for a precise list and interpretation of the TUYA Data Points. We will see what they say. The Z2M issue contains a partial description.

Then, I may be will have a look on how to write a zha-quirk for the device. I'm a decent python programmer, but this also requires to learn ZHA/ZigPy and I'd rather avoid that. Time is rare.

Thomas

Le 07/09/2023 à 13:42, KirkKirk a écrit :

I find it strange that I can't seem to locate the source where I read about zigbee2mqtt supporting it. The threat you mentioned is the only information I came across this time. It's quite puzzling. I do have the Zigbee coordinator USB dongle, and I have successfully paired it with several other sensors.I didn't expect this expensive sensor to be unsupported. I hope someone can help as it looks promising for the purpose but it is now useless for me.

tschiex commented 10 months ago

I looked at this, this afternoon, but Zigbee and zha quirks are challenging to grab and the documentation for zha quirks looks minimal. Instead, the ZigBee specifications are precise but daunting to grab. I tried to get inspiration from existing tuya device quirks, but this device is challenging because it reports measurements (ORP and TDS) that do not exist in the ZCL Measurement cluster list. So I created new clusters with invented (unused) cluster numbers. I post below my current quirk version, but I haven't tried it. For sure, the measurement units/multipliers are likely incorrect.

I assume I'm close to the objective, and my hope is that one of the ZHA developers who knows about Tuya devices will have a look at it and make it work.

The Data point specification is taken from the still infant Z2M handler here: https://gist.github.com/Koenkk/7262a8a36bb84327ff563e0b8256024b

"""Tuya Pool sensor."""

from typing import Any, Dict

import zigpy.types as t
from zigpy.profiles import zha
from zigpy.quirks import CustomDevice
from zigpy.zcl.clusters.general import Basic, Groups, Ota, Scenes, Time
from zigpy.zcl.clusters.measurement import (
    pH,
    SodiumConcentration,
    ElectricalConductivity,
    ChlorineConcentration,
    TemperatureMeasurement,
)

from zhaquirks.const import (
    DEVICE_TYPE,
    ENDPOINTS,
    INPUT_CLUSTERS,
    MODELS_INFO,
    OUTPUT_CLUSTERS,
    PROFILE_ID,
    SKIP_CONFIGURATION,
)
from zhaquirks.tuya import TuyaLocalCluster, TuyaPowerConfigurationCluster
from zhaquirks.tuya.mcu import DPToAttributeMapping, TuyaMCUCluster

class TuyaTemperatureMeasurement(TemperatureMeasurement, TuyaLocalCluster):
    """Tuya local TemperatureMeasurement cluster."""

class TuyapH(pH, TuyaLocalCluster):
    """Tuya local pH cluster with a device RH_MULTIPLIER factor if required."""

class TuyaORP(TuyaLocalCluster):
    """Tuya local Oxydo-Reduction Potential cluster with a device RH_MULTIPLIER factor if required."""

    cluster_id = 0x042E
    name = "ORP Level"
    ep_attribute = "orp_level"

    attributes = {
        0x0000: ("measured_value", t.Single),  # fraction of 1 (one)
        0x0001: ("min_measured_value", t.Single),
        0x0002: ("max_measured_value", t.Single),
    }

    server_commands = {}
    client_commands = {}

class TuyaTDS(TuyaLocalCluster):
    """Tuya local Total Dissolved Solids cluster with a device RH_MULTIPLIER factor if required."""

    cluster_id = 0x042F
    name = "TDS Level"
    ep_attribute = "tds_level"

    attributes = {
        0x0000: ("measured_value", t.Single),  # fraction of 1 (one)
        0x0001: ("min_measured_value", t.Single),
        0x0002: ("max_measured_value", t.Single),
    }

    server_commands = {}
    client_commands = {}

class TuyaSodiumConcentration(SodiumConcentration, TuyaLocalCluster):
    """Tuya local NaCl cluster with a device RH_MULTIPLIER factor if required."""

class TuyaElectricalConductivity(ElectricalConductivity, TuyaLocalCluster):
    """Tuya local Electrical Conductivity cluster with a device RH_MULTIPLIER factor if required."""

class TuyaChlorineConcentration(ChlorineConcentration, TuyaLocalCluster):
    """Tuya local Chlorine Concentration cluster with a device RH_MULTIPLIER factor."""

    def update_attribute(self, attr_name: str, value: Any) -> None:
        """Apply a correction factor to value."""

        if attr_name == "measured_value":
            value = value * (
                self.endpoint.device.RH_MULTIPLIER
                if hasattr(self.endpoint.device, "RH_MULTIPLIER")
                else 100
            )
        return super().update_attribute(attr_name, value)

class PoolManufCluster(TuyaMCUCluster):
    """Tuya Manufacturer Cluster with Pool data points."""

    dp_to_attribute: Dict[int, DPToAttributeMapping] = {
        1: DPToAttributeMapping(
            TuyaTDS.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
        2: DPToAttributeMapping(
            TuyaTemperatureMeasurement.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
        101: DPToAttributeMapping(
            TuyaORP.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
        102: DPToAttributeMapping(
            TuyaChlorineConcentration.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
        7: DPToAttributeMapping(
            TuyaPowerConfigurationCluster.ep_attribute,
            "battery_percentage_remaining",
            converter=lambda x: x * 2,  # double reported percentage
        ),
        105: DPToAttributeMapping(
            TuyaChlorineConcentration.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
        10: DPToAttributeMapping(
            TuyapH.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
        11: DPToAttributeMapping(
            TuyaElectricalConductivity.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
        108: DPToAttributeMapping(
            TuyaElectricalConductivity.ep_attribute,
            "max_measured_value",
            converter=lambda x: x * 100,
        ),
        109: DPToAttributeMapping(
            TuyaElectricalConductivity.ep_attribute,
            "min_measured_value",
            converter=lambda x: x * 100,
        ),
        110: DPToAttributeMapping(
            TuyaORP.ep_attribute,
            "max_measured_value",
            converter=lambda x: x * 100,
        ),
        111: DPToAttributeMapping(
            TuyaORP.ep_attribute,
            "min_measured_value",
            converter=lambda x: x * 100,
        ),
        112: DPToAttributeMapping(
            TuyaChlorineConcentration.ep_attribute,
            "max_measured_value",
            converter=lambda x: x * 100,
        ),
        113: DPToAttributeMapping(
            TuyaChlorineConcentration.ep_attribute,
            "min_easured_value",
            converter=lambda x: x * 100,
        ),
        117: DPToAttributeMapping(
            TuyaSodiumConcentration.ep_attribute,
            "measured_value",
            converter=lambda x: x * 100,
        ),
    }

    data_point_handlers = {
        1: "_dp_2_attr_update",
        2: "_dp_2_attr_update",
        101: "_dp_2_attr_update",
        102: "_dp_2_attr_update",
        7: "_dp_2_attr_update",
        105: "_dp_2_attr_update",
        10: "_dp_2_attr_update",
        11: "_dp_2_attr_update",
        108: "_dp_2_attr_update",
        109: "_dp_2_attr_update",
        110: "_dp_2_attr_update",
        111: "_dp_2_attr_update",
        112: "_dp_2_attr_update",
        113: "_dp_2_attr_update",
        117: "_dp_2_attr_update",
    }

class TuyaPoolSensor(CustomDevice):
    """Tuya Pool sensor."""

    signature = {
        # "profile_id": 260,
        # "device_type": "0x0051",
        # "in_clusters": ["0x0000","0x0004","0x0005","0xef00"],
        # "out_clusters": ["0x000a","0x0019"]
        MODELS_INFO: [
            ("_TZE200_v1jqz5cy", "TS0601"),
        ],
        ENDPOINTS: {
            1: {
                PROFILE_ID: zha.PROFILE_ID,
                DEVICE_TYPE: zha.DeviceType.SMART_PLUG,
                INPUT_CLUSTERS: [
                    Basic.cluster_id,
                    Groups.cluster_id,
                    Scenes.cluster_id,
                    PoolManufCluster.cluster_id,
                ],
                OUTPUT_CLUSTERS: [Ota.cluster_id, Time.cluster_id],
            }
        },
    }

    replacement = {
        SKIP_CONFIGURATION: True,
        ENDPOINTS: {
            1: {
                DEVICE_TYPE: zha.DeviceType.TEMPERATURE_SENSOR,
                INPUT_CLUSTERS: [
                    Basic.cluster_id,
                    Groups.cluster_id,
                    Scenes.cluster_id,
                    PoolManufCluster,
                    TuyaTemperatureMeasurement,
                    TuyapH,
                    TuyaORP,
                    TuyaChlorineConcentration,
                    TuyaTDS,
                    TuyaElectricalConductivity,
                    TuyaPowerConfigurationCluster,
                ],
                OUTPUT_CLUSTERS: [Ota.cluster_id, Time.cluster_id],
            }
        },
    }
tschiex commented 10 months ago

I do not have a Tuya gateway, so I cannot test it, but I also assume that 2 of the clusters (pH and ORP) should include commands to start calibration with the 3 pH and the single ORP buffers that is provided with the device.

@KirkKirk If you have such a gateway, could you try to find how these can be called? Apparently, Tuya devices can be reverse-engineered using the Tuya IoT cloud platform. See Find Tuya Data Points

aberkeley commented 10 months ago

if it helps - seems it is working on deconz: https://forum.phoscon.de/t/tuya-tze200-v1jqz5cy-zigbee-chlorine-meter/3930

and here: https://github.com/dresden-elektronik/deconz-rest-plugin/issues/7215

i have one too but no tuya gateway either - the deconz implementation seems to be working - maybe that was where you saw it working?

KirkKirk commented 10 months ago

I do not have a Tuya gateway, so I cannot test it, but I also assume that 2 of the clusters (pH and ORP) should include commands to start calibration with the 3 pH and the single ORP buffer that is provided with the device.

@KirkKirk If you have such a gateway, could you try to find how these can be called? Apparently, Tuya devices can be reverse-engineered using the Tuya IoT cloud platform. See Find Tuya Data Points

I'm afraid that I don't have any other gateway apart from the SONOFF ZB Dongle-P Zigbee 3.0 USB Dongle Plus. I'm sorry, but my coding knowledge is limited, so I can't offer more assistance beyond what I have already provided. However, I am willing to help if you could guide me on what next steps to take. Thank you !

KirkKirk commented 10 months ago

if it helps - seems it is working on deconz: https://forum.phoscon.de/t/tuya-tze200-v1jqz5cy-zigbee-chlorine-meter/3930

and here: dresden-elektronik/deconz-rest-plugin#7215

i have one too but no tuya gateway either - the deconz implementation seems to be working - maybe that was where you saw it working?

Yes, that's what I saw! After considering my options, I have decided to stick with ZHA as it provides me with all the features I need.

tschiex commented 10 months ago

I bought a Tuya gateway and followed the instructions on Z2M https://www.zigbee2mqtt.io/advanced/support-new-devices/03_find_tuya_data_points.html

and found more DP Ids:

1: TDS 2: Temperature 101: ORP 102: Chlorine 7: Battery Percentage 103: pH Calibration 104: Backlight 105: Backlight Value 10: pH Value 11: EC Value 106: pH Max Value 107: pH Min Value 108: EC Max Value 109: EC Min Value 110: ORP Max Value 111: ORP Min Value 112: Chlorine Max Value 113: Chlorine Min Value 114: PH Calibration 115: EC Calibration 116: ORP Calibration 117: Salt

(Translation of Chinese by Google Translate camera from Tuya IoT platform website).

There are apparently 2 pH calibration DP Ids (and there are 2 pH calibration standards: Asia uses 6.86/4.00/9.18, EU uses 7/4/10). The two pH calibration DPs could be these 2 maybe?

My quirk proposal is therefore not only likely wrong but also incomplete...

KirkKirk commented 10 months ago

@tschiex Have you attempted to retrieve the data from the Tuya integration? If you can integrate it with Home Assistant, I will also purchase the Tuya gateway.

toulbar2 commented 10 months ago

@KirkKirk I will have a try but I'm not sure a failure will be informative: I already bought one Tuya device which is currently used. It's a Wifi device that I manage from HA, using "Local Tuya", not "Tuya" (I prefer to avoid cloud-dependencies, even if this device connects automatically to a Smart Life account).

Now, the Water quality device is ZigBee connected to a Wifi gateway which I registered to Tuya (not SmartLife) because I needed to use the Tuya IoT platform on the cloud to decipher the Data Points. I'm not sure the Tuya integration will cope with this situation, where there are 3 Tuya devices both Wifi and ZigBee and managed by 2 different Tuya cloud entities (Tuya and SmartLife). But I will try. :-)

tschiex commented 9 months ago

Ok, I did reset the tuya gateway and the PhMeter, linked them to my Smartlife account (success), installed the Tuya extension on HA, linked it to the proper account on the cloud and.... I'm sorry to say that the gateway and the pHmeter show no entity. I'm quite confident that with some work, I could get the pghMeter to work on local Tuya. Will try that instead and keep you updated.

tschiex commented 9 months ago

Well. I was wrong. The gateway can be integrated into local tuya (it shows in the list of devices that can be added) but the Zigbee device remains invisible under local Tuya. The only way is to go the ZHA/Z2M way apparently.

KirkKirk commented 9 months ago

THANK YOU @tschiex.

KirkKirk commented 9 months ago

Hi guys, is there any development on this?

alexisml commented 9 months ago

I bought a Tuya gateway and followed the instructions on Z2M https://www.zigbee2mqtt.io/advanced/support-new-devices/03_find_tuya_data_points.html

and found more DP Ids:

1: TDS 2: Temperature 101: ORP 102: Chlorine 7: Battery Percentage 103: pH Calibration 104: Backlight 105: Backlight Value 10: pH Value 11: EC Value 106: pH Max Value 107: pH Min Value 108: EC Max Value 109: EC Min Value 110: ORP Max Value 111: ORP Min Value 112: Chlorine Max Value 113: Chlorine Min Value 114: PH Calibration 115: EC Calibration 116: ORP Calibration 117: Salt

(Translation of Chinese by Google Translate camera from Tuya IoT platform website).

There are apparently 2 pH calibration DP Ids (and there are 2 pH calibration standards: Asia uses 6.86/4.00/9.18, EU uses 7/4/10). The two pH calibration DPs could be these 2 maybe?

My quirk proposal is therefore not only likely wrong but also incomplete...

I took your quirk proposal plus info from:

Fixed a typo in pH that made the imports not work, commented out battery info since it was failing ('TuyaPoolSensor' object has no attribute 'battery_bus'), and updated all cluster ids from the deconz comments. Still no luck, but a lot is going on in several places at the same time.

Here's my gist.

tschiex commented 9 months ago

Thanks Alexis. I'm also following the Z2M thread on this device. Apparently, the device does not report any Data Point (https://github.com/Koenkk/zigbee2mqtt/issues/18704#issuecomment-1730103420). I will try to ZigBee-sniff the pairing process this week-end (never tried this).

tschiex commented 9 months ago

Ok, I have done some Zigbee sniffing for Z2M/Koenkk and the device is now working on Z2M. Apparently there are dataQuery from the device that are useful for starting data exchange. See Z2M issue. Alexis, any hint here?

alexisml commented 9 months ago

I believe we need to migrate this code.

However, I did a search by queryOnDeviceAnnounce and found some devices that use that thingy (as Koenkk mentioned):

Then looked for those in ZHA and found them here. BUT, I don't see any special things on ZHA side? I'll continue digging anyway, but if someone more proficient with ZHA is able to help, that would be awesome.

EDIT: I also suspect that the cluster ids change I did was not needed.

tschiex commented 9 months ago

I was starting to do precisely the same analysis (finding the Z2M converters with the same specificity to find their mirror in ZHA) when I saw your message yesterday. Great!

For the quirk: I took your gist as is, I will compare it with my analysis of Tuya DPs and the other quirks you mention on ZHA. I also hope people knowledgeable in ZHA/zigpy will help. Not before this evening I'm afraid (local French time).

I'm wondering why only temperature shows as a sensor. The other virtual clusters (ORP,...) show in the "Manage your Zigbee device" menu but not beyond that. ANd they all return "None" when one query them.

alexisml commented 9 months ago

Just a few things / ideas to consider:

tschiex commented 9 months ago

Reacting quickly...

alexisml commented 9 months ago

Based on last response in z2m, we need to actually make a call to cluster 0xEF00, attribute 0x03.

https://developer.tuya.com/en/docs/iot/tuya-zigbee-universal-docking-access-standard?id=K9ik6zvofpzql#subtitle-6-Private%20cluster

I'm trying to but failing. And for some reason now I don't even get the quirk loaded for the device either.

MattWestb commented 9 months ago

If the device is OK paired and working you can re-powering the device and the MCU is sending current status for all DP is having / using to the host system = the same as command 0x3.

Put debug for quirks and look then the system is trying loading it and see what is complaining of but normally its some cluster that is not matching in the signature.

alexisml commented 9 months ago
2023-09-25 17:32:48.182 DEBUG (MainThread) [zigpy.zcl] [0x1708:1:0x0000] Decoded ZCL frame: Basic:Read_Attributes_rsp(status_records=[ReadAttributeRecord(attrid=0x0004, status=<Status.SUCCESS: 0>, value=TypeValue(type=CharacterString, value='_TZE200_v1jqz5cy')), ReadAttributeRecord(attrid=0x0005, status=<Status.SUCCESS: 0>, value=TypeValue(type=CharacterString, value='TS0601'))])
2023-09-25 17:32:48.182 INFO (MainThread) [zigpy.device] [0x1708] Read model 'TS0601' and manufacturer '_TZE200_v1jqz5cy' from <Endpoint id=1 in=[basic:0x0000, groups:0x0004, scenes:0x0005, None:0xEF00] out=[ota:0x0019, time:0x000A] status=<Status.ZDO_INIT: 1>>
2023-09-25 17:32:48.182 INFO (MainThread) [zigpy.device] [0x1708] Discovered basic device information for <Device model='TS0601' manuf='_TZE200_v1jqz5cy' nwk=0x1708 ieee=e0:79:8d:ff:fe:b8:76:2a is_initialized=True>
2023-09-25 17:32:48.182 DEBUG (MainThread) [zigpy.application] Device is initialized <Device model='TS0601' manuf='_TZE200_v1jqz5cy' nwk=0x1708 ieee=e0:79:8d:ff:fe:b8:76:2a is_initialized=True>
2023-09-25 17:32:48.183 DEBUG (MainThread) [zigpy.quirks.registry] Checking quirks for _TZE200_v1jqz5cy TS0601 (e0:79:8d:ff:fe:b8:76:2a)
2023-09-25 17:32:48.183 DEBUG (MainThread) [zigpy.quirks.registry] Considering <class 'TS0601_TZE200_v1jqz5cy.TuyaPoolSensor'>
2023-09-25 17:32:48.183 DEBUG (MainThread) [zigpy.quirks.registry] Fail because input cluster mismatch on at least one endpoint
2023-09-25 17:32:48.183 DEBUG (MainThread) [zigpy.quirks.registry] Considering <class 'TS0601_TZE200_v1jqz5cy.TuyaPoolSensor'>
2023-09-25 17:32:48.183 DEBUG (MainThread) [zigpy.quirks.registry] Fail because input cluster mismatch on at least one endpoint

:(

alexisml commented 9 months ago

Ok I got it back by removing the battery information (again). I have played around with ChatGPT, still to no success. You can find many tries and commented out stuff in my gist.

tschiex commented 9 months ago

@alexisml Looking into the ZigBee sniffing traces, I effectively see an Unknown command: 0x03 emitted from the Tuya GW to cluster 0xEF00, endpoint 1. See attached image.

The question would then be, how do we do the same with ZHA/zigpy.

Capture d’écran du 2023-09-26 22-52-00

Also, I don't understand why I see the temperature sensor under HA but not the other sensors (EC, TDS, ORP,...) although the corresponding code/clusters exist in the quirk.

tschiex commented 9 months ago

I have sniffed the device while calibration for the Z2M team. The trace is here. But that does not help at all for the ZHA side.

Looking into ZHA zha-device-handlers, in zhaquirks/tuya/init.py, it seems that only a subset of all Tuya commands are referenced and TUYA_DATA_QUERY (0x03) is not there. I have the feeling that this is the first device that requires this sort of "special command". It's different from the "Enchanted devices", it requires a different spell apparently.

@MattWestb It is not possible to power the device on/off. It's on integrated battery + integrated solar panel. The quirk seems to be loaded and working. From ZHA actions I can query but all the clusters attributes return None. The Tuya Data Query command must probably be issued following the device announcement.

tschiex commented 9 months ago

@MattWestb Here is the device diagnostic file. I would love to be able to send a data_query command to the MCU cluster, but it's not even available in the list of commands.


  "data": {
    "ieee": "**REDACTED**",
    "nwk": 27154,
    "manufacturer": "_TZE200_v1jqz5cy",
    "model": "TS0601",
    "name": "_TZE200_v1jqz5cy TS0601",
    "quirk_applied": true,
    "quirk_class": "ts0601_pool_sensor.TuyaPoolSensor",
    "manufacturer_code": 4098,
    "power_source": "Battery or Unknown",
    "lqi": 47,
    "rssi": null,
    "last_seen": "2023-10-01T16:38:28",
    "available": true,
    "device_type": "EndDevice",
    "signature": {
      "node_descriptor": "NodeDescriptor(logical_type=<LogicalType.EndDevice: 2>, complex_descriptor_available=0, user_descriptor_available=0, reserved=0, aps_flags=0, frequency_band=<FrequencyBand.Freq2400MHz: 8>, mac_capability_flags=<MACCapabilityFlags.AllocateAddress: 128>, manufacturer_code=4098, maximum_buffer_size=82, maximum_incoming_transfer_size=82, server_mask=11264, maximum_outgoing_transfer_size=82, descriptor_capability_field=<DescriptorCapability.NONE: 0>, *allocate_address=True, *is_alternate_pan_coordinator=False, *is_coordinator=False, *is_end_device=True, *is_full_function_device=False, *is_mains_powered=False, *is_receiver_on_when_idle=False, *is_router=False, *is_security_capable=False)",
      "endpoints": {
        "1": {
          "profile_id": "0x0104",
          "device_type": "0x0302",
          "input_clusters": [
            "0x0000",
            "0x0004",
            "0x0005",
            "0x041a",
            "0x042a",
            "0x042b",
            "0x043a",
            "0x0512",
            "0x0514",
            "0xef00"
          ],
          "output_clusters": [
            "0x000a",
            "0x0019"
         ]
        }
      },
      "manufacturer": "_TZE200_v1jqz5cy",
      "model": "TS0601"
    },
    "active_coordinator": false,
    "entities": [
      {
        "entity_id": "sensor.ph_metre_temperature",
        "name": "_TZE200_v1jqz5cy TS0601"
      }
    ],
    "neighbors": [],
    "routes": [],
    "endpoint_names": [
      {
        "name": "TEMPERATURE_SENSOR"
      }
    ],
    "user_given_name": "pH m\u00e8tre",
    "device_reg_id": "bd8b785e83ae81426246eb32f3bd0cd8",
    "area_id": "bureau",
    "cluster_details": {
      "1": {
        "device_type": {
          "name": "TEMPERATURE_SENSOR",
          "id": 770
        },
        "profile_id": 260,
        "in_clusters": {
          "0x0000": {
            "endpoint_attribute": "basic",
            "attributes": {
             "0x0001": {
                "attribute_name": "app_version",
                "value": 65
              },
              "0x0004": {
                "attribute_name": "manufacturer",
                "value": "_TZE200_v1jqz5cy"
              },
              "0x0005": {
                "attribute_name": "model",
                "value": "TS0601"
              }
            },
            "unsupported_attributes": {}
          },
          "0x0004": {
            "endpoint_attribute": "groups",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x0005": {
            "endpoint_attribute": "scenes",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0xef00": {
            "endpoint_attribute": "tuya_manufacturer",
            "attributes": {
              "0xef00": {
                "attribute_name": "mcu_version",
                "value": "1.0.15"
              }
            },
            "unsupported_attributes": {}
          },
          "0x0514": {
            "endpoint_attribute": "temperature",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x0512": {
            "endpoint_attribute": "ph",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x042a": {
            "endpoint_attribute": "orp_level",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x041a": {
            "endpoint_attribute": "chlorine_concentration",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x042b": {
            "endpoint_attribute": "tds_level",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x043a": {
            "endpoint_attribute": "electrical_conductivity",
            "attributes": {},
            "unsupported_attributes": {}
          }
        },
        "out_clusters": {
          "0x0019": {
            "endpoint_attribute": "ota",
            "attributes": {},
            "unsupported_attributes": {}
          },
          "0x000a": {
            "endpoint_attribute": "time",
            "attributes": {},
            "unsupported_attributes": {}
          }
        }
      }
    }
tschiex commented 9 months ago

@alexisml In your gist, I see:

  response = await cluster.write_attributes({
                0x03: b''  # Empty payload
            })

but in my Zigbee trace, it's not writing the attribute 0x03, it's sending the command 0x03 instead.

tschiex commented 9 months ago

@alexisml Here is my current (unsuccessful) variant quirk. I now have a (mute) battery level showing, but no visible sensor. The idea was to try to add the required commands to the "magic spell" of enchanted devices. It does not work but I imagine I also need an enchantable cluster...

The issue looks similar to this one, where this idea of overloading the spell is proposed. I assume that's where @MattWestb already saw this. @javicalle Could you suggest something maybe?

alexisml commented 9 months ago

@tschiex have you tried something new? I'm losing faith and I'm almost ready to migrate to z2m...

tschiex commented 9 months ago

Nothing beyond the quirk above. I'm quite stubborn but also very busy... My next move will be to sniff the adoption of the device with the quirk above to see if the suitable Data query packet is sent or not. If not, I will look into the creation of an Enchantable cluster. I'm surprised to see that nobody from ZHA intervenes to at least help us a bit...

I must say the liveliness on the Z2M side is attractive, but the idea of having an MQTT server forward Zigbee packets just repels me for now :-)

KirkKirk commented 9 months ago

Thanks @tschiex. Thank you for your effort. I apologize that I am unable to offer more assistance than simply testing.

It's strange that ZHA shows no interest in this device. I wonder why because seems a solid one.

KirkKirk commented 8 months ago

Hi guys, I am thinking of moving to Z2M. Can All my current Zigpy devices work fine with ZHA but this sensor is very important for me which makes me think about moving to z2m. Can someone tell me the experience with it?

tschiex commented 8 months ago

Some news to keep the thread alive... I sniffed a new pairing with the modified quirk that overloads the spell function (just added the cluster command 0x03 which seems needed to "awaken" the device:

    async def spell(self) -> None:
        """Initialize device so that all endpoints become available."""
        attr_to_read = [4, 0, 1, 5, 7, 0xFFFE]
        basic_cluster = self.endpoints[1].in_clusters[0]
        await basic_cluster.read_attributes(attr_to_read)
        await basic_cluster.write_attributes({0xffde: 0x13})
        mcu_cluster = self.endpoints[1].in_clusters[3]
        await mcu_cluster.command(0x03)

but sniffing does not show any 0x03 command issued. The spell has not been cast apparently. I assume the lack of any enchantable cluster is the issue. I really don't know which cluster I should make enchantable. Still hoping that a ZHA pro will come and help us...

tschiex commented 8 months ago

@TheJulianJES git blame kindly told me you contributed to the introduction of the spell/enchantable cluster combo in zha-quirk. I hope this message will reach you and you will come and give us a hand. Apparently, ZHA needs to learn at least 2 types of spells.

tschiex commented 8 months ago

I realized that the spell function should be in the Enchantable cluster, not the Enchanted device. I used the TuyaNoBindPowerConfigurationCluster as a possibility, overloading spell with just:

class MyTuyaNoBindPowerConfigurationCluster(TuyaNoBindPowerConfigurationCluster):

    async def spell(self):
        """Cast spell, so the Tuya device works correctly."""
        self.debug("Executing spell on Tuya device %s", self.endpoint.device.ieee)
        # await basic_cluster.write_attributes({0xffde: 0x13})
        mcu_cluster = self.endpoints[1].in_clusters[3]
        await mcu_cluster.command(0x03)
        self.debug("Executed spell on Tuya device %s", self.endpoint.device.ieee)

which is used in the "replacement". The quirk loads but...no measure. I will sniff again to try to see if the suitable 0x03 tuya command is emitted. Full quirk update here.

tschiex commented 8 months ago

The spell has not been cast... No "Unknown Command: 0x03 in the wireshark sniff trace. I really don't understand the logic of ZHA (or my Python skills are just too bad :-( )

tschiex commented 8 months ago

I got the ZHA logs. They are available here.

There are 2 surprising issues. First, the spell is apparently started:

[0xE520:1:0x0001] Executing spell on Tuya device e0:79:8d:ff:fe:da:7d:3b

but I did not see any 0x03 command on Wireshark... and never saw the expected message:

[0xE520:1:0x0001] Executed spell on Tuya device e0:79:8d:ff:fe:da:7d:3b

So, there is a mistake somewhere in the spell function.

Then, ZHA creates 4 sensors, among which battery (but no level reported, just Unknown) and Temperature (but I don't see the sensor in HA).

The most disagreeable thing is that entity discovery is apparently started before the spell starts to be cast. Too soon probably.

Well. Enough for today. ZHA is absorbing much of my time, for no results.

KirkKirk commented 8 months ago

Hopefully @TheJulianJES will jump in and help

TheJulianJES commented 8 months ago

Unfortunately, I don't have much free time at the moment, but looking at the Z2M implementation, it uses the normal Tuya spell that we also use. So I don't think the "spell" is the issue.

Looking at the Z2M code, it seems like the devices has some Tuya data points for getting data and setting data. Using a quirk, it should be possible to map those attributes to a "fake attributes" (random IDs) on a fake/LocalDataCluster. Tuya quirks already have TuyaMCUCluster / TuyaLocalCluster that are based on that.

This won't show sensors in HA though. You could use custom/hacky AnalogOutput clusters and have one endpoint for every data exposed by the sensor, or it could be implemented properly in ZHA, similar to this: sensor.py#L1017-L1027 That would still require the quirk to "convert" Tuya data points to (fake but working) "ZCL attributes".

Maybe take a look at the implementation for the Tuya fingerbot (scroll or open file completely): https://github.com/zigpy/zha-device-handlers/blob/307f92676845241e20cbd79401479139293f7e05/zhaquirks/tuya/ts0001_fingerbot.py#L37-L113 You can probably completely ignore the command() method (and the battery_percentage_remaining, as that maps to another cluster). But the attributes would be added similar to this.

I'd just try to get "converting" all Tuya data points to one "normal cluster" working. You can then read/set the values using the clusters UI in ZHA: Go to the device page -> click on the three dots -> Manage Zigbee device -> Clusters tab -> select custom MCU cluster from quirk in first-drop down -> select attribute in second drop-down -> press read/attribute. For this, always leave the second text field (manufacturer code) empty. Only use the first one for reading/setting attributes.

If you get the attributes "converted" properly, a ZHA PR could be made to expose the custom attributes as custom entities in ZHA.

TheJulianJES commented 8 months ago

Ah, also now saw this version of the quirk. I haven't had a close look, but the approach doesn't look bad (remove the custom spell code though). (and btw, uploading that file with a .py extension will have GitHub show the correct text "colors" to improve readability)

I don't think ZHA creates sensors for the clusters you used. It's simply not added anywhere here from what I see. (So it's expected that you don't get any entities.)

Can you try reading the attribute(s) on those clusters using the clusters UI and see if you get any valid values? That will at least let us know if the quirk works.


I now have a (mute) battery level showing, but no visible sensor.

Do you also not get a sensor entity for the temperature measurement? That should already be supported in ZHA. (And I don't see any obvious issues regarding that in the code. Also check if the attribute is "converted" correctly, so can be read from the clusters UI. Maybe also take a look at other Tuya temperature sensor quirks to see if they're implemented in a similar way.)

tschiex commented 8 months ago

@TheJulianJES Thanks for the messages and sorry for bothering you. You did help. I now have access to the temperature. The scale is not correct but I will dig into this. Using the "Manage Zigbee device" item (which I already tried to use to send commands, to no avail) now does report meaningful measures for the pH for example (incorrect scale again). So it seems that just removing the spell, plus all the changes I made, did the trick.

I will dig into sensor.py and fingerbot.py and will try to finish this on my side and will only bother you again if I'm stuck. I updated the gist, with a nice python extension :-) So, @KirkKirk, if you want to start having fun with it, you can download it and use it as a custom_zha_quirk.

GPSPerth commented 8 months ago

Woohoo, me too...

KirkKirk commented 8 months ago

@TheJulianJES Thank you for the response and help provided to @tschiex

@tschiex As soon as I saw the news, I took action. I downloaded your .py file and am happy to report that the system is responding. Although there is no data from the sensors, I am receiving data and the message in the log "_TZE200_v1jqz5cy TS0601 Attribute Updated event was fired" when I read multiple attributes Ota (Endpoint id: 1, Id: 0x0019, Type: out).

image

image

treepleks commented 8 months ago

@KirkKirk It would be nice if could check that you can read, for example, the "measured_value" in the TuyaPH cluster. I got "79000" which is probably 7.9 (my pH meter is in the air.). Click on "Read attribute". I still have some doubts: the values reported are too stable. Looks fishy.

Capture d’écran du 2023-10-24 13-05-44

Anyaway, yesterday evening, I started to follow the 'clean' path indicated by @TheJulianJES. I added a "Chlorine concentration" sensor in ZHA (it has repercussions in HA too, because some constants are missing and need to be defined there). So now, I also have a Chlorine concentration reported (yes, the temperature scale is wrong by a factor of 10, it's only 21.5 °C here).

Capture d’écran du 2023-10-24 13-03-02

More soon hopefully :-)

tschiex commented 8 months ago

One more...

Capture d’écran du 2023-10-24 22-30-21

KirkKirk commented 8 months ago

Unfortunately, I couldn't get any data. Not sure what I'm doing wrong 🤯 Screenshot_20231025_111655_Home Assistant Screenshot_20231025_111418_Home Assistant

GPSPerth commented 8 months ago

Same here...