Issues getting response from sensors

[romicaiarca]

Hi,

I am trying to implement custom_component for Home Assistant but I have issues getting status from sensors. Here you have the exception that tiny tuya is returning on status call.

Requirement for tinytuya that I added in the component: "requirements": ["tinytuya>=1.10.0"],

I have no idea why I receive this error. Is it possible to receive this because I instantiate 3 tintytuya objects for each parameter from sensor?

Thanks!

Error doing job: Future exception was never retrieved
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/config/custom_components/local_tuya_ths/THSTemperature.py", line 79, in get_data_from_sensor
    data = self.device.status()
  File "/usr/local/lib/python3.10/site-packages/tinytuya/core.py", line 1550, in status
    data = self._send_receive(payload, 0, getresponse=(not nowait))
  File "/usr/local/lib/python3.10/site-packages/tinytuya/core.py", line 912, in _send_receive
    return self.parent._send_receive(payload, minresponse, getresponse, decode_response, from_child=self)
  File "/usr/local/lib/python3.10/site-packages/tinytuya/core.py", line 1043, in _send_receive
    return self._process_message( msg, dev_type, from_child )
  File "/usr/local/lib/python3.10/site-packages/tinytuya/core.py", line 1101, in _process_message
    return self._send_receive( None, minresponse, True, decode_response, from_child=True)
NameError: name 'minresponse' is not defined

[jasonacox]

Hi @romicaiarca - thanks for opening this issue! This is a bug. Both decode_response and minresponse are not in scope in that function. I'll submit a fix.

@uzlonewolf I believe this is all that is missing, but check my PR #267 .

...
        return self._process_message( msg, dev_type, from_child )

    def _process_message( self, msg, dev_type=None, from_child=None ):
...

...
        return self._process_message( msg, dev_type, from_child, minresponse, decode_response )

 def _process_message( self, msg, dev_type=None, from_child=None, minresponse=28, decode_response=True ):
...

[jasonacox]

I instantiate 3 tintytuya objects for each parameter

@romicaiarca would you mind sharing a code snip on how you are doing that? Multiple instances are allowed, but the area for this bug is in the async messaging so I want to make sure I replicate and add to my testing. :)

[uzlonewolf]

I think it's going to be a bit difficult to test for this, it requires having multiple Zigbee devices attached to a gateway and receiving an async update from one (say, temperature change or someone pressed a button) while trying to get a response from a different one.

[romicaiarca]

I instantiate 3 tintytuya objects for each parameter

@romicaiarca would you mind sharing a code snip on how you are doing that? Multiple instances are allowed, but the area for this bug is in the async messaging so I want to make sure I replicate and add to my testing. :)

Hi, yes.

Here you have what I'm doing:

sensor.py

# some imports related home assistant

async def async_setup_platform(
    hass: HomeAssistant,
    config: ConfigType,
    async_add_entities: AddEntitiesCallback,
    discovery_info: DiscoveryInfoType | None = None
) -> None:
    """Set up the sensor platform."""

    sensors = config.get("sensors")
    gw_device_id = config.get("gw_device_id")
    gw_local_key = config.get("gw_local_key")
    gw_ip_address = config.get("gw_ip_address")
    sensorsEntities = sensors["entities"]

    gw = SilverCrestGateway(hass, gw_device_id, gw_local_key, gw_ip_address)

    entities = []

    for deviceEntry in sensorsEntities:
        entities.append(THSTemperature(hass, gw.get_gw, deviceEntry))
        entities.append(THSHumidity(hass, gw.get_gw, deviceEntry))
        entities.append(THSBattery(hass, gw.get_gw, deviceEntry))

    async_add_entities(entities, True)

# SilverCrestGateway

lass SilverCrestGateway():
    """Representation of a Sensor."""

    def __init__(self, hass: HomeAssistant, gw_device_id, gw_local_key, gw_ip_address=None):
        """Initialize the DHT sensor"""
        self.device_id = gw_device_id
        self.local_key = gw_local_key
        self.ip_address = gw_ip_address
        self.hass = hass
        self.name = "Silver Crest Gateway"

        self.gw = tinytuya.Device(
            gw_device_id,
            address=self.ip_address,
            local_key=gw_local_key,
            persist=True,
            version=_VERSION
        )

        self.supported_dps = ['temperature', 'humidity', 'battery_percentage']
        self.ip_address = self.gw.address
        self.data = {}
        self._cached_state = {}

    @property
    def get_gw(self) -> tinytuya.Device:
        return self.gw

    @property
    def get_device_id(self):
        return self.device_id

    @property
    def get_local_key(self):
        return self.local_key

    @property
    def get_ip_address(self):
        return self.ip_address

    @property
    def get_sensor_data(self):
        return self.data

    @property
    def get_supported_dps(self):
        return self.supported_dps

All THS classes are exactly the same, the diference is that I get other dps parameter

class THSTemperature(SensorEntity):
    """Representation of a Sensor."""

    def __init__(self, hass, gw, deviceEntry) -> None:
        """Initialize the DHT sensor"""

        self.hass = hass
        self.device_id = deviceEntry["device_id"]
        self.local_key = deviceEntry["local_key"]
        self._name = (deviceEntry["name"] + " temperature")
        self._attr_name = self._name
        self._attr_state_class = SensorStateClass.MEASUREMENT
        self._attr_native_value = 0
        self.entity_id = "sensor." + "local_tuya_ths." + self._name.replace(" ", "_")

        self.device = tinytuya.Device(
            dev_id=self.device_id,
            cid=self.local_key,
            parent=gw,
            persist=True,
            version=_VERSION
        )

        self._attr_native_unit_of_measurement = TEMP_CELSIUS
        self._attr_device_class = SensorDeviceClass.TEMPERATURE

        self.hass.async_add_executor_job(self.get_data_from_sensor)

    async def async_update(self) -> None:
        """Fetch new state data for the sensor.

        This is the only method that should fetch new data for Home Assistant.
        """

        self.hass.async_add_executor_job(self.get_data_from_sensor)

    def get_data_from_sensor(self):
        data = {}
        data = self.device.status()

        _LOGGER.error(f"get_data_from_sensor {self.name}")

        if "Error" in data:
            _LOGGER.error(f"If error: {data['Err']}: {data['Error']}")
            return -1

        if data and 'dps' in data:
            dps = data['dps']

            if "1" in dps:
                _LOGGER.error(f"if temperature {self.name=} {type(dps['1'])} {dps['1']=}")
                _LOGGER.warning(dps["1"] / 10)
                self._attr_native_value = (int(dps["1"]) / 10)

        return data

The sensor for all 3 properties is the same since the sensor returns 3 dps (1 => temperature, 2 => humidity, 4 => battery)

Please let me know if you need other information regarding my code.

Any way, for the moment I am trying to make a single device instance per sensor and in each async_add_entities element jut to check the status.

[jasonacox]

@romicaiarca - Thanks! As @uzlonewolf mentions, I don't have enough devices to test all cases. I do keep dreaming about creating tuya simulated devices and adding them to a comprehensive GitHub action script test so I don't have to test manually. 😊

@romicaiarca - I just pushed the patch to v1.10.1. Please upgrade and test when you can.

pip install --upgrade tinytuya

"requirements": ["tinytuya>=1.10.1"]

[romicaiarca]

I have pulled the changes on my local. It looks like now I have the follow error:

❯ python -m sensor_local
GW IP found: 192.168.xxx.xxx - getting status
data[dps]: , {'1': 239, '2': 604}
data2[dps]: , {'4': 99}
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/romica.iarca/code/tinytuya/sensor_local.py", line 206, in <module>
    getStatusDeviceViaGateway()
  File "/Users/romica.iarca/code/tinytuya/sensor_local.py", line 165, in getStatusDeviceViaGateway
    print(f"data3[dps]: , {data3['dps']}")
TypeError: 'NoneType' object is not subscriptable

The code that is having this issue:

def getStatusDeviceViaGateway():
    # tinytuya.set_debug(True, False)
    # configure the parent device
    gw = tinytuya.Device(
        devices["gateway"]["ZIGBEE_GATEWAY_ID"],
        address=None,
        local_key=devices["gateway"]["ZIGBEE_GATEWAY_LOCAL_KEY"],
        persist=True,
        version=TUYA_API["VERSION"]
     )

    # print(gw.status())
    print('GW IP found:', gw.address, '- getting status')

    device = tinytuya.Device(
        devices["devices"]["THS_OFFICE_WALL_1"]["id"],
        cid=devices["devices"]["THS_OFFICE_WALL_1"]["node_id"],
        parent=gw,
        persist=True,
        version=TUYA_API["VERSION"]
    )

    device2 = tinytuya.Device(
        devices["devices"]["THS_OFFICE_WALL_2"]["id"],
        cid=devices["devices"]["THS_OFFICE_WALL_2"]["node_id"],
        parent=gw,
        persist=True,
        version=TUYA_API["VERSION"]
    )

    device3 = tinytuya.Device(
        devices["devices"]["THS_BEDROOM"]["id"],
        cid=devices["devices"]["THS_BEDROOM"]["node_id"],
        parent=gw,
        persist=True,
        version=TUYA_API["VERSION"]
    )

    while True:
        # device.heartbeat()
        data = device.status()
        data2 = device2.status()
        data3 = device3.status()

        print(f"data[dps]: , {data['dps']}")
        print(f"data2[dps]: , {data2['dps']}")
        print(f"data3[dps]: , {data3['dps']}")

        time.sleep(3)

[jasonacox]

print(f"data3[dps]: , {data3['dps']}") TypeError: 'NoneType' object is not subscriptable

Hi @romicaiarca - thanks for testing! That error indicates that the data3 status response from device3 does not have a 'dps' value. Can you turn on debugging to see what is happening?

You could put this above the while loop:

tinytuya.set_debug(True)

You could also add a print to see exactly what is being returned:

    while True:
        # device.heartbeat()
        data = device.status()
        data2 = device2.status()
        data3 = device3.status()
        print(data3)

        print(f"data[dps]: , {data['dps']}")
        print(f"data2[dps]: , {data2['dps']}")
        print(f"data3[dps]: , {data3['dps']}")

        time.sleep(3)

[romicaiarca]

python -m sensor_local
DEBUG:TinyTuya [1.10.1]

DEBUG:Listening for device bf583defebc47442a4zgjr on the network
DEBUG:find() received broadcast from '192.168.100.107': {'ip': '192.168.100.107', 'gwId': 'bf583defebc47442a4zgjr', 'active': 2, 'ablilty': 0, 'encrypt': True, 'productKey': 'keyyj3fy8x98arty', 'version': '3.5', 'token': True}
DEBUG:find() is returning: {'ip': '192.168.100.107', 'version': '3.5', 'id': 'bf583defebc47442a4zgjr', 'product_id': 'keyyj3fy8x98arty', 'data': {'ip': '192.168.100.107', 'gwId': 'bf583defebc47442a4zgjr', 'active': 2, 'ablilty': 0, 'encrypt': True, 'productKey': 'keyyj3fy8x98arty', 'version': '3.5', 'token': True}}
GW IP found: 192.168.100.107 - getting status
DEBUG:status() entry (dev_type is v3.5)
DEBUG:building command 10 payload=b'{"gwId":"bf583defebc47442a4zgjr","devId":"bf26e524aed4d6bfe9m2gu","uid":"bf26e524aed4d6bfe9m2gu","t":"1675062543","cid":"a4c1383ef5a956c4"}'
DEBUG:sending payload quick
DEBUG:final payload: b'0123456789abcdef'
DEBUG:payload encrypted=b'00006699000000000001000000030000002c30313233343536373839616213b9483bb1e2d85e9147b3e93fd94ac8b7f44f3b3599e5a01532962557888a8600009966'
DEBUG:received data=b'0000669900000000fafd00000004000000503836353532353134303666336895f0422aef5ecea975d726f19ed9866dd2d3d7b021f7bd3d9474faa3b5b29953eb32d79e70691150452b87d55f869d94676062f7ad5a2806b5047cde5fb211aac9b2eb00009966'
DEBUG:decrypted session key negotiation step 2 payload=b'01f62e70222c81d5&r\x0eK\x81\x90\xc8\xad\x88\xf3\x18\x8bzy\xe8\xc2\x13c\xd2]Q\x91\xe9\xe67\xe3\n\x1c\x02\xbf\xe6\xf0'
DEBUG:payload type = <class 'bytes'> len = 48
DEBUG:session local nonce: b'0123456789abcdef' remote nonce: b'01f62e70222c81d5'
DEBUG:sending payload quick
DEBUG:final payload: b'\x8bbqo\x84\xa3\xae-\x1aQ@\xa1X\xfeY\x88\xe2\x1f\xfcT+<+\xc7|\x9eX\x19\xaa\xdf\x9d:'
DEBUG:payload encrypted=b'00006699000000000002000000050000003c303132333435363738396162a8ea0b6701744044b32f922a044376267e4d96e845adb464f671e881619d7b40d2000ec726e218b4952504476023c85900009966'
DEBUG:Session nonce XOR'd: b'\x00\x00T\x05\x06P\x01\x07\n\x0bS\x01[U\x01S'
DEBUG:Session IV: b'0123456789ab'
DEBUG:Session key negotiate success! session key: b'#\x88.\r\x83\x87\xefn\xa3u\x81\x8a\x07\xe8.\xfd'
DEBUG:sending payload
DEBUG:final payload: b'{"gwId":"bf583defebc47442a4zgjr","devId":"bf26e524aed4d6bfe9m2gu","uid":"bf26e524aed4d6bfe9m2gu","t":"1675062543","cid":"a4c1383ef5a956c4"}'
DEBUG:payload encrypted=b'0000669900000000000300000010000000a730313233343536373839616224cd1e118fc2cdea167c7217e9d9226c2b56f9be59dc11abe2d686a89578d6c335f419c03af07975b4cca6166f80ac659722d088d9e05e53ab3251e50823f0424a46a1fa3277c9d4e1baccb5bc47b670004c52b1bce4cb3e87fdb552533941246a03d708f11448f067e5c04177dbf2f6883268c734785e9472e647f1f801c2d8ce198eda622142a8bc56607acaa0710f427467321ed950cc983b8200009966'
DEBUG:received data=b'0000669900000000fafe00000010000000356462636564643466626230667f19f5b7db5ae71b74e60383f34198ab575b39701dd68514edd6d5c3a237c3c9075d258eedf76e79f400009966'
DEBUG:received message=TuyaMessage(seqno=64254, cmd=16, retcode=1, payload=b'json obj data unvalid', crc=b'\xd6\xd5\xc3\xa27\xc3\xc9\x07]%\x8e\xed\xf7ny\xf4', crc_good=True, prefix=26265, iv=b'dbcedd4fbb0f')
DEBUG:raw unpacked message = TuyaMessage(seqno=64254, cmd=16, retcode=1, payload=b'json obj data unvalid', crc=b'\xd6\xd5\xc3\xa27\xc3\xc9\x07]%\x8e\xed\xf7ny\xf4', crc_good=True, prefix=26265, iv=b'dbcedd4fbb0f')
DEBUG:decode payload=b'json obj data unvalid'
DEBUG:decoded results='json obj data unvalid'
DEBUG:ERROR Invalid JSON Response from Device - 900 - payload: "json obj data unvalid"
DEBUG:Recieved async update for wrong CID None while looking for CID None, trying again
DEBUG:status() received data=None
DEBUG:status() entry (dev_type is v3.5)
DEBUG:building command 10 payload=b'{"gwId":"bf583defebc47442a4zgjr","devId":"bf2527ca991372c229ygxv","uid":"bf2527ca991372c229ygxv","t":"1675062548","cid":"a4c1386873a5625b"}'
DEBUG:sending payload
DEBUG:final payload: b'{"gwId":"bf583defebc47442a4zgjr","devId":"bf2527ca991372c229ygxv","uid":"bf2527ca991372c229ygxv","t":"1675062548","cid":"a4c1386873a5625b"}'
DEBUG:payload encrypted=b'0000669900000000000400000010000000a730313233343536373839616224cd1e118fc2cdea167c7217e9d9226c2b56f9be59dc11abe2d686a89578d6c335f419c03af07975b4cca6166f83fb67c67788d48ce70d57aa6606e51c76ef414a46a1fa3277c9d4e1baccb5bf10b42155140ee4bbb7cf3fd3aab546062642246a03d708f11448f067e5c04177dbf2fd883268c734785e9472e647f1f801c2dd934888da6e2246feea56606f3e72a415faa88d098b3769f3acfd7b00009966'
DEBUG:received data=b'0000669900000000faff0000001000000035323966316334333263383735816df4e55fc3d4d95226711a436c9ca73c8c2448aa53755b517bd6ce652a683019d08848e7540840e700009966'
DEBUG:received message=TuyaMessage(seqno=64255, cmd=16, retcode=1, payload=b'json obj data unvalid', crc=b'{\xd6\xcee*h0\x19\xd0\x88H\xe7T\x08@\xe7', crc_good=True, prefix=26265, iv=b'29f1c432c875')
DEBUG:raw unpacked message = TuyaMessage(seqno=64255, cmd=16, retcode=1, payload=b'json obj data unvalid', crc=b'{\xd6\xcee*h0\x19\xd0\x88H\xe7T\x08@\xe7', crc_good=True, prefix=26265, iv=b'29f1c432c875')
DEBUG:decode payload=b'json obj data unvalid'
DEBUG:decoded results='json obj data unvalid'
DEBUG:ERROR Invalid JSON Response from Device - 900 - payload: "json obj data unvalid"
DEBUG:Recieved async update for wrong CID None while looking for CID None, trying again
DEBUG:status() received data=None
DEBUG:status() entry (dev_type is v3.5)
DEBUG:building command 10 payload=b'{"gwId":"bf583defebc47442a4zgjr","devId":"bf24382be8c7da3f45n8xt","uid":"bf24382be8c7da3f45n8xt","t":"1675062553","cid":"a4c138ad1d974d89"}'
DEBUG:sending payload
DEBUG:final payload: b'{"gwId":"bf583defebc47442a4zgjr","devId":"bf24382be8c7da3f45n8xt","uid":"bf24382be8c7da3f45n8xt","t":"1675062553","cid":"a4c138ad1d974d89"}'
DEBUG:payload encrypted=b'0000669900000000000500000010000000a730313233343536373839616224cd1e118fc2cdea167c7217e9d9226c2b56f9be59dc11abe2d686a89578d6c335f419c03af07975b4cca6166f82fa689774d4d5dee35e04fa3200e90b29ef434a46a1fa3277c9d4e1baccb5be11bb7056480fb6bfe49c6f87acb951592640246a03d708f11448f067e5c04177dbf3f6883268c734785e9472e647f1f801c28acf4edf826c2010f3b1566029bfa97ecf1734a2eb45ace6b6fb796100009966'
DEBUG:received data=b'0000669900000000fb0000000010000000523233346465663332636639631b5913e17257f79ef3e59bfeba7c95788cb91da3badec1132bf3718345eeb9020535ffa5d34a7a9b13c4365f1d6b46c07dd380a3ac98df8751021c04b976b24f1033d7307a3f00009966'
DEBUG:received message=TuyaMessage(seqno=64256, cmd=16, retcode=0, payload=b'{"dps":{"1":232,"2":569},"cid":"a4c138ad1d974d89"}', crc=b'\xdf\x87Q\x02\x1c\x04\xb9v\xb2O\x103\xd70z?', crc_good=True, prefix=26265, iv=b'234def32cf9c')
DEBUG:raw unpacked message = TuyaMessage(seqno=64256, cmd=16, retcode=0, payload=b'{"dps":{"1":232,"2":569},"cid":"a4c138ad1d974d89"}', crc=b'\xdf\x87Q\x02\x1c\x04\xb9v\xb2O\x103\xd70z?', crc_good=True, prefix=26265, iv=b'234def32cf9c')
DEBUG:decode payload=b'{"dps":{"1":232,"2":569},"cid":"a4c138ad1d974d89"}'
DEBUG:decoded results='{"dps":{"1":232,"2":569},"cid":"a4c138ad1d974d89"}'
DEBUG:status() received data={'dps': {'1': 232, '2': 569}, 'cid': 'a4c138ad1d974d89', 'device': Device( 'bf24382be8c7da3f45n8xt', address=None, local_key='', dev_type='v3.5', connection_timeout=5, version=3.5, persist=True, cid='a4c138ad1d974d89', parent='bf583defebc47442a4zgjr', children={} )}
{'dps': {'1': 232, '2': 569}, 'cid': 'a4c138ad1d974d89', 'device': Device( 'bf24382be8c7da3f45n8xt', address=None, local_key='', dev_type='v3.5', connection_timeout=5, version=3.5, persist=True, cid='a4c138ad1d974d89', parent='bf583defebc47442a4zgjr', children={} )}
Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/Users/romica.iarca/code/tinytuya/sensor_local.py", line 207, in <module>
    getStatusDeviceViaGateway()
  File "/Users/romica.iarca/code/tinytuya/sensor_local.py", line 164, in getStatusDeviceViaGateway
    print(f"data[dps]: , {data['dps']}")
TypeError: 'NoneType' object is not subscriptable

[uzlonewolf]

Are you sure those cid values are correct? With one of them working fine it makes me think they might not be. If they are correct then can you try changing the device id from devices["devices"][...]["id"] to devices["devices"][...]["node_id"]?

[romicaiarca]

Are you sure those cid values are correct? With one of them working fine it makes me think they might not be. If they are correct then can you try changing the device id from devices["devices"][...]["id"] to devices["devices"][...]["node_id"]?

They are correct. The id property is copied from iot and there the property is node_id. What I saw is that sometimes the sensor is sending only some dps, most frequent the response dps doesn't include battery_percentage (dps[4]). It is possible that sometime the sensor sends some json that cannot be decoded... Not sure why.

[jasonacox]

It is odd that it is only that one. The others seem to be working?

[jasonacox]

@romicaiarca A new version has been released (v1.12.0) with updates by @uzlonewolf for gateway (parent/child linkage). I don't know if this will help your use case, but it would be interesting to see if the mapping from wizard makes it any easier.

# upgrade
pip install --upgrade tinytuya

# run wizard
python3 -m tinytuya wizard

[uzlonewolf]

Thanks for the bump @jasonacox ! While I don't think those updates will do anything for this issue, it did remind me that the SilverCrest gateway and Zigbee devices I ordered came in a while ago. Unfortunately the gateway I got uses protocol v3.3 with no updates available and works fine with the above code. Looking at the code and the above log messages a bit closer, I think I see the problem: v3.5 devices need DP_QUERY mapped to DP_QUERY_NEW, but Zigbee devices need the CONTROL payload to be empty; for now I think I'll add a new "zigbee_v35" device type while thinking about possible ways to rework this to keep it maintainable long-term.

[jasonacox]

Makes sense. Great discovery, regardless. Like the device22, I keep thinking we will find some elegant algorithm that dictates why, but perhaps these one-off devices are less one-off and just the way Tuya does things sometimes. 🤷