Bluetooth Gateway and RGBW light issues

[bnazari]

First of all, amazing project. I had no idea how complex this was going to be when I got into this. Now for the pleas of help:

I have a Tuya RGBW bulb that is bluetooth only, connecting through a bluetooth gateway. App is working fine, etc.Eventual goal is for HA to control all these with LocalTuya, but figured that wouldn't work without this working first. After running tinytua devices, I can only see the GW (and a few of the DPS, but not all). The bulb does not return anything.

I then attempted this:

import tinytuya
import time

tinytuya.set_debug(True)
gwID = "AAAAA"
lightID = "BBBBB"
lightCID = "CCCCC"
localKey = "DDDDD"
ip = "10.0.1.118"

gw = tinytuya.Device(gwID, ip, localKey)
gw.set_version(3.3)
gw.set_socketPersistent(True)

light1 = tinytuya.BulbDevice(lightID, cid=lightCID, parent=gw)

print(light1.status())

payload=light1.generate_payload(tinytuya.CONTROL, data={'1': True,'2': 0, '4': 1000, '3': 100}, gwId=gwID, devId=lightID, uid=lightCID)
light1._send_receive(payload)
time.sleep(5)
payload=light1.generate_payload(tinytuya.CONTROL, data={'1': False, '2': 0, '4': 1000, '3': 100}, gwId=gwID, devId=lightID, uid=lightCID)
light1._send_receive(payload)

Directly writing to DPS does work. Bulb turns on and off, but I can't seem to get any status off of it.

This is the output:

DEBUG:TinyTuya [1.12.0]

DEBUG:status() entry (dev_type is zigbee)
DEBUG:building command 10 payload=b'{"gwId":"AAAAAA","devId":"BBBBB","uid":"eb2133twqaqmh93d","t":"1679091603","cid":"CCCCC"}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000010000000a000000886b91e7e8fe625a4c661b84ddae8d6ffd77ce31286de3f3ab9e30fcf13184458b748e0c54753c1e0a351bd1f34b0b29f1f18addc0895fb31a8babe6f28f5cabefac518bf20cc3d19dc2bb09c5a4296be873f14362d0665f86d5f4d9b9d23eac899db678632fa34cb81ac72449a90040e2ea8d24ff7326eca63062a0c3275ac8e9a26bde2a0000aa55'
DEBUG:received data=b'000055aa000000010000000a0000002c00000001f3a0620de89c5266eccc34a2f2c93aabc3e8f0f932022f136daf839a9a3ebb7b6802cdcd0000aa55'
DEBUG:received message=TuyaMessage(seqno=1, cmd=10, retcode=1, payload=b'\xf3\xa0b\r\xe8\x9cRf\xec\xcc4\xa2\xf2\xc9:\xab\xc3\xe8\xf0\xf92\x02/\x13m\xaf\x83\x9a\x9a>\xbb{', crc=1745014221, crc_good=True, prefix=21930, iv=None)
DEBUG:raw unpacked message = TuyaMessage(seqno=1, cmd=10, retcode=1, payload=b'\xf3\xa0b\r\xe8\x9cRf\xec\xcc4\xa2\xf2\xc9:\xab\xc3\xe8\xf0\xf92\x02/\x13m\xaf\x83\x9a\x9a>\xbb{', crc=1745014221, crc_good=True, prefix=21930, iv=None)
DEBUG:decode payload=b'\xf3\xa0b\r\xe8\x9cRf\xec\xcc4\xa2\xf2\xc9:\xab\xc3\xe8\xf0\xf92\x02/\x13m\xaf\x83\x9a\x9a>\xbb{'
DEBUG:decrypting=b'\xf3\xa0b\r\xe8\x9cRf\xec\xcc4\xa2\xf2\xc9:\xab\xc3\xe8\xf0\xf92\x02/\x13m\xaf\x83\x9a\x9a>\xbb{'
DEBUG:decrypted 3.x payload='json obj data unvalid'
DEBUG:payload type = <class 'str'>
DEBUG:decoded results='json obj data unvalid'
DEBUG:ERROR Invalid JSON Response from Device - 900 - payload: "json obj data unvalid"
DEBUG:Recieved async update for wrong CID None while looking for CID None, trying again
DEBUG:status() received data=None
None
DEBUG:building command 7 payload=b'{"t":"1679091608","gwId":"AAAAAA","cid":"CCCCCC","dps":{"1":true,"2":0,"4":1000,"3":100}}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000020000000700000087332e3300000000000000000000000033709b219083dea486766b60cbd2751d4faed815b6122f7f7166dfb9fbb2891ea4c65e142da96861e27a3d752cc0f3615438a5d92eacdfcb75cf65e35e805ccc1fe087177cee5ea9e78e832291a2151d6c8f9fe518aa1e38f39f429b0dc9e1ff9bf08b5a4ce02e2e63eee5cc3b85e2ae94afd28c0000aa55'
DEBUG:received data=b'000055aa00000002000000070000000c0000000018cfc5da0000aa55'
DEBUG:received null payload (TuyaMessage(seqno=2, cmd=7, retcode=0, payload=b'', crc=416269786, crc_good=True, prefix=21930, iv=None)), fetch new one - retry 0 / 5
DEBUG:building command 7 payload=b'{"t":"1679091618","gwId":"AAAAA","cid":"CCCCC","dps":{"1":false,"2":0,"4":1000,"3":100}}'
DEBUG:sending payload
DEBUG:payload encrypted=b'000055aa000000030000000700000097332e330000000000000000000000001c474460231f7e708ab5eac41edb20a84faed815b6122f7f7166dfb9fbb2891ea4c65e142da96861e27a3d752cc0f3615438a5d92eacdfcb75cf65e35e805ccc1fe087177cee5ea9e78e832291a2151d51f8c1fdb8c13fe9834b40077d936be995838416f1503536eb8beb97af898d74bc4366caace41e46a1b4d72880df707cd68e201c0000aa55'
DEBUG:received data=b'000055aa00000003000000070000000c00000000c5591c5f0000aa55'
DEBUG:received null payload (TuyaMessage(seqno=3, cmd=7, retcode=0, payload=b'', crc=3310951519, crc_good=True, prefix=21930, iv=None)), fetch new one - retry 0 / 5

I presume this has to do with the payload not being something recognized, but I don't know how to proceed from here. Any help would be greatly appreciated.

Incidentally, the light is question is : 8W DC10-30V Tuya Bluetooth G4 LED

[bnazari]

To make things even more weird, the bulb isn't even type A... Seems like the color is set through DP 11.

This is off the Tuya IOT API:

"result": {
    "category": "dj",
    "functions": [
      {
        "code": "switch_led",
        "dp_id": 1,
        "type": "Boolean",
        "values": "{}"
      },
      {
        "code": "work_mode",
        "dp_id": 2,
        "type": "Enum",
        "values": "{\"range\":[\"white\",\"colour\",\"scene\",\"music\"]}"
      },
      {
        "code": "bright_value",
        "dp_id": 3,
        "type": "Integer",
        "values": "{\"min\":10,\"max\":1000,\"scale\":0,\"step\":1}"
      },
      {
        "code": "temp_value",
        "dp_id": 4,
        "type": "Integer",
        "values": "{\"min\":0,\"max\":1000,\"scale\":0,\"step\":1}"
      },
      {
        "code": "colour_data_raw",
        "dp_id": 11,
        "type": "Raw",
        "values": "{\"maxlen\":\"255\"}"
      },
      {
        "code": "music_data_raw",
        "dp_id": 13,
        "type": "Raw",
        "values": "{}"
      },
      {
        "code": "do_not_disturb",
        "dp_id": 34,
        "type": "Boolean",
        "values": "{}"
      }
    ],

[uzlonewolf]

Hi @bnazari ! What gateway do you have? None of the 3 I have are doing this, but someone else is also reporting that they have the same problem. I have a couple new BLE/combo gateways coming tomorrow to try and find one which does this.

In the meantime, what happens if you set tle light ID to be the same as the CID? So, lightCID = lightID = "CCCCC"

[bnazari]

Hi @uzlonewolf! So I ended up with 3 of them as well. I was planning on returning anything that didn't work.

MOES ZigBee 3.0 & Bluetooth & Mesh Hub - This one registered on the Tuya app, but then I couldn't even control the bulb via the app. Did an upgrade on the firmware once I registered it. Detected as 3.4 MOES Tuya Bluetooth Gateway - Worked fine with app. Also on 3.4. Going to try again with this one. I believe it detected only DP 34 on it. Will check again with this one as I never tried to directly send commands to the bulb. eMylo Zigbee Hub Gateway, 2 in 1 Zigbee 3.0 Bluetooth Tuya Smart Gateway Didn't update the firmware with this one. Running 3.3. This is the only one I'm currently using.

As far as setting light ID to the the CID, it gave the same "DEBUG:decrypted 3.x payload='json obj data unvalid'" error. It gave mixed results on the DPS commands. Once it was able to turn the light on, but not off. Another run it was able to turn on and off. Having the separate DeviceID and CID seems to give consistent results.

From what I can tell, something is wonky with interpreting the payload. Going to look into that a bit more and see if it is something obvious. Good luck with your hubs tomorrow!

[jasonacox]

I need to get a hub to add to my test rig suite too. Thanks for these links @bnazari .

@uzlonewolf I wonder if we should create a discussion or separate ZIGBEE.md doc to help capture some of the complexities, example (reference) working devices and How-Tos? As a minimum, I'll add a zigbee label for issues.

[uzlonewolf]

Well, the first BLE hub I ordered came in, https://www.amazon.com/dp/B09TZH95FG . Ordered last night and it arrived 8am this morning. Yay Amazon overnight! Unfortunately(?) it does not throw the "unvalid" error and works just fine with the examples/zigbee_gateway.py code.

I went ahead and sniffed the app traffic anyway and interestingly the app sends command with nothing but the cid set. When it first connects it sends a single Ethernet packet containing 3 TuyaMessages:

cmd=16, {"cid":"tuya96########14"}
cmd=16, {}
cmd=64, {"data":{"cids":[]},"reqType":"subdev_online_stat_query"}

To which the device responds with 3 Ethernet packets:

cmd=16, {"dps":{"2":false,"8":"click","9":85,"10":0,"11":"up_on","12":100,"15":0,"17":true,"101":false},"cid":"tuya96########14"}
cmd=16, {"dps":{"4":false,"32":"normal"}}
cmd=64, {"reqType":"subdev_online_stat_report","data":{"online":["tuya96########14"]}}

After that it's the usual request/response. I'll see if I can come up with some code which can send almost-empty commands like that.

[jasonacox]

@uzlonewolf I'm going to order a different hub. What device are you testing? I need to order one of those to test.

[bnazari]

Okay, more updates. The MOES Tuya Bluetooth Gateway, running 3.4 doesn't respond to the direct DPS commands and doesn't decrypt the status payload. I don't want to flood the comments with debug output, unless you want to see it.

Still can't understand what's going on with the payloads coming in. Hopefully will figure out with the new hub.

@jasonacox I can get you one of these bulbs. Ordered mine through through AliExpress G4 RGBW bulb I'm in Pasadena, so if I get it in the mail today, you should have it pretty quickly.

@uzlonewolf Just ordered the same hub as the one you just got. I won't upgrade the firmware, just in case.

Thank you both for the help with all this. Nothing like a weekend puzzle to solve.

[uzlonewolf]

I ordered some of those LEDs off Ali, but as they are going to take like 3-6 weeks to get here I also picked up a https://www.amazon.com/dp/B0BLJN2BWW as it was available for overnight delivery and looked interesting.

I also ordered all 3 hubs @bnazari posted, they just won't get here until next week.

[uzlonewolf]

@bnazari I would like to see the debug output actually! The more verbose the better. Let's see that binary gobblygook monster barf all over the place! :) 50/50 I may need the local key too if it's not the one you posted originally.

[uzlonewolf]

Oh, and for some reason that first hub I got didn't want to show the local key via the wizard. I had to get it via the "pretend to be the app" method.

[jasonacox]

@jasonacox I can get you one of these bulbs. Ordered mine through through AliExpress G4 RGBW bulb I'm in Pasadena, so if I get it in the mail today, you should have it pretty quickly.

That's awesome @bnazari ! Send me your contact info so I can pay you. If you let me buy two, I'll send one to @uzlonewolf - jason @ jasonacox.com.

@uzlonewolf I saw the FingerBot too... might just have to get one too just for the fun of it. ;)

[bnazari]

For the life of me, I couldn't figure out why it was spitting out "json obj data unvalid" Should have taken a hint from the "unvalid" part. Turns out, the payload of the response from the bulb isn't a json, it is a string that says "json obj data unvalid" Kept thinking something is wrong with the decryption part until I did it manually and realized it was a string.

[uzlonewolf]

Alright, here's what I've found so far. That first BLE+ZigBee hub I bought, https://www.amazon.com/dp/B09TZH95FG , came with firmware version 1.3.2 (WBRG1 main module) and 1.1.9 ( ZS3L ZigBee module). It says there is a firmware update available, however it is only for the ZigBee module - not the main module. It is 2.4.1 and can be downloaded at https: images.tuyaus.com/smart/firmware/upgrade/20221208/1670484983-si32_mg21_115200_pta_665_OTA_2.4.1.bin . This firmware uses protocol v3.3 and works just fine with the existing code.

My 2nd hub actually came in around noon (Amazon strikes again!) and is BLE-only. https://www.amazon.com/dp/B097XR41X4 . This one came with firmware version 1.3.2 and only has a single CR3L module. That firmware also uses protocol v3.3 and works just fine with the existing code. There is a firmware upgrade for it to 1.9.9 which can be downloaded at https: images.tuyaus.com/smart/firmware/upgrade/20221109/1667990073-oem_rtl8720cm_gw_plug_UG_1.9.9.bin . Unlike the earlier firmware, this one is protocol v3.4 and does not support status polling. After kludging v3.4 support into my PCAP parser, it turns out the module sends the "unvalid" error to the Smart Life app as well. With that being the case I'm not too hopeful about being able to make it work in TinyTuya. I'll get a bit more invasive tomorrow to see what I can dig up.

[uzlonewolf]

Ha! I decided to throw some shit at the wall to see if I could get it to stick, and I figured out how to tease the status out of it. Working on my v3.4 gateway:

import time
import tinytuya

tinytuya.set_debug(True)

tinytuya.payload_dict['v3.4'][tinytuya.DP_QUERY]["command"] = {"protocol":4, "t": "int", "data": {}}

gw = tinytuya.Device( 'eb6######', address='172.20.#.#', persist=True, version=3.4 )
zigbee1 = tinytuya.Device( 'ebb0######', cid='tuya96########14', parent=gw )

print( gw.status() )
print( zigbee1.status() )

print(" > Begin Monitor Loop <")
pingtime = time.time() + 9
polltime = time.time() + 3

while(True):
    if( pingtime <= time.time() ):
        payload = gw.generate_payload(tinytuya.HEART_BEAT)
        data = gw.send(payload)
        pingtime = time.time() + 9
        if data:
            print( data )

    if( polltime <= time.time() ):
        polltime = time.time() + 3
        print( '================' )
        print( zigbee1.status() )

    data = gw.receive()

    if data:
        print( data )

I'll get a proper PR written tomorrow.

[jasonacox]

Oh, and for some reason that first hub I got didn't want to show the local key via the wizard. I had to get it via the "pretend to be the app" method.

The hub that I just picked up, eMylo 5.0 Bluetooth&3.0 ZigBee Gateway 3 In 1 (v3.4 device, Firmware: Main Module: v1.4.8 ZigBee Module: v2.4.1), has the same issue. Wizard didn't pick up the local key (not in tuya-raw), but I was able to get it the "app" way as you mention. I looked at the "result" raw from the cloud.getdevices() call and there is no local_key.

(edit) I was able to get the local_key using the https://openapi.tuyaus.com/v1.1/iot-03/devices/{DeviceID} call.

import tinytuya
c = tinytuya.Cloud()

# fetch device with local_key
device = c._tuyaplatform(ver='v1.1', uri='iot-03/devices/xxxxxx')

We could add a loop to Wizard to try to poll the individual devices that are missing their local_key. I also wonder if there is an updated v1.1 way to fetch all devices (currently using /v1.0/iot-01/associated-users/devices or /v1.0/users/{user}/devices).

[jasonacox]

(continued) Using "API Explorer" the standard URI to pull user devices does show the local_key for my gw device, but c.getdevices() is clearly not getting it. The API curlcall that works from the Explorer does show the local_key. I can't figure out what I missing in the Wizard code that isn't replicating this unless the way we are computing token/signature is off.

curl --request GET "https://openapi.tuyaus.com/v1.0/users/{USER}/devices" \ --header "sign_method: HMAC-SHA256" \ --header "client_id: {API_KEY}" \ --header "t: 1679268929387" \ --header "mode: cors" \ --header "Content-Type: application/json" \ --header "sign: {SIGNATURE}" \ --header "access_token: {TOKEN}"

[uzlonewolf]

getdevices() uses /v1.0/iot-01/associated-users/devices which is under the API Explorer as "Smart Home Device System" -> "Device Management" -> "Batch query for the list of associated App user dimension devices" and for me is not returning the local key either.

[jasonacox]

Ok - I think this is overloading this Issue. I'm moving the Wizard issue to a new issue #305 .