jasonacox / tinytuya

Python API for Tuya WiFi smart devices using a direct local area network (LAN) connection or the cloud (TuyaCloud API).
MIT License
956 stars 172 forks source link

DpID only available from cloud, not found locally, fixable? #391

Open backcountrymountains opened 1 year ago

backcountrymountains commented 1 year ago

I have a tuya mini-split AC/Heat pump. Everything is working great except for the energy measurement.

In the logs on iot.tuya.com, I can access the energy values from DpID 116. However, I have not found a way to query DpID 116 locally, even using set_dpsUsed({"116": None}) or updatedps(index=[116], nowait=False))

Is it possible that only the cloud service can query that DpID? How can I get the data from the device?

My device logs are here

I also posted a comment here where someone had what I thought was a related issue. I used tinytuya to monitor the status of the device during large changes in power usage and found no changes in any of the DpIDs that would seem to correlate to any kind of power measurement.

It is also possible to go to iot.tuya.com->Device Control->Query Properties and input my device_id and the code totalP (found below in Data Model) and get the power reading as follows:

{
  "result": {
    "properties": [
      {
        "code": "totalP",
        "custom_name": "",
        "dp_id": 116,
        "time": 1690487036000,
        "value": 246
      }
    ]
  },
  "success": true,
  "t": 1690487113389,
  "tid": "1a63ac572cb611ee8c70fe98ebe72f12"
}

I also did the Query Things Data Model on iot.tuya.com and translated the output.

result.model,success,t,tid
"{""modelId"":""000003lixy"",""services"":[{""actions"":[],""code"":"""",""description"":"""",""events"":[],""name"":""default service"",""properties"":[{""abilityId"":1,""accessMode"" :""rw"",""code"":""Power"",""description"":"""",""extensions"":{""iconName"":""icon-dp_power"",""attribute"":""5""},""name"":""switch"",""typeSpec"":{""type"":""bool"",""typeDefaultValue "":false}},{""abilityId"":2,""accessMode"":""rw"",""code"":""temp_set"",""description"":""Celsius: 16~31
Fahrenheit: 61~88"",""extensions"":{""iconName"":""icon-dp_temp""},""name"":""set temperature"",""typeSpec"":{""max"":880,""min"":160,""scale"":1,""step"":5,""type"":""value",""typeDefault Value"":160,""unit"":""℃""}},{""abilityId"":3,""accessMode"":""ro"",""code"":""temp_current"",""description"":"""",""extensions"":{""iconName"":""icon-dp_sun""},""name"":""Current Temperature"",""typeSpec"":{""max"":100,""min"":-20,""scale"":0,""step"":1,""type"":""value"",""typeDefaultValue"":-20,""unit"":""℃""}},{""abilityId"":4,""accessMode"":""rw"","" "code"":""mode"",""description"":"""",""extensions"":{""iconName"":""icon-dp_mode"",""attribute"":""4""},""name"":""working mode"",""typeSpec"":{""range"":[""cold"",""hot"",""wet"","" "wind"",""auto""],""type"":""enum"",""typeDefaultValue"":""cold""}},{""abilityId"":5,""accessMode"":""rw"",""code"":""windspeed"",""description"":""Strong/high wind/medium high wind/middle wind/medium low wind/low wind/mute/ Automatic"",""extensions"":{""iconName"":""icon-dp_wind"",""attribute"":""4""},""name"":""wind speed"",""typeSpec"":{""range"":[""strong"",""high"",""mid_high"",""mid"",""mid_low"","" low"",""mute"",""auto""],""type"":""enum"",""typeDefaultValue"":""strong""}},{""abilityId"":18,""accessMode"":""rw"",""code"":""humidity_current"",""description"":"""",""extensions"": {""iconName"":""icon-dp_wet""},""name"":""current humidity"",""typeSpec"":{""max"":100,""min"":0,""scale"":0,""step"":1,""type"":""value"",""typeDefaultValue"":0,""unit"":""%""}},{" "abilityId"":20,""accessMode"":""ro"",""code"":""Fault"",""description"":"""",""extensions"":{""iconName"":""icon-dp_warming"",""scope"":""fault""},""name"":""Fault Warning"",""typeSpec"":{ ""label"":[""E0"",""E1"",""E2"",""E3"",""E4"",""E5"",""E6"",""E7"",""E8"",""E9"",""EA"",""Eb"",""EC"",""EE"",""EF"",""EP"",""EU"","" "EH"",""EJ"",""En"",""Ey"",""F9"",""FA"",""H1"",""H2"",""P0"",""P1"",""P2"",""P4"",""P5""],""maxlen"":30,""type"":""bitmap"",""typeDefaultValue"": 0}},{""abilityId"":101,""accessMode"":""ro"",""code"":""pm25"",""description"":"""",""name"":""PM2.5"",""typeSpec"":{""max"":3000,""min"":0,""scale"":0,""step"":1,"" "type"":""value"",""typeDefaultValue"":0,""unit"":""""}},{""abilityId"":105,""accessMode"":""rw"",""code"":""sleep"",""description"":""none,standard,elderly,children"",""name"":""sleep"",""typeSpec"": {""range"":[""off"",""normal"",""old"",""child""],""type"":""enum"",""typeDefaultValue"":""off""}},{""abilityId"":110,""accessMode"":""ro"",""code"":""markbit"",""description"":"" To indicate whether this function is available.
0. Is the temperature adjustable in dehumidification mode?
1. Is the temperature adjustable in the air supply mode?
2. Is the temperature adjustable in automatic mode?
3. Fresh air volume identification
4. Vector air supply
5. Sweep left and right
6. Photosensitive
7. Intelligent dehumidification and mildew prevention
8. Humidity sensor
9. Evaporator cleaning
10. Save money and see it
11. Power Statistics
12. Generator mode
13. High temperature wind/cool wind
14. Air quality detection function
15. Set to empty (formerly: humidity function)
16. Set it to empty (formerly: equipment operation saves money and can be seen, temperature curve display)
17, 8 ℃ heating
18. Dirty and blocked filter function
20. Whether there is PM2.5
21. Temperature scale switch, 1 is Fahrenheit, 0 is Celsius
22. soft wind
23. Left and right wide-angle air supply
"",""extensions"":{""scope"":""fault""},""name"":""Identifier"",""typeSpec"":{""label"":[""0"",""1"",""2"",""3"",""4"",""5"",""6"",""7"",""8"",""9"",""10 "",""11"",""12"",""13"",""14"",""15"",""16"",""17"",""18"",""19"",""20"",""21"",""22"",""23""],""maxlen"":24,""type"":""bitmap"",""typeDefa ultValue"":0}},{""abilityId"":113,\"accessMode\":\"rw\",\"code\":\"up_down_sweep\",\"description\":\"None/up and down air supply/up air supply/down air supply\", \"name\":\"Sweep up and down\",\"typeSpec\":{\"range\":[\"0\",\"1\",\"2\",\"3\ "],\"type\":\"enum\",\"typeDefaultValue\":\"0\"}},{\"abilityId\":114,\"accessMode\":\"rw\", \"code\":\"left_right_sweep\",\"description\":\"None/Left and Right Sweep/Left Sweep/Middle Sweep/Right Sweep/Left Sweep/Right Sweep/Wide Angle Sweep Wind\",\"name\":\"Sweep left and right\",\"typeSpec\":{\"range\":[\"0\",\"1\",\"2\", \"3\",\"4\",\"5\",\"6\",\"7\"],\"type\":\"enum\",\"typeDefaultValue\":\"typeDefaultValue\":\" "0\"}},{\"abilityId\":115,\"accessMode\":\"ro\",\"code\":\"totalN\",\"description\":\"\" ,\"extensions\":{\"trigger\":\"direct\"},\"name\":\"Power Integer\",\"typeSpec\":{\"max\":1000000,\ "min\":0,\"scale\":0,\"step\":1,\"type\":\"value\",\"typeDefaultValue\":0,\"unit\":\"type\":\"value\",\"typeDefaultValue\":0,\"unit\":\ "\"}},{\"abilityId\":116,\"accessMode\":\"ro\",\"code\":\"totalP\",\"description\":\"\", \"extensions\":{\"trigger\":\"direct\"},\"name\":\"Power Decimal\",\"typeSpec\":{\"max\":1000000,\" min\":0,\"scale\":0,\"step\":1,\"type\":\"value\",\"typeDefaultValue\":0,\"unit\":\" \"}},{\"abilityId\":119,\"accessMode\":\"rw\",\"code\":\"money\",\"description\":\"None/default Electricity/Rated Power Saving/Fixed Temperature Power Saving\",\"name\":\"See if you can save money\",\"typeSpec\":{\"range\":[\"0\",\" 1\",\"2\",\"3\"],\"type\":\"enum\",\"typeDefaultValue\":\"0\"}},{\"abilityId\": 120,\"accessMode\":\"rw\",\"code\":\"energy\",\"description\":\"\",\"name\":\"generator mode\" ,\"typeSpec\":{\"range\":[\"off\",\"L1\",\"L2\",\"L3\"],\"type\":\"enum\ ",\"typeDefaultValue\":\"off\"}},{\"abilityId\":122,\"accessMode\":\"ro\",\"code\":\"fault2\",\ "description\":\"\",\"extensions\":{\"scope\":\"fault\"},\"name\":\"Fault Alarm 2\",\"typeSpec\": {\"label\":[\"P6\",\"P7\",\"P8\",\"P9\",\"PA\",\"F0\",\"F1\", \"F2\",\"F3\",\"F4\",\"F5\",\"F6\",\"F7\",\"F8\",\"Fb\",\" FC\",\"FE\",\"FF\",\"FH\",\"FP\",\"FU\",\"Fj\",\"Fn\",\"Fy\" ",\"bf\",\"bc\",\"bj\"],\"maxlen\":27,\"type\":\"bitmap\",\"typeDefaultValue\":0}} ,{\"abilityId\":123,\"accessMode\":\"rw\",\"code\":\"boolCode\",\"description\":\"two bytes,\\n The first byte: \\nbit0: eco,\\nbit1: intelligent dehumidification and anti-mildew, \\nbit2: evaporator cleaning, \\nbit3: light, \\nbit4: buzzer, \\nbit5: health,\ \nbit6: empty (original clean, repeat), \\nbit7: photosensitive. \\n\\nThe second byte: \\nbit0: dry, \\nbit1: empty (originally high temperature wind/cool wind), \\nbit2: empty (originally horizontal wind swing), \\nbit3 : Empty (originally vertical wind swing) \\nbit4: 8°C heating\\nbit5: Filter detection\\nbit6: Fahrenheit plus 1 flag\\nbit7: Soft wind\",\"name\":\" Boolean\",\"typeSpec\":{\"maxlen\":255,\"type\":\"string\",\"typeDefaultValue\":\"\"}},{\"abilityId\ ":125,\"accessMode\":\"ro\",\"code\":\"airquality\",\"description\":\"Excellent/Good/Medium/Poor/Very Poor/Very Poor\ ",\"name\":\"Air Quality\",\"typeSpec\":{\"range\":[\"great\",\"good\",\"middle\",\"bad \",\"verybad\",\"veryverybad\"],\"type\":\"enum\",\"typeDefaultValue\":\"great\"}},{\"abilityId\":126 ,\"accessMode\":\"rw\",\"code\":\"up_down_freeze\",\"description\":\"Current position freeze frame/up freeze frame/up freeze frame/middle freeze frame/down freeze frame /down frame/\",\"name\":\"up and down frame\",\"typeSpec\":{\"range\":[\"0\",\"1\",\"2\ ",\"3\",\"4\",\"5\"],\"type\":\"enum\",\"typeDefaultValue\":\"0\"}},{\" abilityId\":127,\"accessMode\":\"rw\",\"code\":\"left_right_freeze\",\"description\":\"Current position freeze/left freeze/left freeze/center Freeze/Right Freeze/Right Freeze/Wide-angle Freeze/Left Wide-angle Freeze/Right Wide-angle Freeze\",\"name\":\"Left and Right Freeze\",\"typeSpec\":{\"range\":[\ "0\",\"1\",\"2\",\"3\",\"4\",\"5\",\"8\",\"6\",\"7 \"],\"type\":\"enum\",\"typeDefaultValue\":\"0\"}},{\"abilityId\":128,\"accessMode\":\"ro\" ,\"code\":\"style\",\"description\":\"On-hook/cabinet machine\",\"name\":\"Type\",\"typeSpec\":{\" range\":[\"0\",\"1\"],\"type\":\"enum\",\"typeDefaultValue\":\"0\"}},{\"abilityId\" :129,\"accessMode\":\"rw\",\"code\":\"kwh\",\"description\":\"Optional 1~5kwh\",\"name\":\ "Power\",\"typeSpec\":{\"range\":[\"1\",\"2\",\"3\",\"4\",\"5\"], \"type\":\"enum\",\"typeDefaultValue\":\"1\"}},{\"abilityId\":130,\"accessMode\":\"rw\",\"code \":\"savemoney_temp\",\"description\":\"Used to save money and see the constant temperature and power saving temperature settings\",\"name\":\"fixed temperature\",\" typeSpec\":{\"max\":31,\"min\":26,\"scale\":0,\"step\":1,\"type\":\"value\",\ "typeDefaultValue\":26,\"unit\":\"℃\"}},{\"abilityId\":131,\"accessMode\":\"ro\",\"code\":\" dirty_filter\",\"description\":\"\",\"name\":\"filter dirty and clogged\",\"typeSpec\":{\"type\":\"bool\",\ "typeDefaultValue\":false}},{\"abilityId\":132,\"accessMode\":\"rw\",\"code\":\"hot_cold_wind\",\"description\":\" \",\"name\":\"High temperature wind/cool wind\",\"typeSpec\":{\"type\":\"bool\",\"typeDefaultValue\":false}},{ \"abilityId\":133,\"accessMode\":\"rw\",\"code\":\"wind\",\"description\":\"0: horizontal vertical off\\n1: horizontal On Vertical Off\\n2: Horizontal Off Vertical On\\n3: Horizontal and Vertical On\",\"name\":\"Horizontal Swing Wind/Vertical Swing Wind\",\"typeSpec\":{\"range\ ":[\"0\",\"1\",\"2\",\"3\"],\"type\":\"enum\",\"typeDefaultValue\":\"0\ "}},{\"abilityId\":134,\"accessMode\":\"ro\",\"code\":\"work_time\",\"description\":\"\",\" name\":\"running state time\",\"typeSpec\":{\"maxlen\":255,\"type\":\"string\",\"typeDefaultValue\":\"\"} },{\"abilityId\":135,\"accessMode\":\"ro\",\"code\":\"run_time\",\"description\":\" is used for running time accumulation, each Report once every 2 minutes. So the running time is 2*times\",\"name\":\"running time\",\"typeSpec\":{\"max\":65525,\"min\":0,\"scale\ ":0,\"step\":1,\"type\":\"value\",\"typeDefaultValue\":0,\"unit\":\"time\"}},{\"abilityId \":136,\"accessMode\":\"rw\",\"code\":\"temp_set_f\",\"description\":\"\",\"extensions\":{\"iconName \":\"icon-dp_temp\",\"attribute\":\"4\"},\"name\":\"Temperature Setting-℉\",\"typeSpec\":{\"max\ ":88,\"min\":61,\"scale\":0,\"step\":1,\"type\":\"value\",\"typeDefaultValue\":61,\" unit\":\"℉\"}}]}]}"
   }

I tried to get the markbit information but it came back with "2228796" which is 28 bits instead of 24 so I don't know what I'm supposed to do with that.

{
  "result": {
    "properties": [
      {
        "code": "markbit",
        "custom_name": "",
        "dp_id": 110,
        "time": 1690491443689,
        "value": 2228796
      }
    ]
  },
  "success": true,
  "t": 1690492729478,
  "tid": "2dcbd0792cc311ee9fecaa50f188a4b0"
}

Anyway, this seemed like a unique issue so I'm looking for guidance here.

uzlonewolf commented 1 year ago

Are you opening a persistent connection to catch asynchronous updates? Some DPs are only broadcasted when the device feels like it, you cannot query them directly.

it came back with "2228796" which is 28 bits

How do you figure? The 22nd bit (bit 21) is the highest bit set in that number. 2228796 = 00100010 00000010 00111100

backcountrymountains commented 1 year ago

I used the monitor.py script and it did occasionally show a log for temperature readings that was not queried in addition to the result of status queries. But I didn't get any result using d.generate_payload(tinytuya.UPDATEDPS,['116']) or d.generate_payload(tinytuya.UPDATEDPS) (116 is the DpID for "totalP" that shows some sort of power analog on iot.tuya.com)

Received Payload: {'devId': '75767832c45bbeda0fa3', 'dps': {'1': False, '2': 670, '3': 22, '4': 'cold', '5': 'high', '18': 0, '20': 0, '101': 0, '105': 'off', '110': 2228796, '113': '0', '114': '0', '119': '0', '120': 'off', '123': '0018', '125': 'great', '126': '0', '127': '0', '128': '0', '129': '1', '130': 26, '131': False, '132': False, '133': '0', '134': '{"t":1690814535,"s":false,"clr":true}'}}

Apparently I don't know how bits work. Mapping your bits to the table from Query Things Data Model (which doesn't include a "19" for some reason) gives:

Bit Index Description
0 0 Is the temperature adjustable in dehumidification mode?
0 1 Is the temperature adjustable in the air supply mode?
1 2 Is the temperature adjustable in automatic mode?
0 3 Fresh air volume identification
0 4 Vector air supply
0 5 Sweep left and right
1 6 Photosensitive
0 7 Intelligent dehumidification and mildew prevention
0 8 Humidity sensor
0 9 Evaporator cleaning
0 10 Save money and see it
0 11 Power Statistics
0 12 Generator mode
0 13 High temperature wind/cool wind
1 14 Air quality detection function
0 15 Set to empty (formerly: humidity function)
0 16 Set it to empty (formerly: equipment operation saves money and can be seen, temperature curve display)
0 17 8 ℃ heating
1 18 Dirty and blocked filter function
1 19 ---not listed in result?---
1 20 Whether there is PM2.5
1 21 Temperature scale switch, 1 is Fahrenheit, 0 is Celsius
0 22 soft wind
0 23 Left and right wide-angle air supply

This doesn't totally reconcile with what I can get from my device; I don't have #20. pm2.5 or #14. Air quality available in the iot.tuya.com logs of my device, but I do have #11 Power in the online logs.

I don't see anything about power in the app. How technically challenging is it to sniff what the device is sending to iot.tuya.com? Is there a guide for setting up MITM?

uzlonewolf commented 1 year ago

If it only updates asynchronously then sending UPDATEDPS will not do anything, you can only wait for the device to broadcast it. I have never seen a device which uploaded a DP to the cloud but did not broadcast it to the LAN.

As for those bits, you start counting from the right (bit 0 is the right-most bit):

Bit Index Description
0 0 Is the temperature adjustable in dehumidification mode?
0 1 Is the temperature adjustable in the air supply mode?
1 2 Is the temperature adjustable in automatic mode?
1 3 Fresh air volume identification
1 4 Vector air supply
1 5 Sweep left and right
0 6 Photosensitive
0 7 Intelligent dehumidification and mildew prevention
0 8 Humidity sensor
1 9 Evaporator cleaning
0 10 Save money and see it
0 11 Power Statistics
0 12 Generator mode
0 13 High temperature wind/cool wind
0 14 Air quality detection function
0 15 Set to empty (formerly: humidity function)
0 16 Set it to empty (formerly: equipment operation saves money and can be seen, temperature curve display)
1 17 8 ℃ heating
0 18 Dirty and blocked filter function
0 19 ---not listed in result?---
0 20 Whether there is PM2.5
1 21 Temperature scale switch, 1 is Fahrenheit, 0 is Celsius
0 22 soft wind
0 23 Left and right wide-angle air supply
uzlonewolf commented 1 year ago

As for MITM-ing the cloud, it's not for the faint of heart. Tuya uses SSL/TLS in PSK mode and you need to get a full-chip firmware dump of your device to get the key to decrypt it. Every device has a different PSK loaded at the factory and so if you have multiple devices you must dump every single one of them. Devices will reject any SSL/TLS negotiation that does not contain the device-specific PSK. Once you have extracted the PSK from the firmware dump it's not too difficult and you can use it to decrypt packet captures without needing to do a full MITM connection intercept.

backcountrymountains commented 1 year ago

Well those bits align pretty well. All I get using monitor.py for asynchronous updates is: {'devId': 'xxxxxxxxxxxxxxxxxxxxxxx', 'dps': {'3': 22}, 't': 1690819410} (DpID 3 is temperature in °C)```

Is there a guide for getting the firmware dump? I've messed around with ESPhome chips and have a FTDI uart to usb thing.

uzlonewolf commented 1 year ago

Hmm, looking at the bit list I wonder if setting bit 11 (turning 2228796 into 2230844) would cause it to start sending updates. d.set_value("110", 2230844)

As for dumping the firmware, it is going to depend on which chip the device uses. I have devices with ESP, Realtek, and Beken chips, so there is no one-size-fits-all guide. Beken is pretty well supported with the OpenBeken project, Realtek is not supported at all AFAIK. I'd start by Google-ing "dump <your chip part #> firmware"

backcountrymountains commented 1 year ago

I don't seem able to change DpID 110. In the jumble-fudge of invalid json I got from iot.tuya.com it says {""abilityId"":110,""accessMode"":""ro"",""code"":""markbit"", which makes me think the value is read-only. However, I also can't change DpID 119, the "money" code, that I think should also enable power logging and it's "rw". So I have no idea.

I still don't understand how iot.tuya.com is getting values every 2 minutes almost exactly:

Time Device Event DP ID Event Details Source Source Details
2023-08-01 13:53:03 Report current temperature 25℃ device itself  
2023-08-01 13:51:58 Report 电量小数 246 device itself  
2023-08-01 13:51:58 Report 运行时间 1次 device itself  
2023-08-01 13:50:01 Report 电量小数 246 device itself  
2023-08-01 13:50:01 Report 运行时间 1次 device itself  
2023-08-01 13:50:00 Report current temperature 25℃ device itself  
2023-08-01 13:48:04 Report 电量小数 246 device itself  
2023-08-01 13:48:04 Report 运行时间 1次 device itself

运行时间 is:

{\"abilityId\":135,\"accessMode\":\"ro\",\"code\":\"run_time\",\"description\":\" is used for running time accumulation, each Report once every 2 minutes. So the running time is 2*times\",\"name\":\"running time\"

Might have to crack it open and see what I can find. My router says that the device is "ESP_DA0FA3 - 192.168.1.xx" so I'm guessing it's an ESP.

Thanks for the help.

jasonacox commented 1 year ago

I still don't understand how iot.tuya.com is getting values every 2 minutes almost exactly

From my experience, Tuya devices are cloud-first designed. They also have a local access that we exploit for TinyTuya and the SmartLIfe app uses for faster control, but it seems clear to me that their firmware directs them to send updates to the cloud with highest priority.

Let us know what you find if you decided to crack it open. 😄

Thanks!