search

codetheweb / tuyapi

🌧 An easy-to-use API for devices that use Tuya's cloud services. Documentation: https://codetheweb.github.io/tuyapi.
MIT License
2.04k stars 332 forks source link

Support 0d devices #389

Open msillano opened 3 years ago

msillano commented 3 years ago

I'm using tuyaApi in node-red, via node node-red-contrib-tuya-smart-device. I have 20+ Tuya devices, like switches, sensors, ad some ZigBee devices. All devices work well in smartLife app. The node-red actual comportment is: 1) All devices (except some sensors) accepts "SET" commands:

Example, using a real device, a switch, the command (sets the switch on for 10 s) : payload: { multiple: true, data: { 1: true, 102: 10}} produces the answer: 12 Dec 11:50:26 - [info] [tuya-smart-device:fe3fbeb2.ac31f] Data from device: "{"dps":{"1":true,"102":10},"t":1607770225}"

Using a virtual device, i.e. a radiator controller and ZigBee hub, the command (sets the target temperature) payload: { "devId": "60a423fffeb5b90d", "dps": 103, "set": 220 } produces the answer: 12 Dec 12:06:24 - [info] [tuya-smart-device:a9ea9736.7483f8] Data from device: "{"dps":{"103":220},"cid":"60a423fffeb5b90d","type":query,"t":1607771183}"

  1. I get a message also when some values are changed in smartLife, e.g.:

12 Dec 12:14:08 - [info] [tuya-smart-device:fe3fbeb2.ac31f] Data from device: "{"dps":{"1":true},"t":1607771648}" 12 Dec 12:16:12 - [info] [tuya-smart-device:a9ea9736.7483f8] Data from device: "{"dps":{"101":47},"cid":"00158d00056e5022","t":1607771771}"

  1. I get also messages in case of an event on the device, or from the sensors (without any request):

Switch countdown end: 12 Dec 11:50:36 - [info] [tuya-smart-device:fe3fbeb2.ac31f] Data from device: "{"dps":{"1":false,"102":0},"t":1607770235}" Temperature from a ZigBee sensor, about every 3600s (request by Tuya cloud?): 12 Dec 12:14:28 - [info] [tuya-smart-device:a9ea9736.7483f8] Data from device: "{"dps":{"103":215},"cid":"60a423fffeb5b90d","t":1607771667}"

It sounds good, so all ok? Sorry, not really.

  1. Any 'GET' command in any device (real, virtual, ...) in any form, don't work! Always I get:

12 Dec 11:32:32 - [error] [tuya-smart-device:fe3fbeb2.ac31f] json obj data unvalid

This is a big problem because I cannot poll any RO device data points, so some parameters are unreachable.

(see also issue #246 and issue #23) I tryed also a tuyapi update to version 6, but no changes.

I found it very strange that the answer to a 'SET' command from node-red is always handled well, and the answer to a 'GET' command gets always an error (but the answer is well processed if the 'GET' is from the cloud).

Any idea?

Best regards m.sillano

Apollon77 commented 3 years ago

There are tuya devices that do not allow to query values via GET ... this is an device issue ... not this library. So they can only have values when they send them by themself. sorry

codetheweb commented 3 years ago

There are tuya devices that do not allow to query values via GET ... this is an device issue ... not this library. So they can only have values when they send them by themself. sorry

I don't think this is a thing - at least, I haven't ran into it.

What protocol version are you using? Have you tried both 3.1 and 3.3?

Might be related: https://github.com/codetheweb/tuyapi/issues/234

Apollon77 commented 3 years ago

I also saw that from my users. The devices only push data, not possible to GET. Protocal was 3.3 in my examples when I remember correctly (protocol came from proxy sync still, so from official Tuya data)

msillano commented 3 years ago

Thanks for the comments... I understand: the capacities of commercial devices are minimal. I will test further on all my tuya devices.

But we have anyway a problem: the handling of the unallowed operations by device+tuyApi.

Example of unallowed operation:

So a correct flow in tuyapi can be:

  1. tests the JSON parameters for formal correctness, not for semantics, on input node data
    • If data are a not valid JSON structure: ABORT, message "json obj data unvalid"
    • if data valid: transmit data to device
  2. on device answer
    • no answer: quiet exit. no data output (maybe a INFO ?)
    • correct data: returns JSON data

    • response exists, but don't give a valid JSON structure: INFO with data hex dump + quiet exit, no data in output.

      Abut "response exists, but don't produce a valid JSON structure" I fond that: device (sirene) response to a change (type) done by smartLive:

data: "3.34��o0a�1� ���~q����Q�S↵����7nKC�K������→����O��� ��6EU: ̤sG�x��tpa" deviceId: "42027807d8bfc0c5831e" deviceName: "Sirena" excode: "332e33000000000000003400000001fdfd6f300861fd31fd20fdfdfd7e161610180671fdfdfdfd0451fd530afdfdfdfd1b376e4b43fd4b60fd6002fdfdfdfdfd09fdfdfdfd4ffdfd1efd20fdfd3645553a0c247347fd78fdfd747061"

The field "excode" is added by code: const buf = Buffer.from(msg.payload.data, 'ascii'); msg.payload.excode =buf.toString("hex");

Very strange is the presence of many 0x00 bytes in a JSON value (not visible in the string !) On same device, if I send a tentative {"operation":"SET", "dps":1, "set":1} via node-.red:

data: "8e�cH[��*����QPjp|a�ttDk{��" deviceId: "42027807d8bfc0c5831e" deviceName: "Sirena" excode: "183865fd63485bfdfd2afdfdfdfd51506a707c61fd0e747410441e6b7bfdfd15"`

crazy behavior :)

Best regards m.s.

codetheweb commented 3 years ago

I think there might be a misunderstanding. json obj data unvalid is sent from the device, not TuyAPI.

But there's a couple issues with doing it this way. First, silently failing is very rarely the best option; if an error occurs the user needs to know about it. More importantly, TuyAPI doesn't really have a way to check if a given payload is valid or not. There's no known schema to check against.

msillano commented 3 years ago

Thanks, codetheweb:, for your message.

1) I know that "json obj data unvalid" is sent from the device... But I said: device+tuyApi. A good API must overcome device limits, bugs, quirks. etc, to do a regular and optimal behavior to API users.

2) I know that API "does not know schema" (really, the general schema is known, the unknown is the schema used by a specific device). So the test on input must be only a validation test, not semantic against some specific schema ( i.e. the test checks only if the input object is a JSON object "well-formed").

3) consistency on API output and on error handling makes a better API, more easy to use. Remember: we have anyway a "silent failing" in case of extra data or bad DPS (on SET)! (i.e. in case of not correct JSON against the specific schema). But this is the same as using a 'GET' on a device not implementing it, so same handling!

Rules:

  1. If the input is not a well Formed JSON structure --> ERROR "JSON obj not well-formed" (or some like this) (bad input parms : API aborts)
  2. In the API output, the DPS field (if they exist) MUST always contain the set/get data point values (the new device status
  3. In case of device error (no output at all, no JSON output, messages like "json obj data unvalid", not standard behavior, etc.)
    • The API is without output, and sends an ERROR/INFO/DUMP trace messages,
    • As an alternative, the API output presents the ERROR/INFO/DUMP message on a dummy JSON output, in special EXTRA fields, which may be useful for a specific recovery process.

So the programmer can write always: if (output.payload.data.dps[1] !== undefined) status.on = output.payload.data.dps[1];

It is important to consider the difference between development (programming) errors, where the messages are helpful to the programmer, and runtime errors: in this case a "silent failing" is a valid option.

In hope to help create better tuyapi. :)

Best Regards m.s.

codetheweb commented 3 years ago

( i.e. the test checks only if the input object is a JSON object "well-formed").

I'm having a hard time understanding what you mean by this, could you give an example?

It is important to consider the difference between development (programming) errors, where the messages are helpful to the programmer, and runtime errors: in this case a "silent failing" is a valid option.

Agreed, but hard to implement. Maybe could configure behavior based on NODE_ENV=[production|prod] if needed.

In general, this package is supposed to be a more low-level driver than a comprehensive solution. I'm happy for additional packages to be built on top of this that include built in device schemas, retry functionality, computed state, etc.

msillano commented 3 years ago

The tuyapi ver. "6.1.1" introduces two new options:

Putting both to false is a partial solution to commands (like 'GET' and 'schema') not handled by some devices and occasionally reported by the device using the misunderstandly "json obj data invalid".

The only open issue is the 'ERROR' level in case of connection loss: using retry this event is not an ERROR requiring programmer intervention, better if it becomes a 'warning' (errors: "Error from socket", "find() timed out. Is the device powered on and the ID or IP correct?").

But I'm using the "node-red-contrib-tuya-smart-device" ver. 1.2.1, which uses tuyapi 5.3.1. While waiting for an update, my personal solution is to modify "node_modules / tuyapi / index.js" (5.3.1) so:

1) comment out lines 317-332: the complete block: for (const packet of packets). 2) comment out line 404: this.get (); 3) changed line 623 to: throw new Error (` Retry connection: ID $ {this.device.id} `); more informative.

Best regards m.s.

codetheweb commented 3 years ago

The only open issue is the 'ERROR' level in case of connection loss: using retry this event is not an ERROR requiring programmer intervention, better if it becomes a 'warning' (errors: "Error from socket", "find() timed out. Is the device powered on and the ID or IP correct?").

I'm a little confused on what you're saying here. Are you suggesting that TuyAPI should automatically attempt to re-connect after a 'bad' (user did not ask for it) disconnect?

msillano commented 3 years ago

That is the normal behavior, having many devices: some can disconnect for many reasons, like power down, interferences etc. The nodes using tuyapi use an infinite retry strategy: the loss of connection is an event, handled by the application, not a fatal error.

Better if Tuyapi doesn't throw an error in case of 'bad' disconnect, but only info or warning.

                  29 Dec 09:17:31 - [info] [tuya-smart-device:ba1d97f9.86c698] Cannot find the device, re-trying...
                  29 Dec 09:17:32 - [info] [tuya-smart-device:ba1d97f9.86c698] initiating the find command
                  29 Dec 09:17:42 - [error] [tuya-smart-device:ba1d97f9.86c698] Error: Rety connection: ID 56685573d8bfc0508837
                  29 Dec 09:17:42 - [info] [tuya-smart-device:ba1d97f9.86c698] Cannot find the device, re-trying...
                  29 Dec 09:17:43 - [info] [tuya-smart-device:ba1d97f9.86c698] initiating the find command

Changing argument, how about protocol version 3.3 ? (see previous post).

best regards m.s.

presidentio commented 3 years ago

It looks like I found the solution. You need to send just { cid: 'virtual_device_id' } got get all dps. virtual_device_id can be grabbed from any response

Apollon77 commented 3 years ago

@presidentio interesting ... but what you mean with "can be grabbed from any response"? Is it also in the "discovery telegram"? or do we need to get at least one "pushed telegram"? is that cid identically to the devId or gwId?

presidentio commented 3 years ago

@Apollon77 To get cid you can connect to any device with enabled debug logging and update any state manually on the device or through the mobile application. In the logs you will see a json object. One of its fields is cid

Apollon77 commented 3 years ago

Sure, but I search for an automated way of getting it ;-)

That's why my questions. Did you checked if the value is the same as devId or gwId?

msillano commented 3 years ago

Presentio > It looks like I found the solution. You need to send just { cid: 'virtual_device_id' }

That about virtual-devices, i.e. sub-devices, linked to tuya via a gateway. I found, using a thermostat (SEA801-ZIGBEE Programmable Thermostat), a Zigbee gateway and tuya-smart-device using tuyapi6.1.1:

1) Changing the target temperature via smartLife app, I catch on tuya-smart-device the message:

       payload: object
         deviceId: "12373b1b789b5994cro7p"
         deviceName: "Zigbee Gateway"
         data: object
              cid: "890423fffeb5b90d"
              t: 1609852060
              dps: object
                      103: 200

This message is sent by the Zigbee Gateway, and cid identifies univocally the sub-device: the Thermostat#2. Using _tuya-smart-device_, if I send this 'SET' payload to Gateway:

{
    "devId": "890423fffeb5b90d",
    "dps": "103",
    "set": 205
}

I get the same response message from the device! (note: I MUST use the undocumented 'devId' parameter here).

This standard behavior is not the same for all sub-devices, all dps, all commands: SET/GET/MULTIPLESET/SCHEMA. In my experience, any device is someway special!

Testings with many devices I found 4 kinds of answers:

  1. standard (JSON as expected)
  2. none
  3. the unclear message 'json obj data unvalid' (it looks like a default in the device internal parser, it can mind: "nothing to do, or because the required command is not implemented, or because the JSON is erroneous").
  4. a binary undecoded message like " 3.34���o0�a�1� "

Best regards. m.s.

rospogrigio commented 3 years ago

Hi everybody, I am sharing my experience with Tuya devices that will probably help you handle the "json data unvalid" problem. I am owner of an HA integration called localtuya, that handles communication Tuya devices locally, but without using tuyapi (see https://github.com/rospogrigio/localtuya ). What we've discovered by combining source codes from different authors, is that there are 2 types of Tuya devices, that we call "type_0a" and "type_0d" devices, and they need to be queried in different ways in order to get the DPs values. 0x0a (DP_QUERY) and 0x0d (CONTROL_NEW) are respectively, the hex command that needs to be sent in order to retrieve the DPs.

  1. type_0a devices require, as said and you know, the 0x0a command, with just this payload: {"gwId": "<deviceID>", "devId": "<deviceID>"} , and it will just return a json with the full list of DPs and their current value
  2. type_0d devices are a bit trickier. If they are queried with the 0x0a command, they reply the well-known "json data unvalid" message. They require the 0x0d command instead, and a payload composed as follows: {"devId":"<deviceID>","uid":"<deviceID>","t":"<timestamp>","dps":{"1":null,"2":null,"100":null,"102":null,"103":null...}} Please note the "dps" list: this is a list of the DPs values you want to retrieve, pre-set at null; DP 1 should always be present. The device will reply with the current values of all the available DPs in the list. Since with a new device you might not know which DPs are available, when we want to autodetect them we just try requesting a list of all DPs from 1 to 30 and 100 to 120, since all devices encountered so far have DPs in this ranges. Unfortunely the payload string cannot exceed 255 chars, so we divide it into several requests of 10 DPs each. Finally, the last tricky bit is that these devices most of the times give an empty reply, BUT if you just repeat the recv() call, you finally get your reply with the DPs values.

@codetheweb I don't know how much of the above was of your knowledge, but I think it might help you review the code in order to support the "type_0d" devices too. Feel free to have a look at https://github.com/rospogrigio/localtuya/blob/master/custom_components/localtuya/pytuya/__init__.py and get inspiration. Even with this, though, I am having difficulties communicating with zigbee wifi gateways so maybe we can help each other. Let me have your thoughts, hope this can be helpful, bye!

codetheweb commented 3 years ago

Thanks for the info @rospogrigio, it ties together everything. Makes sense now.

Open to ideas on how to implement 0d devices. Should we query for DPS parameters automatically in connect(), or force the user to do so?

Also, do you know if there's a way to autodetect the device type @rospogrigio?

Apollon77 commented 3 years ago

I think detection is exactly send get and if you get back "unvalid json" then remember the type.

And yes this should be done on get I think, so that it is irrelevant fpr the library user which type it is. you can remember on the first get what it is for all subsequent calls in with using this istance and/or also support an (optional) init flag so that you can diretcly pass in the type when you know from before. Same: when you have the schema then you could also ad an init option where you can directly initialize the "Known" DPs and can so skip the auto detection

These infos are awesome

Apollon77 commented 3 years ago

PS: hehe detection done as I proposed :-) https://github.com/rospogrigio/localtuya/blob/master/custom_components/localtuya/pytuya/__init__.py#L546

rospogrigio commented 3 years ago

I'm very glad to be helpful. Then you might be able to help me handle devices that communicate through a Wifi Zigbee gateway ;-) since I'm getting the same "unvalid json" and I don't know how to get past this... Oh, and credits go to @tradeface , since he was the first that had the "0d" thing sorted out.

codetheweb commented 3 years ago

I don't currently have enough bandwidth to work on implementing 0d devices, but I'm happy to accept PRs. 😄

This issue is also getting kinda messy, so I started a new discussion here. Please move discussion about gateways there.

rospogrigio commented 3 years ago

@codetheweb you might be interested to read also what beatmag is reporting in https://github.com/rospogrigio/localtuya/issues/194 , he has done some interesting discoveries too...

codetheweb commented 3 years ago

@BarryW55 I don't currently have a timeline on implementing support; partly because I don't have any 0d devices.

As always, happy to accept PRs. :)

lknop commented 3 years ago

@codetheweb I think I could get this to work, but as I am new to the party and js is not my weapon of choice, please confirm my assumptions about what should be done:

Question: how to determine the actual dps payload to send? In one of the threads the following was proposed: dps: { '1': null, '2': null, '3': null, '101': null, '102': null, '103': null } EDIT: I have read up above about the workaround to query the dps ID in sets of 10. My question after reading the code is - should the subsequent calls be performed from within the get(options={}) method or should the api return after every request. This applies both to dps query and the switching to the CONTROL_NEW command.

codetheweb commented 3 years ago

possibly save this flag for the given device

Save for the current constructed instance yes, for future program executions no.

possibly give the user an option to set the flag manually upfront

Definitely.

should the subsequent calls be performed from within the get(options={}) method or should the api return after every request. This applies both to dps query and the switching to the CONTROL_NEW command.

If you're asking whether await get() should resolve before dps queries / switching to the 0d scheme is finished I'd say no.

Everything else sounds good, thanks for taking a look!

BarryW55 commented 3 years ago

@iknop If there is anything I can do to help test, please let me know