tsightler
commented
4 years ago Hmm...maybe none of this is necessary as it seems most of this is already implemented by the tuyagateway project. Perhaps it's time to freeze and retire this project.
Closed tsightler closed 4 years ago
tsightler
commented
4 years ago Hmm...maybe none of this is necessary as it seems most of this is already implemented by the tuyagateway project. Perhaps it's time to freeze and retire this project.
thomasdannenmuller
commented
4 years ago Hmm...maybe none of this is necessary as it seems most of this is already implemented by the tuyagateway project. Perhaps it's time to freeze and retire this project.
Interesting found !
thomasdannenmuller
commented
4 years ago I giving it a try right now. While the architecture seems fine to me, the configuration of gismocaster is a kind of nightmare, but it looks powerful. The docker integration has to be reworked, it seems ok for tuyagateway, but gismocatcher is not.
Your idea to build the discovery topics from tuya-cli output seems much more smarter to me.
tsightler
commented
4 years ago I saw a lot of issues around gismocaster on the project page so that definitely seems to be a pain point. Regardless, most people seem to be able to get it to work eventually, and it seems that part would be easier to fix vs building a different tool completely. It looks like gismocaster just stores it's config information in MQTT via retain, which I'm not a big fan of doing as I find people have a tendency to treat their MQTT broker as throwaway, they'll remove/reinstall it, etc.
That being said, it seems like a very active project and I don't know if it's a good idea to try to compete with it. Maybe I should just look to create a simple command line tool that builds a gismocaster compatible config in MQTT from the tuya-cli output.
Thanks for testing it out, I might give it a spin myself as well before making a decision whether to go forward with this project or not.
tsightler
commented
4 years ago So I've decided to go ahead and attempt a 3.0 version of this project, even with tuyagateway being already available. I've pushed the first set of features to the dev branch which does the following:
This has allowed me to remove a lot of code and simplify the processing of MQTT topics so now I'm going to work on that part. Currently the plan is to attempt to detect switches/dimmers/lights based on common DPS values and, if the profile does not match, simply default to only publishing DPS values, which can be used with pretty much any device. I've even thought about implementing some basic templating (DPS to capability mapping) so users could contribute working maps. After that I'll implement some basic MQTT discovery for known devices.
Admittedly, I'm not sure if this project will be easier to use than tuyagateway, or really offer any benefits over it, but the slightly different approach to device configuration might appeal to some, especially those using the project for non-Home Assistant cases.
thomasdannenmuller
commented
4 years ago Drops support for the old topic based config in favor of reading the file "devices.conf" in the same format as output from "tuya-cli wizard" during startup By default devices.conf should have "name", "id" and "key" for each device, so script will use automatic discovery which will find device on network, and discover protocol version where possible. If you want to manually specific IP address of device you can edit file and add "ip" and "version" parameters, with version being 3.1 or 3.3 (default is 3.1 if not specified).
When i first came to tuya-mqtt i though it could be cool to have that. I've build the same data structure to store all the necessary to build the mqtt topics, and issue commands, So i'll naturally say go for this.
Shortened device topic to tuya/
/. Example if device with name="Kitchen Lights" is tuya/kitchen_lights/state or if name is not available then tuya/ /state
Sounds good too, like this we will have far nicer topic tree.
According tuyagateway : While gismocaster brings a good way to build the HA discovery conf, it's really a pain to setup. I've encontered lots of troubles to make it work, and dispite it's allowing to build discovery for lots off device classes I guess that on the tuyagateway side lot of work reamain to be done. To me tuya-mqtt as better coverage off the tuyapi to mqtt interface.
I can't say for now if i'dd prefer to use a gismocaster like tool or configuring myself HA. I guess the zigbee2mqtt like design you propose is way better. 👍
tsightler
commented
4 years ago I've continue to push updates to the dev tree. Currently it's working pretty well with simple devices like switches, sockets and dimmers, and it works fine with other devices I've tested as well. You can now monitor and set any and all DPS values using
Tuya JSON
tuya/<device_id>/dps/state
tuya/<device_id>/dps/commandThese topics allow you to set and get raw Tuya JSON. This was previously possible with just /dps, but now requires /dps/command topic. You can set multiple topics at once, etc., just like previous versions.
Simple String Data
tuya/<device_id>/dps/<#>/state
tuya/<device_id>/dps/<#>/commandIf you don't want to use JSON you can also get and set individual values from each DPS key using these topics. While it was always possible to read these, the development version now allows you to set values with these topics as well so, for example, a simple dimmer switch might look like this:
tuya/
To change brightness state simply send "100" message to tuya/
Since you can access and set and DPS value, you can use tools like MQTT explorer monitor DPS values while you change settings in the Tuya/SmartLife app and quicly learn values.
The current branch also implements the first concept of templating using a simple JSON model, for example, a simple device like the dimmer above might look like this:
{
"state": { "dpsKey": 1, "dpsType": "bool" },
"brightness_state": { "dpsKey": 2, "dpsType": "int", "minVal": 0, "maxVal": 255 }
}"bool" devices show "ON" or "OFF based on true/false (in the future I'll allow to set the desired bool translation) and the corresponding command topic is automatically enable and will accept simple "true/false", "on/off", and "0/1" to change the state.
So with that simple template you end up with a device like this
tuya/
Currently the code tries to guess the device type based on the returned DPS values and their values, but right now it's only simple switches and dimmers. You can specify a device template by modifying the devices.conf file and adding type: "dimmer" or type: "switch" (the format is JSON5 so the syntax is more forgiving than regular JSON). I hope to grow the available pre-defined templates over time.
I still need to work on transforms, mainly for colors, since the Tuya color format doesn't map directly to what most home automation platforms expect out of the box, but there are a lot of different lights and I only have one so I may need some help there.
After some of that works done it's pretty easy to build MQTT discovery topics for Home Assistant (and others that support Home Assistant MQTT discovery protocol, like OpenHAB).
Overall I think this is a solid base for 3.0 for tuya-mqtt. For the simplest of case with just switches you can mostly just take the output from tuya-cli wizard, save it as devices.conf, and fire it up. For more advanced cases you'll have to map DPS values.
Hopefully over the next few weeks I'll finish the template engine and leads get the color transform working, but I'm already using it for most of my devices with good success.
Flip76
commented
4 years ago I just opened my toy box and I still have a lot of different tuya devices (light bulbs, switches, sockets, gateway, IR-remote control, etc.) - looking forward to test it with the 3.0 version :-)
tsightler
commented
4 years ago You can always pull the dev branch right now, it's poorly documented, but it does work. I'm actively using it for integration with one socket, a dimmer that won't flash Tasmota (so far), and RGBW light (although the light only works for on/off right now).
That being said, after wrangling with the code as it was for quite a while, and trying to just overlay the changes I had planned over it, I gave up and decided on a near complete rewrite. I looked at a number of other projects, including tuyagateway, but it seemed not very usable when I tried it, so I decided to pretty much follow the model of the homebridge-tuya. They already use a similar configuration concept (modified devices file) and I really like the way they split supported devices into their own files. It very much mirrors the model I used for ring-mqtt, for which I used the homebridge-ring project as my guide.
Based on this I started a complete refactor and already have it working with my switch and dimmer, and am now working on the light and a "custom" component where a user can build their own topic mapping (for advanced users only). The new code is simpler for me as a novice coder, to follow and each device now has it's own data handler so no complex logic or event handlers feeding data into the main code, etc.
I hope to get this in the next few days and push 3.0-beta2. I certainly won't be able to support as many devices as homebridge-tuya right off the bat, but at least the framework for adding new devices will be easy and the "custom" device will still allow people to build their own or access the saw DPS json/values.
Flip76
commented
4 years ago Actually I am using hombridge-tuya too but I've got several issues with it (when device get's switched off, it remains on in homekit and the massive logging is slowing down the whole homebridge-system)... But the handling is quite easy and straight forward. I also tried tuyagateway but I didn't get it to work properly...
So you wrote ring-mqtt too? I am currently using an older OH-plugin. But in order to get a real-time notification in OH, I exposed a switch to Amazon Echo and defined a rule which enables the exposed switch in realtime when somebody rings the bell. Unfortunately I have to wait several minutes until the video is available at ring servers. Then I download it and with ffmpeg I cut out a single picture which I send to my messenger.... Would ring-mqtt announce a "ding" in real-time?
Will clone the dev branch in a minute :-)
tsightler
commented
4 years ago Yeah, ring-mqtt is my project from the start, while tuya-mqtt is a project written by a previous author and I'm just a maintainer. Unfortunately there's no true "real time" with Ring cameras as the API for cameras only supports polling, however, the polling interval is 2 seconds (same as the Ring app) so it's usually fairly fast. However, ring-mqtt doesn't support any video so you'd have to obtain that via other means, which I guess you are already doing.
Anyway, I guess that's pretty offtopic for this thread. I thought I was making really good progress with tuya-mqtt, but I've hit quite a snag. Apparently with some of the newer devices it's not possible to get the initial state of the device with the get() call, you just receive an invalid response and I'm not sure there's any workaround for this in TuyAPI, at least, I haven't figured one out yet. It's still possible to control the device and state updates are sent on state changes, but nobody wants to have to toggle all of the settings just to get state. I see some workarounds in some of the other Tuya libraries, but none of them seem to work reliably, or maybe I just don' t understand them enough to implement them correctly.
If I can overcome this issue, I think I'm probably only a few hours of hacking away from being able to get a 3.0-beta2 release as otherwise the code seems to work well enough, but not getting the initial state makes the overall solution not viable from my perspective.
tsightler
commented
4 years ago Made a lot of progress tonight, figured out a way around the initial state issue and I have it working quite well for the devices I have access to (admittedly only a few). Have Home Assistant discovery working quite well, full color working for the light. I only have 3 specific device files written so far (swtich/socket, dimmer, RGBTW light, which are the devices I have), and I'm going to build a "generic" device which will allow access to device states/command via DPS topics directly and, in the future, hopefully a simple template based model for mapping DPS values to friendly topics.
Once those tasks are done I'll upload a new 3.0 beta and hopefully somebody can test it out on a wider set of devices. On the topic of devices, homebridge-tuya has device specific support for more than a dozen devices. It will probably be quite a while before I can add support for anywhere near that many, so I'd be curious if there is a priority of devices that people would like to see support for, especially knowing that I don't have those devices.
Flip76
commented
4 years ago I will setup all my Tuya devices over the weekend and let it run. Meanwhile I had a first try yesterday with the dev-branch. Unfortunately this didn't work:
mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/dps/2/command -m ON
nor this:
mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/868c7fa1XXXXXXXX/192.168.103.15/dps/2/command -m ON
I'll pull the latest changes and try it again
tsightler
commented
4 years ago Were you able to see state changes for the device at tuya/23804861d8bfXXXXXXXX/dps/2/state?
Also, most of the tuya devices I've seen require "true/false", not ON/OFF. Other than manually mapping with a schema there's no way to know (that I'm aware of) if any give DPS is a string, a number, or a boolean, so there is no processing on command messages sent to DPS indexes, you must send the command exactly as the DPS expects. If it uses true/false you must send true/false.
For example, on my simple dimmer, it's on/off control is DPS 1, if I want to use DPS to control it I have to send:
mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/dps/1/command -m true
Certainly the second command would not work. Also, if you have a friendly name in your devices.conf file then the topic will be the friendly name, converted to all lower case, with spaces converted to underscores. So, for example, if your devices.conf looks like this:
[
{
"name": "Kitchen Table",
"id": "86435357d8b87654321",
"key": "8b2a69c61234567"
}
]
Then the topic would be tuya/kitchen_table/dps/2/command, however, this should be very easy to see by manipulating the device via the Tuya/SmartThings app and seeing what topics are displayed.
In the new beta, you will be able to configure supported devices and their DPS values (if different from defaults) or, for the custom device, define a set of DPS values that should be queried for state during startup so you will be able to see the state topics without manipulating them elsewhere.
Basically, the summary version is that, when using the DPS topics directly, the code is not going to help you with any friendly transforms, you have to know exactly what format for values that DPS expects and send exactly that. You can even have more control by using the JSON command topic.
Normally, devices map values to "friendly" topics, so, for example, while my switch above sends and requires true/false on DPS1, I'd prefer to send and receive "on/off" since that's what Home Assistant expects by default (although of course it can be changed). When I map the value to a "friendly" topic I simply tell it that DPS/1 is a "bool" and now, any values going to/from DPS 1 via that friendly topic will also be transformed into ON/OFF and you can send ON/OFF, 0/1, true/false as commands and, since the code knows it's a "bool" it will translate them into true/false on the backend. In beta 2, you'll (hopefully) be able to create a simple template for unknown devices to perform "DPS" to "Friendly" topic mapping. For example, here's what a template might look like for my simple dimmer:
state: {
key: this.config.dpsPower,
type: 'bool'
},
brightness_state: {
key: this.config.dpsBrightness,
type: 'int',
min: (this.config.brightnessScale = 1000) ? 10 : 1,
max: this.config.brightnessScale,
scale: this.config.brightnessScale
}The above says that DPS value 1 is a bool value and should be mapped as such to /state topic and brightness is an interger, with value values between 10-1000. So, before mapping I could control my device with:
tuya/
After mapping I could still use the DPS values if I want, but instead I can use (an equivalent "command" topic is created for all mapped "state" topics automatically":
tuya/
This allows you to create a simple template for any device. Of course a template is not required, you can always manipulate DPS values directly, but templates can be more forgiving. Also, some DPS values aren't that usable without some transforming, for example, the color values of a light bulb. Tuya uses a hex format that needs to be translated for most tools to be able to easily consume it. Of course, you could translate it in Node Red, or in the home automation templating engine, but I've written helper functions that allow translating and mapping to "friendly" topics that are already in a format expected by common Home Automation systems.
Flip76
commented
4 years ago Ah ok, I missed that! I thougt ON/OFF will be automatically translated... The command for the current master branch which works is:
mosquitto_pub -t tuya/ver3.3/23804861d8bfXXXXXXXX/868c7fa1XXXXXXXX/192.168.103.15/command -m "{ \"dps\": 1, \"set\": true }"
Do you get the protocol version automatically or do I still have to put it somewhere?
Flip76
commented
4 years ago Unfortunately it doesn't work:
The command I used:
mosquitto_pub -t tuya/23804861d8bfXXXXXXXX/dps/1/command -m true
The devices.conf looks like:
{ name: 'Steckdosenleiste', id: '23804861d8bXXXXXXXX', key: '868c7fa1XXXXXXXX' },
Instead this still works with the dev-branch:
mosquitto_pub -t tuya/ver3.3/23804861d8bfXXXXXXXX/868c7fa1XXXXXXXX/192.168.103.15/command -m "{ \"dps\": 1, \"set\": true }"
Flip76
commented
4 years ago Oh hang on! Look like the problem is between my ears :) I just did a git clone but I forgot to add the branch :-)
git clone --single-branch --branch dev https://github.com/TheAgentK/tuya-mqtt.git
git branch
Will try again now :-)
Flip76
commented
4 years ago So I tried now:
mosquitto_pub -t tuya/steckdosenleiste/dps/1/command -m true
and the debug shows:
TuyAPI:mqtt Receive settings {"topic":"tuya/steckdosenleiste/dps/1/command","message":"true"} +5s
TuyAPI:mqtt Received command for DPS1: true +2ms
TuyAPI:device set: { dps: '1', set: true } +5s
TuyAPI SET Payload: +5s
TuyAPI {
TuyAPI devId: '23804861d8bfXXXXXXXX',
TuyAPI gwId: '23804861d8bfXXXXXXXX',
TuyAPI uid: '',
TuyAPI t: 1601557917,
TuyAPI dps: { '1': true }
TuyAPI } +0ms
(node:9376) UnhandledPromiseRejectionWarning: Error: No connection has been made to the device.
at TuyaDevice._send (/srv/openhab2-conf/scripts/test/tuya-mqtt/node_modules/tuyapi/index.js:235:13)
at /srv/openhab2-conf/scripts/test/tuya-mqtt/node_modules/tuyapi/index.js:215:14
at new Promise (<anonymous>)
at TuyaDevice.set (/srv/openhab2-conf/scripts/test/tuya-mqtt/node_modules/tuyapi/index.js:212:12)
at /srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-device.js:158:25
at new Promise (<anonymous>)
at TuyaDevice.set (/srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-device.js:157:16)
at processDpsKeyCommand (/srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-mqtt.js:314:16)
at /srv/openhab2-conf/scripts/test/tuya-mqtt/tuya-mqtt.js:413:29
at processTicksAndRejections (internal/process/task_queues.js:97:5)
(Use `node --trace-warnings ...` to show where the warning was created)
(node:9376) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 2)
(node:9376) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.Seems like the IP couldn't be detected. But this would be a normal behaviour in my environment. I seperated all IoT devices into it's own VLAN. Only the Raspberry (where I am running tuya-mqtt) is allowed to communicate with this VLAN and vice versa... But the Raspberry itself is in my Live VLAN (IP-Address 192.168.100.3/23) whilst the tuya-devices having 192.168.103.xxx/24.
Would it be possible to add the ip-addresses to the devices.conf manually and read them from there?
tsightler
commented
4 years ago Yep, just add ip and version (for protocol version, either 3.1 or 3.3) options to your devices.conf like the below example:
{
name: 'Your Device',
id: '86435357d80123456789',
key: '8b2a699876543210',
ip: '192.168.10.10',
version: '3.3'
}Yes, now it works :-)
tsightler
commented
4 years ago Great, please poke around with it and let me know how it works. The core DPS logic is unchanged in the next beta, with the exception that the "custom" device will let you configure the list of DPS topics that should be queried for state during startup since right now there's no state information until you make a change.
I had previously planned to try to discover this automatically, a schema query works for a lot of older devices, but the new ones, they just ignore that or return null data via TuyAPI, so I'm going to have to table that idea for now. I don't think having to specify the DPS values is that huge of an issue for people that have already had to go through the effort to get device keys from Tuya!
Flip76
commented
4 years ago Just a thought... Would it be possible to add the dps-value in the devices.conf? So for example: One monitor is on the powerstrip dps:1 and the other one on dps:2.... If I can define the dps in the config, I would be able to use a friendly name there too... Something like:
{
name: 'Steckdosenleiste',
id: '23804861d8bfXXXXXXXX',
key: '868c7fa1XXXXXXXX',
ip: '192.168.103.15',
version: '3.3'
},
{
name: 'Monitor 1',
id: '23804861d8bfXXXXXXXX',
key: '868c7fa1XXXXXXXX',
ip: '192.168.103.15',
dps: '1',
version: '3.3'
},
{
name: 'Monitor 2',
id: '23804861d8bfXXXXXXXX',
key: '868c7fa1XXXXXXXX',
ip: '192.168.103.15',
dps: '2',
version: '3.3'
}I don't think I will go that far having to have the same device mulitple times, and it's doesn't map to the device model of most home automation systems very well. Most home automations systems use some concept of a device with capabilities (Home Assistant uses device/entitiy, OpenHAB uses things/channels), the idea being that the "device" represents the physical device and the "capability" represents the individual components of that device that can be controlled or monitored.
I like to follow that same structure in the topic model so that it maps easily while following the above you would end up with three different "devices", rather than a single device with multiple control points. However, with beta2 you will be able to get very close to your goal with the generic device using something simple like this:
{
name: 'Steckdosenleiste',
id: '23804861d8bfXXXXXXXX',
key: '868c7fa1XXXXXXXX',
ip: '192.168.103.15',
version: '3.3',
template: {
monitor1_state: {
key: 1,
type: 'bool'
},
monitor2_state: {
key: 2,
type: 'bool'
}
}
}So then you can control devices with friendly topics like:
tuya/steckdosenleiste/monitor1_command tuya/steckdosenleiste/monitor2_command
I think that's pretty reasonable and maps to the device model cleanly.
Flip76
commented
4 years ago That's even better!! Great idea!! The big problem with home automation systems (like openhab) is the conversion for switches (on/off) to (true/false)... You always need to transform that values and in case of openhab a simple map-transformation doesn't seem to work. So I guess I have to write a little js to convert those values to switches and vers visa...
Flip76
commented
4 years ago Oh I see the on/off mqtt-channels in openhab offer a custom setting to handle this - now it works as expected :-)
tsightler
commented
4 years ago Yep, that was the idea, there are just so many devices out there there's no way I'm going to be able to support them all, as can be seen by other projects trying to do so. So, in the end, the idea was to create a tool that exposes all of the data in a raw way, and allows users to either use that (if they are advanced they can use the DPS/<#> or DPS JSON methods with whatever transform engine they are comfortable with) or, in other cases, just create a simple mapping using the custom device. Once someone creates a mapping that works, it's quite east to turn that into a "supported" device that can be referenced as a specific device type without having to map as the device files simply use the very same template engine, they are just codified into a device file so that you can reference them by name in the config instead of having to put the entire template there, and they include options to override specific variables, etc., similar to homebridge-tuya. Trying to strike a balance between "simple" and "flexible".
As it turns out, tuyagateway is using a similar concept, with a web based tool to configure, but I couldn't find a lot of people having success with it and, after playing with it for a couple of days, I never could get even my simple dimmer to do anything other than turn on/off (couldn't control brightness, etc).
I'm really close to meeting this goal, but I'm starting to run out of time to work on it. Hopefully I'll get it done tonight or by this weekend and tested enough to push beta2 and you can play around with it and let me know how it works for you, but it's good to know that to core code for 3.0 is working for you even now the next beta should be far more reliable.
Flip76
commented
4 years ago I absolutely agree, tuyagateway and gismocaster seems way to complicated for that. I will setup all my tuya devices and will have an play around with it over the weekend!
Thank you so much for this great tool!
tsightler
commented
4 years ago I've pushed 3.0.0-beta2 to the dev branch. There's still a lot of work to do, and there's only pre-built support for 3 specific devices (simple switch, dimmer, and RGBTW light, although only those with HSB style color, not HSBHEX, and no color temperature).
However, I've also implemented the generic device support so that you can build simple templates for devices like described above. Using the example above you'd just need:
{
name: 'Steckdosenleiste',
id: '23804861d8bfXXXXXXXX',
key: '868c7fa1XXXXXXXX',
ip: '192.168.103.15',
version: '3.3',
type: 'GenericDevice',
template: {
monitor1_state: {
key: 1,
type: 'bool'
},
monitor2_state: {
key: 2,
type: 'bool'
}
}
}Other types exist such as "int" and "str" and "hsb", you can see examples in the current device files as the device specific files are basically nothing but pre-defined templates that allow setting variables in the config file and a device specific discovery function for Home Assistant.
There's still some behaviors that I need to fix, for example, if a device gets an update it updates every attributes instead of just the one that changed. I don't like this, and it doesn't impact most use cases to get updates with the same data, but I plan to fix it.
For RGBTW lights I don't like how white mode support is implemented. Right now the bulb switches to color mode any time you set a color, and switch to white mode if you update the white level. I plan to also implement switch to white mode if saturation is zero or if you change color temperature (however, the one light I own doesn't happen to have color temperature, but I think I can implement it without real).
But, overall, I'm really happy where the base of this code is now. Yes, there's lots of improvements I could make, but it's been working very reliably with my devices for the last week.
If you do update, please make sure to run 'npm i' to update TuyaAPI to the custom version I'm now using which addresses some limitations in the official version.
Flip76
commented
4 years ago Thanks for all your effort! I pulled the new version, adjusted the devices.donf but for some power plugs I get an error (which I didn't see in the previous version):
tuya-mqtt:mqtt Received MQTT message -> {"topic":"tuya/tisch/command","message":"ON"} +2s
mqttjs:client _sendPacket :: (mqttjs_4e406ada) :: start +1ms
mqttjs:client sendPacket :: packet: { cmd: 'puback', messageId: 30, reasonCode: 0 } +1ms
mqttjs:client sendPacket :: emitting `packetsend` +0ms
mqttjs:client sendPacket :: writing to stream +0ms
mqtt-packet:writeToStream generate called +2s
mqtt-packet:writeToStream generate: packet.cmd: puback +1ms
mqtt-packet:writeToStream writeVarByteInt: writing to stream: <Buffer 02> +0ms
mqtt-packet:writeToStream writeNumberCached: number: 30 +0ms
mqtt-packet:writeToStream writeNumberCached: <Buffer 00 1e> +0ms
mqttjs:client sendPacket :: writeToStream result true +2ms
mqttjs:client sendPacket :: invoking cb +0ms
(node:22625) UnhandledPromiseRejectionWarning: TypeError: Cannot read property 'hasOwnProperty' of undefined
at GenericDevice.processDeviceCommand (/srv/openhab2-conf/scripts/tuya-mqtt/devices/tuya-device.js:130:47)
at GenericDevice.processCommand (/srv/openhab2-conf/scripts/tuya-mqtt/devices/tuya-device.js:184:18)
at MqttClient.<anonymous> (/srv/openhab2-conf/scripts/tuya-mqtt/tuya-mqtt.js:133:32)
at MqttClient.emit (events.js:315:20)
at Arguments.<anonymous> (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:1259:27)
at Object.MqttClient.options.customHandleAcks (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:183:150)
at MqttClient._handlePublish (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:1252:15)
at MqttClient._handlePacket (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:410:12)
at work (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:321:12)
at Writable.writable._write (/srv/openhab2-conf/scripts/tuya-mqtt/node_modules/mqtt/lib/client.js:335:5)
(node:22625) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). To terminate the node process on unhandled promise rejection, use the CLI flag `--unhandled-rejections=strict` (see https://nodejs.org/api/cli.html#cli_unhandled_rejections_mode). (rejection id: 8)I was able to get rid of the errer when I use SimpleSwitch instead of GenericDevice
tsightler
commented
4 years ago Can you please post the device.conf file you used when you received the error?
On Sat, Oct 3, 2020 at 12:59 AM Flip notifications@github.com wrote:
I was able to get rid of the errer when I use SimpleSwitch instead of GenericDevice
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/TheAgentK/tuya-mqtt/issues/41#issuecomment-703047198, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEHV7G3I5EENS4UQNASZTA3SI2VUVANCNFSM4RSHYLBQ .
Flip76
commented
4 years ago With this config I've got the error (but it was my fault case this device is just an ordinary power plug):
{
name: 'Tisch',
id: 'bf0751f4b74d39XXXXXXXX',
key: 'eacb979dXXXXXXXX',
ip: '192.168.103.14',
version: '3.3',
type: 'GenericDevice'
}
And with this config it works perfect:
{
name: 'Tisch',
id: 'bf0751f4b74d39XXXXXXXX',
key: 'eacb979dXXXXXXXX',
ip: '192.168.103.14',
version: '3.3',
type: 'SimpleSwitch'
}
OK, so no custom template for the GenericDevice, that's the issue. By default (i.e. with no custom template) the GenericDevice doesn't create any friendly topics at all because it doesn't know anything about the device so it doesn't know what to create. Using the GenericDevice this way you can only access the device via the DPS topics.
However, the command processing code currently expects there to be at least one friendly topic. I admittedly didn't test the "no template" case very well (I fired it up and monitored for DPS state changed, but didn't send commands). It's easy to fix, I'll push an update shortly.
tsightler
commented
4 years ago I pushed a fix for this case to dev, just initialized deviceTopics to empty if there's no template, that should do it.
Flip76
commented
4 years ago Thanks! Will pull it in a minute. I am just setting up all tuya devices in order to add them to the devices list
Flip76
commented
4 years ago So I played around and for the switches / power strips everything works perfect! Next I tried to get the devices behind the Zigbee-Gateway to run but I couldn't make it...
Here is the output of "tuya-cli wizard":
[
{
name: 'Zigbee Wireless Gateway（TYGWZW-01）',
id: 'bf1568c46aaec7XXXXXXXX',
key: 'a1214b1fXXXXXXXX'
},
{
name: 'Temperatur- und Feuchtigkeitssensor',
id: 'bf86ac4f5ce854XXXXXXXX',
key: 'a1214b1fXXXXXXXX'
},
{
name: 'Security Remote Controller',
id: 'bfd6d2a11bae7dXXXXXXXX',
key: 'a1214b1fXXXXXXXX'
}
]As you can see, sub-devices of the gateway have different id's but they always have the same key. So I tried to different combinations in the devices.conf:
{
name: 'TempSensor',
id: 'bf86ac4f5ce854XXXXXXXX',
key: 'a1214b1fXXXXXXXX',
ip: '192.168.103.12',
type: 'GenericDevice',
template: {
temperature_state: {
type: 'str'
},
}
}
---
{
name: 'Gateway',
id: 'bf1568c46aaec7XXXXXXXX',
key: 'a1214b1fXXXXXXXX',
ip: '192.168.103.12',
type: 'GenericDevice',
template: {
temperature_state: {
id: 'bf86ac4f5ce854XXXXXXXX',
type: 'str'
},
}
}
---
{
name: 'Gateway',
id: 'bf1568c46aaec7XXXXXXXX',
key: 'a1214b1fXXXXXXXX',
ip: '192.168.103.12',
type: 'GenericDevice',
template: {
temperature_state: {
id: 'bf86ac4f5ce854XXXXXXXX',
key: 'a1214b1fXXXXXXXX',
type: 'str'
},
}
}As you see, in the 1st example I tried to add the temperature sensor as a normal device. In the 2nd example I tried to use the Gateway and the sensor as a Generic Device with id only. The 3rd try was same as the 2nd only adding the key too...
Flip76
commented
4 years ago Anyhow, I will go ahead trying the lightbulbs and I also found a Tuya universal IR Remote Control and a WiFi relais in my toybox :-)
And this is the temperature sensor I‘m trying to add:
Flip76
commented
4 years ago I succeeded in adding the power plug with meter:
{
name: 'SwitchMeter',
id: '2302731368c6XXXXXXXX',
key: '79728e3cXXXXXXXX',
ip: '192.168.103.26',
version: '3.3',
type: 'GenericDevice',
template: {
power_state: {
key: '1',
type: 'bool'
},
kwh_today_state: {
key: '2',
type: 'int'
},
current_ma_state: {
key: '4',
type: 'int'
},
current_w_state: {
key: '5',
type: 'int'
},
voltate_state: {
key: '6',
type: 'int'
}
}
}Now it get's even clearer what the key in the template means... It's the DPS right?
tsightler
commented
4 years ago Yes, that's correct, the key is the DPS key. It's sounds like you're having pretty good success so far.
I have no idea how the Tuya gateway works but it's not surprising to me that it doesn't work as I've never seen one. I see you can get the ids/keys from tuya-cli, so my question is, can you control it with tuya-cli? If you can, then it should be possible to get it to work. That being said, I'm pretty confident a better option would be to just get a Zigbee stick and control them directly.
Flip76
commented
4 years ago The funny thing is that I can use tuya-cli wizard only... All other commands (set/get) just fail with parse error for all my devices (even the switches which I can control with tuya-mqtt)...
On the other hand, the subdevices of the Zigbee-gateway are only sending data (not receiving) - like temp sensor and the remote control...
But you are right! I already looked at zigbee-mqtt as I have a couple of Xiaomi-devices which I want to control via zigbee-mqtt. I already ordered the stick a couple of days ago...
What I wanted to ask... In the mqtt explorer I see that another topic „homeassistant“ appeared, representing the state_topics from tuya-mqtt... Do you write to a second topic somewhere?
One other thing I realized... Above (at my last posting) you see that I configured some keys as int... But you initially send „None“ as value which causes an error on OpenHAB side as it expects a number... Would it be possible to send 0 to int-defined keys?
tsightler
commented
4 years ago First, let me say thanks to you for all the testing. Having what amounts to nearly a complete rewrite tested on so many devices out of the gate is hugely valuable, especially since I only have a small number of devices myself.
If you do test the light bulb, make sure to pull the latest code I pushed last night, lots of fixes there for the color support and also added support for old style HSBHEX bulbs, although you have to manually specify the type (I'll try to auto-detect in a future version). It still has a few issues, especially if it receives different commands for H, S, B separately in fast order. I need to add some code to deal with this rapid command issue.
The topics you are seeing being written to homeassistant are MQTT discovery messages. Home assistant implemented their own protocol for allowing devices to send a specially formatted MQTT message with all of the Home Assistant configuration data so that users don't have to do anything for devices to "just work". In home assistant my switch/socket/dimmer/light all just show up automatically. This of course doesn't work with the generic device because it has no way to now how the friendly topics should be mapped to Home Assistant settings. However, this might still be useful for OpenHAB as I believe it implements Home Assistant style auto topic discovery, although I'm not sure how complete the support is (in Home Assistant you can auto-discover any MQTT device with a supported integration).
Regarding the Zigbee gateway, just looking at the output from the wizard, I'm thinking that's the reason there is a gwId parameter for DP_QUERY. Right now the TuyAPI code always sets the DevID and GwID to the same, because, but I'm betting that with a gateway you should set the GwID to the DevID for the gateway while the DevID should be the actual DevID you want to talk to. Admittedly this is just conjecture on my part as I can find no code that talks the Tuya protocol that appears to support Zigbee gateways. I'm thinking it might be possible to figure it out if I had access to one, but the requests to support it aren't very common, probably because it's just easier to use Zigbee directly and bypass the Tuya stuff.
It's not an all surprising that you can use tuya-cli with "wizard" but not other command. The "wizard" command is not talking to the device, rather it's talking to the Tuya cloud app account to get the device keys. All other commands attempt to talk to the device. Lots of newer tuya devices do not respond to the TuyAPI get() request, so tuya-cli can set values, and get the response, but not get current values. As a matter of fact, the dev version of tuya-mqtt is currently using a modified TuyAPI that I branched which attempts to work around these issues by quietly issues a set() with a null value when the get() command fails, for all devices I have access to this manages to cajole them into sending a response. It seems this is mostly working for your devices as well.
Thanks for the feedback on the issue with int values. I certainly shouldn't provide a string in that case. However, I wonder if I should just suppress any non-int updates at all vs sending a zero, since that would not accurately reflect the value from the device. I guess I'm not even sure how it's managing to get some invalid value as, theoretically even the initial value should be valid. I guess I need to dig through that code to see if I can figure that out. Does the DPS value always reflect an integer?
I still need to implement "float" as well. I think with bool/int/float/string/hsb/hsbhex that should be enough to build a template for most devices, although there might be a few that need something special, covers for example.
tsightler
commented
4 years ago Regarding the int issue you reported, it's really good feedback. I wonder if you could test the latest code that I pushed late last night. That code implements granular DPS updates. With the prior code, any time data was received from the device it updated all topics, while now, data updates only trigger updates of the specific values/topics that changed. I'm hopeful that might get rid of the erroneous data. Regardless, I will probably try to implement some additional logic for this.
Once again, thanks very much for your valuable feedback and willingness to help test when the code is in such a highly fluid state. I hope that once you're done, you'll share your experience with the OpenHAB community.
Flip76
commented
4 years ago Oh no, you don‘t need to thank me! I have to thank you for the great work! Rest assured that I will let the OH-community know about this great way to handle tuya devices. There are many people looking for a reliable solution and yours is definitely the most promising!
Before I always looked into getting Homebridge devices exposed to mqtt but I didn‘t get it to work (but still would be nice to have). Admittedly, I am not a JS/Java/Python guy. My world is SQL,PL/SQL, Shell- and Perl-Scripting... But it seems I should definitely get a heads up to those techniques...
So if I understood right, you are creating the topics for Home Assistant? Indeed, I immediately saw those channels in the Openhab Inbox - so autodiscovery works fine! I will have a play around tomorrow with this too! Also I will pull the latest version and check for the int-issue...
Btw: If you are interested in having a play with my devices, I can setup a new tuya account, add my toys there and grant you access to my lab-env (which contains the latest stable OH-version 2.5.9) next to tuya-mqtt...
Flip76
commented
4 years ago PS: When you create a mqtt channel in OH, you have a list of choices:
Regarding to your question if the DPS always reflects an int-value... The int-value I tested was the voltage of the switch. This one should always be integer. But there might be other cases... Given that you can define the data type for each dps in the template maybe it would make sense to „initialize“ the value with a default of the data type. As to my knowledge, OH always accepts NULL for any data type. Another approach could be to let the user define the initialization value within the template?
tsightler
commented
4 years ago Yeah, I guess what I'm not understanding is how there could be a data update in my code that is somehow sending "None". I mean, I know what part of the code is generating this, but the check is there to cover the case where a user defines a field as Int, but the actual value returned by the DPS key is not an Int. I wanted an easy way to identify this case if someone did that. However, somehow this is happening in your case even though it seems obvious you have selected a DPS that can only be an int.
I've also been considering adding the ability to define some simple math transforms on DPS values when they are Int/Float values (and perhaps some regex support for str values). Do you think that would be useful? I know you can usually do this in the automation platform, but sometimes those tools can be quite unfriendly. I was thinking just simple add/subtract/multiply/divide functions so you could have "key" = 1, "type"="int" and "keyTranform"="*10" to, for example, multiple the int value by 10.
tsightler
commented
4 years ago One thing I had a question about, with the RBGTW light, I've been tweaking on that code and trying to figure out how to deal with switching between "white" and "colour" mode. Obviously there's a mode topic for manual switching, so that's fine, but I've found that a lot of Home Automation systems don't seem to have any specific switch for changing modes so, to make it work reasonably, you have to change modes automatically based on the color data provided.
This is the behavior currently:
Switch to color mode when receiving any hue (H) or brightness (B) change, or when receiving as saturation (S) change with S > 0.
Switch to white mode when receiving any white level (W) change, or when receiving a saturation (S) change to S= 0.
Implementing this logic with Home Assistant seems to work perfectly. If a change the color on the color wheel to anything other than the center, I get color mode, if I set it to center (i.e. saturation 0) the light switched to white mode (this makes sense to me as 0% saturation is white). If I change the brightness (part of HSB) it switches or stays in color mode. Changing "White Level" slider automatically changes to white mode as well, and, once I implement color temerpature, changing that would also switcht to white mode.
As well as this seems to work for Home Assistant, it doesn't work as well when those lights are presented to other devices, especially integrations that have more limited color implementations, for example that only support combined brightness or named colors. Some of these don't send anything at all for "white", instead just sending the "on" command, which is just weird to me, but I guess is perhaps some behavior from Hue bulbs, which apparently always go to white on power on?
So, on these devices I have no way to actually get the bulb to white mode. Even changing the brightness will flip the light to color mode based on the rules above (because those devices don't support white level or understand modes).
It a really difficult problem to overcome. I could ignore brightness for mode switch, and just have brightness control the brightness in color mode or white mode (just doing the DPS translation silently on the backend based on the mode), but this means that, for example, you can't have different brightness for white and color mode, and a lot of bulbs are much brighter in white mode than in color.
Another option would be to only have brightness control to actually report and control the white level brightness when in white mode, and the color level brightness when in color mode. This really isn't "that" difficult, but seems a little ugly. However, this would still leave me with no real way to get the light into white mode from devices that don't have a white level or saturation control.
Perhaps I'm getting too in the weeds on this and there's not that many scenarios that don't support full color modes now, but, for example, right now I can't get my Alexa to set the bulb to white mode via Home Assistant using any method because, when I tell Alexa to set it to white, it just sends an "on" command and no color. My only thought is to have an option to "set to white with "on" command", but the problem with that is that Home Assistant sends on commands with every state update, so I'd have to have logic to attempt to suppress mode changes for a few hundred ms, if it's received in case it's a spurious ON (I have no idea why Home Assistant sends "on" every time, to me that seems unnecessary).
Anyway, I'd love to hear thoughts on the best way to deal with it. Perhaps hearing how some other tools like OpenHAB deal with such devices would help.
Flip76
commented
4 years ago I've also been considering adding the ability to define some simple math transforms on DPS values when they are Int/Float values (and perhaps some regex support for str values). Do you think that would be useful? I know you can usually do this in the automation platform, but sometimes those tools can be quite unfriendly. I was thinking just simple add/subtract/multiply/divide functions so you could have "key" = 1, "type"="int" and "keyTranform"="*10" to, for example, multiple the int value by 10.
That would actually help a lot... For example: The switch with meter delivers a voltage value of 2246 - so you have to divide by 10... But then it's not an integer anymore...
Flip76
commented
4 years ago Regarding the lights, I just checked on my Philips Hue lights and Openhab uses 3 channels (color, brightness, color_temperature) sending 0 as temperature seems to activate white-mode... But I will check with the tuya light bulbs...
Flip76
commented
4 years ago I just tried to add my 1st tuya test bulb:
{ name: 'Test Bulb', id: '56100002c44fXXXXXXXX', key: '848c69c9XXXXXXXX', ip: '192.168.103.23', type: 'RGBTWLight' }
The channel immediately appears in the homeassistant-topic (and also in the openhab-inbox) but I can't see the device under the tuya-topic:
The log shows:
2020-10-05T09:28:12.812Z tuya-mqtt:tuya Connected to device Test Bulb (192.168.103.23, 56100002c44fXXXXXXXX, 848c69c9XXXXXXXX)
2020-10-05T09:28:12.814Z tuya-mqtt:tuya Home Assistant config topic: homeassistant/light/56100002c44fXXXXXXXX/config
2020-10-05T09:28:12.815Z tuya-mqtt:tuya {
name: 'Test Bulb',
state_topic: 'tuya/test_bulb/state',
command_topic: 'tuya/test_bulb/command',
brightness_state_topic: 'tuya/test_bulb/brightness_state',
brightness_command_topic: 'tuya/test_bulb/brightness_command',
brightness_scale: 1000,
hs_state_topic: 'tuya/test_bulb/hs_state',
hs_command_topic: 'tuya/test_bulb/hs_command',
white_value_state_topic: 'tuya/test_bulb/white_value_state',
white_value_command_topic: 'tuya/test_bulb/white_value_command',
white_value_scale: 1000,
unique_id: '56100002c44fXXXXXXXX',
device: {
ids: [ '56100002c44fXXXXXXXX' ],
name: 'Test Bulb',
mf: 'Tuya',
mdl: 'RGBTW Light'
}
}
2020-10-05T09:28:12.815Z mqttjs:client publish :: message `{"name":"Test Bulb","state_topic":"tuya/test_bulb/state","command_topic":"tuya/test_bulb/command","brightness_state_topic":"tuya/test_bulb/brightness_state","brightness_comma
nd_topic":"tuya/test_bulb/brightness_command","brightness_scale":1000,"hs_state_topic":"tuya/test_bulb/hs_state","hs_command_topic":"tuya/test_bulb/hs_command","white_value_state_topic":"tuya/test_bulb/white_value_state","white_value
_command_topic":"tuya/test_bulb/white_value_command","white_value_scale":1000,"unique_id":"56100002c44fXXXXXXXX","device":{"ids":["56100002c44fXXXXXXXX"],"name":"Test Bulb","mf":"Tuya","mdl":"RGBTW Light"}}` to topic `homeassistant/l
ight/56100002c44fXXXXXXXX/config`Does the device need some additional parameters in the devices.conf?
tsightler
commented
4 years ago Does the device need some additional parameters in the devices.conf?
It technically doesn't need them, but they might be required based on the device. Obviously I need to create documentation for the different devices to show what parameters are available for each, but for now the easiest way is to simply open the file for the device (in this case rgbtw-light.js) and typically around line 7 or so there will be a commented section for "Set device specific variables" that looks like this (example from rgbtw-light):
this.config.dpsPower = this.config.dpsPower ? this.config.dpsPower : 1
this.config.dpsMode = this.config.dpsMode ? this.config.dpsMode : 2
this.config.dpsWhiteValue = this.config.dpsWhiteValue ? this.config.dpsWhiteValue : 3
this.config.whiteValueScale = this.config.whiteValueScale ? this.config.whiteValueScale : 1000
this.config.dpsColorTemp = this.config.dpsColorTemp ? this.config.dpsColorTemp : 4
this.config.dpsColor = this.config.dpsColor ? this.config.dpsColor : 5
this.config.colorType = this.config.colorType ? this.config.colorType : 'hsbhex'The word after "this.config" is the configurable variable, and the value on the far right is the default, so in the case above the custom options and their defaults are as follows:
dpsPower: 1 <- DPS for light power state dpsMode: 2 <- DPS for white/color mode dpsWhiteLevel: 3 <- DPS for white mode brightness whiteValueScale: 1000 <- Scale of the white value setting dpsColorTemp: 4 <- DPS for white color temperature (Warning: not currently implemented) dpsColor: 5 <- DPS for HSB color values colorType: 'hsbhex' <- The Tuya color format for this bulb (hsb/hsbhex)
So, for example, my generic Tuya bulb that I purchase recently (which does not have color temperature) uses this config:
{
name: 'Office Lamp Color',
id: 'eba86a1731xxxxxxxxxxx',
key: '6ce9xxxxxxxxxxxx',
type: 'RGBTWLight',
dpsPower: 20,
dpsMode: 21,
dpsWhiteValue: 22,
dpsColor: 24,
whiteValueScale: 1000,
colorType: 'hsb'
}For right now the defaults are simply the defaults that tuya-mqtt has always used for these values so I assume they must work with most older devices. I believe that perhaps the default settings are common for older bulbs and the settings I use above are common for newer devices so I was thinking, in the future, I could add some auto-detection logic. For example I could query DPS 2 and DPS 21, and if either contain the words "white" or "colour", then I can be pretty sure it's a bulb, then I could query DPS 5 and DPS 24 and determine if it's HSB/HSBHEX based on format. But I'll admit I have no idea how consist the DPS values are between devices. Looking around it doesn't seem they are super consistent.
So, otherwise, the easiest thing to do is to start with it as a generic-device, then play around with it and monitor the DPS values that are returned, then configure them as appropriate.
I pushed v2.1.0 yesterday, which is mostly a backlog of small fixes and support for 3.3, but I think it's time to perform a more significant revamp with the goal of simplifying the use of tuya-mqtt with various home automation tools, based off my experience building and maintaining the ring-mqtt project. Below is a current list of things I'm thinking, it's somewhat of a major shift, but I think it will make using this project a LOT easier for many users as the barrier is already quite high with the need to create accounts with Tuya and get keys I think it would at least be good to make using this tool as easy as possible.
Open to hear all thoughts on the below ideas.
Use devices configuration file instead of requiring set to MQTT topic to establish initial connection
Simplify topics -- topics would be of the following formats: tuya//state <-- String (ON/OFF)
tuya//command <-- String (ON/OFF, 0/1, true/false, etc)
tuya//brightness_state <-- Value (1-255)
tuya//brightness_command <-- Value (1-255)
tuya//color_state
tuya//color_command
tuya//dps/state <-- Get raw TuyAPI JSON
tuya//dps/command <-- Set raw TuyAPI JSON
tuya//dps/1/state <-- String (get DPS value)
tuya//dps/1/command <-- String (set DPS value)
Friendly name would be based on output of devices from tuya-cli, which includes device name from Tuya app, for example "Kitchen Lights" becomes tuya/kitchen_lights. Simply delete friendly name from device config to use device ID instead.
The state/color/brightness states and commands would use the format of Home Assistant, which I believe is also easily consumable for OpenHAB and other home automation tools (standard format use by, for example, Tasmota).
Overall I believe implementing simple command for state/brightness/color will make integration of those devices easier, and allowing individual DPS states to be set without JSON templates will make other setups easier as well, although it will still be possible to get/set via tuya based raw JSON values using dps and dps/set
Ideally implement Home Assistant style MQTT discovery (this is a stretch goal) This will make it easier to integrate into other tools (like OpenHAB) that also support Home Assistant style MQTT discovery. Initially probably only for sockets/lights/switches since I don't have other devices.