Version 3.0 Discussion

[tsightler]

I pushed v2.1.0 yesterday, which is mostly a backlog of small fixes and support for 3.3, but I think it's time to perform a more significant revamp with the goal of simplifying the use of tuya-mqtt with various home automation tools, based off my experience building and maintaining the ring-mqtt project. Below is a current list of things I'm thinking, it's somewhat of a major shift, but I think it will make using this project a LOT easier for many users as the barrier is already quite high with the need to create accounts with Tuya and get keys I think it would at least be good to make using this tool as easy as possible.

Open to hear all thoughts on the below ideas.

Use devices configuration file instead of requiring set to MQTT topic to establish initial connection

• Format is exactly the same as tuya-cli wizard option produces for output
• Validates and connects to devices on startup and sends state updates

Simplify topics -- topics would be of the following formats: tuya//state <-- String (ON/OFF) tuya//command <-- String (ON/OFF, 0/1, true/false, etc) tuya//brightness_state <-- Value (1-255) tuya//brightness_command <-- Value (1-255) tuya//color_state tuya//color_command tuya//dps/state <-- Get raw TuyAPI JSON tuya//dps/command <-- Set raw TuyAPI JSON tuya//dps/1/state <-- String (get DPS value) tuya//dps/1/command <-- String (set DPS value)

Friendly name would be based on output of devices from tuya-cli, which includes device name from Tuya app, for example "Kitchen Lights" becomes tuya/kitchen_lights. Simply delete friendly name from device config to use device ID instead.

The state/color/brightness states and commands would use the format of Home Assistant, which I believe is also easily consumable for OpenHAB and other home automation tools (standard format use by, for example, Tasmota).

Overall I believe implementing simple command for state/brightness/color will make integration of those devices easier, and allowing individual DPS states to be set without JSON templates will make other setups easier as well, although it will still be possible to get/set via tuya based raw JSON values using dps and dps/set

Ideally implement Home Assistant style MQTT discovery (this is a stretch goal) This will make it easier to integrate into other tools (like OpenHAB) that also support Home Assistant style MQTT discovery. Initially probably only for sockets/lights/switches since I don't have other devices.

[tsightler]

BTW, I noticed that OpenHAB requires a single topic with comma separated HSB values and also requires the scale for saturation and brightness to be 100. Right now the RGBTW light uses the "native" brightness scale for the color type, which for Tuya HSB is 1000 and for HSBHEX is 255. Home Assistant can deal with this because it puts the H and S components on one topic, while the B component is on a separate topic where you can define the specific scale.

Based on the fact that OpenHAB appears to require 100 scale, and Home Assistant can work with that scale, I think I'm just going to modify the code to use 100 scale for all MQTT values and do the translation to Tuya scale values on the backend.

[Flip76]

Thanks again for the detailed information! Don't know why but I can't get the light bulb to appear as tuya-topic (not even when I define it as generic device)

[Flip76]

BTW, I noticed that OpenHAB requires a single topic with comma separated HSB values and also requires the scale for saturation and brightness to be 100. Right now the RGBTW light uses the "native" brightness scale for the color type, which for Tuya HSB is 1000 and for HSBHEX is 255. Home Assistant can deal with this because it puts the H and S components on one topic, while the B component is on a separate topic where you can define the specific scale.

Based on the fact that OpenHAB appears to require 100 scale, and Home Assistant can work with that scale, I think I'm just going to modify the code to use 100 scale for all MQTT values and do the translation to Tuya scale values on the backend.

Yes I saw this too :-)

==> /var/log/openhab2/openhab.log <== 2020-10-05 21:08:21.034 [WARN ] [ab.binding.mqtt.generic.ChannelState] - Command '1000' not supported by type 'ColorValue': 1000 is not a valid string syntax

This is from the autodiscovered item homeassistant/light/56100002c44fXXXXXXXX/config

[tsightler]

Thanks again for the detailed information! Don't know why but I can't get the light bulb to appear as tuya-topic (not even when I define it as generic device)

Hmm, weird. You can always run in debug mode (DEBUG=tuya-mqtt:*) and see what's happening.

[tsightler]

Also, would be interested to know if you can control the device with tuya-cli. In general, if it doesn't work with tuya-cli it probably won't work with tuya-mqtt as we both use the same TuyAPI on the backend to do the actual communication with the devices. Admittedly, current dev branch is using a TuyAPI that I modified to work better with newer devices as, with current release version of TuyAPI /tuya-cli, the get() function doesn't return anything, you can only get data updates.

However, tuya-cli should be able to control any Tuya device, so, for example, sending true false to dps 1 or to dps 20, should still be able to turn the bulb on/off with tuya-cli to at least prove that there's some level of control for the device (assuming the device uses DPS 1/20, it could always use something else).

[Flip76]

2020-10-05T14:42:22.352Z tuya-mqtt:tuya Search for device id 56100002c44fXXXXXXXX 2020-10-05T14:42:22.363Z tuya-mqtt:tuya Found device id 56100002c44fXXXXXXXX 2020-10-05T14:42:22.430Z tuya-mqtt:tuya Connected to device Test Bulb (192.168.103.23, 56100002c44fXXXXXXXX, 848c69c9XXXXXXXX)

But no Data from device :-(

You are right, I am not able to set a state via tuya-cli - the command just hangs. Don't know how to check but could this bulbs running protocol version 3.2 (which is not supported buy tuyapi yet)?

[tsightler]

Also, make sure that you are using the very latest code (I pushed code until the wee hours this morning) and be sure to run "npm i" in the tuya-mqtt to make sure the custom TuyAPI is installed, otherwise there could be cases where it would hang as in the official TuyAPI there's no timeout for send() commands for example, so it will just wait forever.

That being said the GenericDevice shouldn't be sending any get()/set() commands by default, it should just be listening for data events from the device. I've never seen a device that didn't send STATUS command events for data updates, but I guess they could be out there.

[tsightler]

It's possible it's a 3.2 device, I really don't know. I guess it's based on how deep you want to go. If you are able to get packet traces of the Tuya app talking to the device locally, but you'd have to block the device that is running the app from being able to access the internet to force it to talk locally/directly to the device, otherwise it talks to Tuya cloud. I've heard rumors of devices that even the app can't control locally, only via Tuya cloud, but I've never seen one of those devices.

Other options would be trying it with a simple test app using an alternative Tuya protocol implementation like Tuyaface (tuyagateway seems unreliable, but my experience with Tuyaface itself is pretty good).

[Flip76]

I did a get now before I issued a set via tuya-cli and it worked:

[22:20:38] root@openhab:/srv/openhab2-conf/scripts/tuya-mqtt# tuya-cli get --ip 192.168.103.23 --id 56100002c44fXXXXXXXX --key 848c69c9XXXXXXXX --all --protocol-version 3.3
{
  devId: '56100002c44fXXXXXXXX',
  dps: {
    '20': true,
    '21': 'white',
    '22': 1000,
    '23': 1000,
    '24': '000003e803e8',
    '25': '020e0d0000e80383000003e803e8',
    '26': 0
  }
}
[22:20:49] root@openhab:/srv/openhab2-conf/scripts/tuya-mqtt# tuya-cli set --ip 192.168.103.23 --id 56100002c44fXXXXXXXX --key 848c69c9XXXXXXXX --protocol-version 3.3 --dps 20 --set false
Set succeeded.

[tsightler]

Ah, great, so really, your device is very similar to mine (except mine will not respond to a get schema), and yours appears to actually have color temperature (so hey, once I implement that you can test it too) so a device.conf similar to what I posted above should work. Just make sure you update to the very latest code I pushed this morning.

[Flip76]

I do a "git pull" and a "npm i" everytime I start testing :-) But the device still doesn't appear as tuya-topic :-( And I just saw thet tuya-cli get/set is only working when tuya-mqtt is stopped - but I guess that's normal right?

[tsightler]

Yep, normal, most devices only allow a single local connection at once. It's not clear to me how the device could work with tuya-cli and not with tuya-mqtt, underneath both use the same API and perform effectively identical functions. Are you able to perform tuya-cli get of a specific DPS?

Probably the best thing to do is to run with DEBUG=TuyAPI and see if you see the raw data coming from the device. That would hopefully at least tell me if something is hanging in tuya-mqtt or in TuyAPI.

[Flip76]

I can get a single DPS with tuya-cli:

tuya-cli get --ip 192.168.103.23 --id 56100002c44fXXXXXXXX --key 848c69c9XXXXXXXX --protocol-version 3.3 --dps 20
true

If I start tuya-mqtt it seems it doesn't get any dps values:

2020-10-05T15:52:21.944Z TuyAPI Socket connected.
2020-10-05T15:52:21.947Z TuyAPI GET Payload:
2020-10-05T15:52:21.948Z TuyAPI {
  gwId: '56100002c44fXXXXXXXX',
  devId: '56100002c44fXXXXXXXX',
  t: '1601913142',
  dps: {},
  uid: '56100002c44fXXXXXXXX'
}

But when I trigger an event, I can see tuyapi receives data:

# Power-off device via tuya app
  TuyAPI Received data: 000055aa00000000000000080000006b00000000332e330000000000000005000000011a29e98a83edd93f3e494cd9fb809d2c89b396e1e4429fd89d209d7c8360b96caaef1ecfb581cfebfb9c25078d346dc7b4b535ebc569b544b72cb2d564b1d27e340cf1b9034bed0eaf8255c97c34624ccf4b90d00000aa55 +5s
  TuyAPI Parsed: +2ms
  TuyAPI {
  TuyAPI   payload: '3.3\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0005\u0000\u0000\u0000\u0001\u001a)銃��?>IL����,�����B�؝ �|�`�l��\u001eϵ�����%\u0007�4mǴ�5��i�D�,��d��~4\f�\u0003K�\u000e��U�|4bL',
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 8,
  TuyAPI   sequenceN: 0
  TuyAPI } +0ms

# Power-on device via tuya app
  TuyAPI Received data: 000055aa00000000000000080000006b00000000332e330000000000000006000000011a29e98a83edd93f3e494cd9fb809d2c89b396e1e4429fd89d209d7c8360b96c6d631d4336f2e0cb2fe112ef795cea04eece147447798171e3ae98f5012d87289e179e74db28a7949aff39752661dde2647dc1700000aa55 +4s
  TuyAPI Parsed: +2ms
  TuyAPI {
  TuyAPI   payload: '3.3\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000\u0001\u001a)銃��?>IL����,�����B�؝ �|�`�lmc\u001dC6���/�\u0012�y\\�\u0004��\u0014tGy�q㮘�\u0001-�(�\u0017�t�(����9u&a��',
  TuyAPI   leftover: false,
  TuyAPI   commandByte: 8,
  TuyAPI   sequenceN: 0
  TuyAPI } +0ms

[Flip76]

One more thing: When I trigger the power-on/power-off via the tuya app, I see in the debug log (when running : DEBUG=* node tuya-mqtt.js)

tuya-mqtt:tuya Data from device not encrypted: 33ILB lmcC6yyct9ua +4s

[tsightler]

OK, that's super userful as it tells me that TuyAPI is failing to decrypt the payload, specifically this:

payload: '3.3\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0006\u0000\u0000\u0000\u0001\u001a)銃��?>IL����,�����B�؝ �|�`�lmc\u001dC6���/�\u0012�y\\�\u0004��\u0014tGy�q㮘�\u0001-�(�\u0017�t�(����9u&a��',

That should be the decrypted response from the status message (commandByte: 8), which should be the JSON representing the state change or, worst case, a human readable string (if it's an error). The above looks like it's instead a failed decrypt. This also explains why you get the message "Data from device not encrypted". Inside tuya-mqtt there is a super simple parser which attempts to detect if the response from TuyAPI is JSON or not and outputs the "Data from device not encrypted".

So clearly the problem is that TuyAPI is returning invalid data to tuya-mqtt, data that can't be acted on, so it's just logging it (I admittedly should change the log messages, that's a holdover from the old tuya-mqtt code) and ignoring it.

Now we need to focus on why TuyAPI is returning bad data when obviously it CAN return good data since it works with tuya-cli. So, I'm going to ask you to do something obvious, double/triple check that the key in devices.conf is correct. I know that's probably not the problem, because obviously you know what you're doing, but that's by far the most common reason for invalid decrypt, so I have to ask.

Assuming the key is correct, I'm wondering if you'd be willing to share your devices.conf file with me, with key in tact (via private email of course). It could be some type of parsing error on the key on my side, but if I had the key and can try parsing the "received data" packets in your debug output above and hopefully figure out this mystery.

[Flip76]

As I copy & paste the id, key and ip from the devices conf to the tuya-cli command, I'm pretty sure the values are correct - and I just double checked.... But..... Believe it or not: Just right now the device appeard in the tuya topic:

Honestly, I have no idea why it works now... The only thing I did was to change the friendly name from "Test Bulb" to "TestBulb" and stop/start tuya-mqtt again (like I did 20 times before)...

PS: Of course I can send you my devices.conf by mail if you want

[Flip76]

I went ahead and I checked the possibilities on OH (by crosschecking with my Philips Hue devices). For the hue-devices we typically use 4 channels:

• switch for on/off
• colorpicker (for color)
• dimmer (for brightness)
• dimmer (for color temp)

The mqtt-channel in OH gives me the following choices:

[tsightler]

Hmm...interesting, I guess it would be good to know what happens if you put the space back in, if that was the only thing that changed it could be some type of weird parsing issue, although I mostly just rely on the JSON5 package for this and all of my device friendly names have spaces in them. Anyway glad it's working now, at least proves it's nothing major from a code perspective.

Regarding color, how do you set the light to white mode if it's currently a color? Is it the color temp setting? How about the dimmer? Does it behave differently if the light is in white mode vs color mode? In other words, if the light is white does Dimmer set white brightness and if the light is in Color mode does it set the Color brightness? The Tuya bulb (and some others) have separate brightness for color mode vs white mode. This is easily handled in Home Assistant since it has both "brightness", which sets color mode brightness, and "White Level" which sets white mode brightness, but I'm just not sure if OpenHAB or others have such a concept.

[Flip76]

I put the space back in and it still works... So I can only guess that it might not have been a space there before (maybe a special character caused by file-encoding with my editor)...

Regarding the color mode: Seems like OH doesn‘t work the same way like HA.

I researched the forum and I found this (basically about Philips Hue but I think same applies for the Tuya light bulbs):

https://community.openhab.org/t/hue-hsb-how-to-find-correct-color-warm-cold-white/82964/7

I am just thinking of using a dimmer item to handle the white color and a switch to toggle between white/color mode - can‘t think of another workaround...

[Flip76]

Another possible approach is described here:

https://www.openhab.org/addons/bindings/milight/

Basically they switch to white mode if the saturation is below 50%...

[tsightler]

Another possible approach is described here:

https://www.openhab.org/addons/bindings/milight/

Basically they switch to white mode if the saturation is below 50%...

Yeah, this is basically what the code does now, switches to white mode if saturation is 0%, because 0% saturation is white! But, I could change it to say 10% The 50% value seems too high to me as there's quite a bit of difference between 10% of 50%, but I can barely perceive the difference between 10% and 0% so perhaps that's a good threshold.

The only problem with this is that it doesn't address the issue for visualizations that don't have such flexibility. For example, Alexa seems to only show on/off, brightness and a preset selection of colors. I can seem to set the saturation via Alexa so there's no way to get it back to white. When I tell Alexa to set the color to white, it just sends "on", which is just weird to me, although maybe I could build a special mode that, if it received "on", but no other commands within 200ms, it assumes a switch to white mode.

Oh well, I have ideas on how to deal with this, just some minor tweaks to the logic that's already there and I think it will be mostly good.

So what's left before I can publish at least a working version 3.0? I think the following:

1) Update RGBTW white/color logic 2) Add color temp support for RGBTW 3) Rebase brightness/white level in RGBTW to 100 instead of 1000 (works for both Home Assistant and OpenHAB). 4) Write some auto-discovery rules for RGBTW for the most common values 5) Add the basic math transform functions 6) Create some documentation

My goal is to do 1-5 tonight, and then work on 6 through the week, to hopefully publish 3.0 next weekend. Then I can start working on adding additional devices as users submit working templates (or some I can just figure out based on existing devices in tuya-homebridge).

It seems like, even in it's current state, the code is mostly working for you and your devices (minus the Zigbee gateway), is that a fair statement?

[Flip76]

Have you ever tried using „cold white“ with Alexa? At least this works for my Hue bulbs...

Tuya-mqtt 3.0 already exceeded my expectations! I was almost about giving up on controlling my tuya devices via OpenHab. Thanks to you everything works like a charme now!

3 more ideas (but I think the first 2 are mainly a job for tuyapi):

• Automatically identifying the protocol version (this might be very helpful - especially for beginners)

• Handling of non-reachable devices (for example in the case of power loss). Currently mqtt doesn‘t change the state to off when the device has been toggled by Alexa/Tuya app. I guess thats the nature of those devices as they can‘t send a last will right? But as the devices get handled locally, why not set the state to off/false when the device isn‘t reachable within the network? Maybe there is a good reason not to do so but I can‘t see it at the moment...

• Automatically reread the devices.conf on changes. This might be helpful when running tuya-mqtt as a systemctl service. So there is no need to restart the service after a change in devices.conf. Maybe it‘s worth to add a quick „how-to run tuya-mqtt as service“ to the documentation - would be helpful for beginners

But all those points are just goodies! tuya-mqtt does already more than anyone would expect!

I also monitored the cpu/memory usage throughout the whole week and it‘s working absolutely reliably without any memory/cpu leak.

About the zigbee-gateway: As you said it might be not too much work to implement it. If you wanna try it, I can always get you access to my lab-env. But as discussed before, the best solution might be to directly communicate via a usb-stick with the devices (still waiting for my stick to give it a try).

If you have a paypal-me, it would be a pleasure to get you a „big coffee“ :-)

[Flip76]

PS: Whenever you feel ready to release 3.0, please let me know and I will write a review for the OH-community.

[tsightler]

Have you ever tried using „cold white“ with Alexa? At least this works for my Hue bulbs...

Interesting, I'll try it. Maybe it will work once I implement color temperature.

• Automatically identifying the protocol version (this might be very helpful - especially for beginners)

Automatic protocol detection already works if you use discovery vs specifying the IP address. You just can't use it because if your network setup. It works by listening for UDP packets on the network the device is on, but that only works if the server running tuya-mqtt and the devices are on the same network. I suppose it would be possible for TuyAPI to simply try each protocol in succession if the initial connection doesn't work, but indeed that's mostly a function of TuyAPI (although theoretically I could probably implement it on the tuya-mqtt side as well).

• Handling of non-reachable devices (for example in the case of power loss). Currently mqtt doesn‘t change the state to off when the device has been toggled by Alexa/Tuya app. I guess thats the nature of those devices as they can‘t send a last will right? But as the devices get handled locally, why not set the state to off/false when the device isn‘t reachable within the network? Maybe there is a good reason not to do so but I can‘t see it at the moment...

I'm not 100% sure what the request means. If you change the state to off when it's toggled by Alexa/Tuya app, it should definitely reflect in the UI. Certainly for me every update I make in the app is properly reflected in Home Assistant nearly instantly. Perhaps you mean if the Tuya device itself is powered off completely or something? I actually did plan to implement availability status, which in Home Assistant properly shows the devices as available/unavailable, it will be pretty easy.

• Automatically reread the devices.conf on changes. This might be helpful when running tuya-mqtt as a systemctl service. So there is no need to restart the service after a change in devices.conf. Maybe it‘s worth to add a quick „how-to run tuya-mqtt as service“ to the documentation - would be helpful for beginners

Yeah, I'll probably just copy the instructions from ring-mqtt for the most part, just update for tuya-mqtt, and I'll almost certainly create a Docker build as well.

About the zigbee-gateway: As you said it might be not too much work to implement it. If you wanna try it, I can always get you access to my lab-env. But as discussed before, the best solution might be to directly communicate via a usb-stick with the devices (still waiting for my stick to give it a try).

The biggest problem is that it would require changes to TuyAPI, which is not really my project and will require a little more digging than I have time for right now. Perhaps I will try to find a hub on ebay or something and play around with it in the future.

If you have a paypal-me, it would be a pleasure to get you a „big coffee“ :-)

It's really appreciated but I always tell people to donate to some worthy cause instead. I'm in the fortunate place to not really need much, but there are plenty of people who could use a little help these days. If you did send me an amount I'd just donate it myself, so you can cut out the middle-man.

[Flip76]

Sorry for the lack in response - I had 2 busy days at work.

• Automatically identifying the protocol version (this might be very helpful - especially for beginners)

Automatic protocol detection already works if you use discovery vs specifying the IP address. You just can't use it because if your network setup. It works by listening for UDP packets on the network the device is on, but that only works if the server running tuya-mqtt and the devices are on the same network. I suppose it would be possible for TuyAPI to simply try each protocol in succession if the initial connection doesn't work, but indeed that's mostly a function of TuyAPI (although theoretically I could probably implement it on the tuya-mqtt side as well).

You are right - I forgot about my VLAN-situation here :-)

• Handling of non-reachable devices (for example in the case of power loss). Currently mqtt doesn‘t change the state to off when the device has been toggled by Alexa/Tuya app. I guess thats the nature of those devices as they can‘t send a last will right? But as the devices get handled locally, why not set the state to off/false when the device isn‘t reachable within the network? Maybe there is a good reason not to do so but I can‘t see it at the moment...

I'm not 100% sure what the request means. If you change the state to off when it's toggled by Alexa/Tuya app, it should definitely reflect in the UI. Certainly for me every update I make in the app is properly reflected in Home Assistant nearly instantly. Perhaps you mean if the Tuya device itself is powered off completely or something? I actually did plan to implement availability status, which in Home Assistant properly shows the devices as available/unavailable, it will be pretty easy.

Sorry for the misleading information... You are right - when I use Alexa/Tuya app, the update comes instantly. But if, for expample, somebody unplugs the device from power, the last state will remain until the device will be plugged again right?

About the zigbee-gateway: As you said it might be not too much work to implement it. If you wanna try it, I can always get you access to my lab-env. But as discussed before, the best solution might be to directly communicate via a usb-stick with the devices (still waiting for my stick to give it a try).

The biggest problem is that it would require changes to TuyAPI, which is not really my project and will require a little more digging than I have time for right now. Perhaps I will try to find a hub on ebay or something and play around with it in the future.

Never mind! I'm looking forward to have a play with zigbee2mqtt once my usb-stick arrives

If you have a paypal-me, it would be a pleasure to get you a „big coffee“ :-)

It's really appreciated but I always tell people to donate to some worthy cause instead. I'm in the fortunate place to not really need much, but there are plenty of people who could use a little help these days. If you did send me an amount I'd just donate it myself, so you can cut out the middle-man.

Fair enough! I just contributed to a nearby Orphanage as here in TH they are not supported by the state and the Covid-situation hit them hard too!

[tsightler]

Sorry for the lack in response - I had 2 busy days at work.

Ha, no issues, I'm also quite busy these days. I've managed to get most of the work above done, but still need to implement color temperature controls for lights (I've ordered a light that actual has color temp to make this easier for me to test).

I think I found and fixed the hanging problem you mentioned above, it was really an error in TuyAPI that could cause set() to hang forever, especially if you didn't use await as concurrent set requests could reproduce a hang easily. This was made wrose because I had also implement fallback from DP_QUERY to sending a CONTROL with null for newer devices that don't respond to DP_QUERY, but this meant that set() was being called even for get() and had more potential to hang. I've implemented a queue and timeout for set() in my branch of TuyAPI (and submitted a PR to the original project).

I've also implement automatic discovery for RGBTW lights, it attempts to determine if the light is old/new style by querying DPS 2 and then DPS 21 and then checking the color value for HSBHEX vs HSB format. This should make most RGBTW lights work without needed any config, but of course the config options are still there..

I've also rebased all values in RGBTW lights to 100, instead of previously where they were the raw scale used by the Tuya bulb itself (255 or 1000) and I added a hsb_state topic that includes all three values in comma separated format as that's what it appears OpenHAB uses. This was easy as I built the HSB/HSBHEX types to use a components template. I'd be interested to know if that works easier with OpenHAB now.

Finally, I added some simple math expression evaluation (right now using eval but I will probably change it to use MathJS evaluate function). Use is pretty simple I think, here's a simple example of converting 1000 scale to 100:

brightness_state: { 
                key: 2,
                type: 'int',
                min: 1,
                max: 100,
                scale: 100,
                stateMath:  '/10',
                commandMath: '*10'
            }

Basically "stateMath" is the math to perform on the source value before being published as the state value, and "commandMath" is the math to be performed on the received command before being sent to the device. The min/max values are validate prior to command math. The above example takes the 1000 scale "white value" from my dimmer and translates it to 100 scale, by applying "stateMath" transform, and then, when I send values to the command topic it validates that the command is between the value of 1-100 and then performs "commandMath" to translate the value back to the 1000 scale.

For int values it performs the math and then, rounds and converts back to integer. For float values you get the decimal values. Thoughts on this feature are welcome as I wasn't sure of the best way to do it, but the above seemed pretty easy overall.

Fair enough! I just contributed to a nearby Orphanage as here in TH they are not supported by the state and the Covid-situation hit them hard too!

Thanks for that! Exactly the type of place that I would have looked for.

[tsightler]

Pushed 3.0-beta4. Adds support for heartbeat (to report device online/offline via /status topic). Includes a lot of (mostly) minor fixes to device handling, adds support for color temperature. I've acquired a few more bulbs with color temperature and also with both old and new style HSB/HSBHEX color and settings so I've been able to test a wider variety of bulbs. Overall things seems to be working quite well although there are still a few behaviors I don't like that I believe I can improve.

Still have a handful of things to add, and then lots of documentation to update, but this code it working very well for me over the last week.

I'm still looking for some feedback on the RGBTW bulb support around how to handle white/color mode and brightness. The Tuya bulbs have separate brightness for white vs color mode (you can see this in the tuya app), which is not uncommon, especially for bulbs that are much brighter in white mode than in color mode (like most Tuya bulbs). Below is the current behavior:

White mode 1) White mode brightess is reported/controlled by white_value_state/command topic 2) Color temperature is reported/controlled by color_temp_state/command topic

Color mode Color mode actually provides 3 topics: 1) Color mode brightness is reported/controlled by brightness_state/command topic 2) Color mode Hue/Saturation is reported/controlled by hs_state/command topic 3) Color mode Hue/Saturation/Brightness is reported/controlled by hsb_state/command topic

Why three topics with overlapping controls? Some automation platforms, like Home Assistant, require separate topics for brightness vs hue/saturation. I'm not sure why, but probably it makes it easier to control brightness alone vs always needing to send all three components, or maybe it's because some bulbs use a single brightness control for both color/white mode so having it separate makes sense. However, other platforms, like OpenHAB, appear to strongly prefer a single topic with all three components comma separated. Offering both makes it easy to consume this integration with either platform.

Switching between white/color mode works as follows:

1) Any time white level (white brightness) or color temp are changed, switched to white mode 2) Any time saturation is below 10%, switches to white mode 3) If hue is changed, and saturation is above 10%, switch to color mode 4) If saturation is increased above 10%, switch to color mode 5) If color mode hue/brightness is changed, but saturation remains below 10%, stay in white mode

I feel like the above is reasonable logic, simple and the code is easy/straightforward (there's something to be said for this), but it causes some behaviors that I don't really like. For example, some bulbs automatically switch to color mode if you change any color value, so, if you change hue or color brightness, even if the saturation is <10%, you'll get a quick flicker while the bulb changes to color mode and is immediately flipped to white mode. I admittedly think this will be pretty uncommon, but it just bugs me.

Another issue is that some bulbs completely ignore any changes to the color attributes when the bulb is in white mode. So, for example, if you change the color brightness value while the saturation is <10%, so the bulb is in white mode, and then later switch to color mode, the brightness will still be the old value. Other bulbs always accept changes to any value, and only switch modes on manual command to do so, which is, IMO, the best behavior.

So, the issue is that it's difficult to get consistent behavior because different bulbs behave differently and I have no way to know what bulbs behave in what way.

The code currently attempts to somewhat compensate for this by caching the HSB color state while the bulb is in white mode and basically "faking" the updates of colors, storing them for a future switch to color mode. It complicates the code, but gives reasonably consistent behavior across all bulb types.

Currently, since I'm not sure how common the use of "white level" for white brightness is outside of Home Assistant, I'm thinking I will rename the brightness topics to

color_brightness_state/command white_brightness_state/command

Feedback is welcome.

[Trauma]

I've just tested the last version with a version 3.3 RGBTWLight which is branded LSC Smart Connect LED with Home Assistant, and it just work really fine !

And i have to say that you're white/color switching implementation just feels really natural to me. Great job ! Actually tuya-mqtt implementation is way better to me than the cloud tuya api integration (or at least it works better for this bulb).

About the topics naming, since I stumble on understanding it's meaning in the past, i'll up vote that too. But in a "it works out of the box" experience it doesn't matter that much ;)

I've actually noticed only one strange behavior in home assistant, when switching from color to white mode : the bulb icon stick to the latest known color, and it do not seems to change unless you manually pick a white color in the color picker. I do not know how mqtt light integration handle this, i've seen you've added a mode_state which seems to be intended for this purpose, but could not find any occurence in HA doc.

[tsightler]

Thanks for the testing, I've really put a lot of effort into getting the logic down and it really seems to work well for me, but it's always good to have validation so I'm glad to hear it's working well for you.

I've actually already changed the naming at this point, although hopefully, with Home Assistant, the MQTT discovery just works so no real need to know about them, but I thought it might be a little more clear for others, and especially for non-HA users.

Regarding the issue with Home Assistant showing the bulb icon in color, the issue here is that Home Assistant doesn't really have any concept of white/color mode. Yes, I have a topic for it, but it's mostly for the non-Home Assistant use case. Basically, Home Assistant itself is completely ignorant that white/color mode exist. Because of this, Home Assistant always shows an approximation of the bulbs current HSB value but, when the bulb is in white mode, the HSB value does matter for the Tuya bulb as it only uses white level/color temp in that mode. So basically, the bulb is in white mode, but Home Assistant still uses the color value.

The way around this is if you set the saturation to zero, which is also a way to switch to white mode, then Home Assistant knows that there is no color and the icon changes to the normal color. I've considered automatically setting the saturation to zero whenever white mode is enabled, which would work around this limitation with Home Assistant, but this has a side effect I don't really like. Right now, assume you have to light set a color like blue, with full saturation and brightness. If you then do nothing but switch from color to white mode, and then immediately back to color mode (for example using the mode topic or the app) the light will still have the exact same color settings as before the switch, so it will still be blue with full saturation and brightness.

However, if I switched the behavior to set the saturation to zero automatically whenever white mode is activated then performing the simple step of switch from color->white->color would lead you with a bulb in color mode but with 0% saturation. That just seems wrong to me.

I do have an idea how I can use an already existing feature of the code to work around this problem with minimal changes. I'll give it a shot and see how it works.

[tsightler]

After thinking about the "Home Assistant shows color even when light is in white mode" issue for a night I came upon a solution that seemed super simple which I've implemented in a current code push. The logic is as follows:

On any switch between color/white mode, no matter how that switch is triggered, it will always trigger a resend of the HSB color states to the friendly device topics. If the light is in color mode, the lights actual HSB value is sent, but, if the light is in white mode, the H and B values are sent as normal, but saturation value is always sent as 0%. Since it's only overriding the state in the friendly topics, not actually changing the bulbs state, if you change modes in, for example, the Tuya/SmartLife app between white/color, Home Assistant will always show the correct state regardless. I also think this more accurately reflects the bulb state since white mode is effectively 0% saturation.

@Trauma If you have a chance to test these latest minor changes, I'd really appreciate your feedback.

This leaves me with just a few minor things left, I need to monitor the Home Assistant status topic so I can send state updates whenever Home Assistant is restarted, and I need to write all of the documentation. Then I can publish 3.0 and start working on expanding device support to more devices.

[Flip76]

A short update from my side: Everything works so far but I can see a lot of connection timeouts (almost every 1 minute):

Oct 14 23:39:57 openhab systemd[1]: Started tuya-mqtt.
Oct 14 23:40:17 openhab node[32087]: /etc/openhab2/scripts/tuya-mqtt/node_modules/tuyapi/index.js:337
Oct 14 23:40:17 openhab node[32087]:           reject(new Error('connection timed out'));
Oct 14 23:40:17 openhab node[32087]:                  ^
Oct 14 23:40:17 openhab node[32087]: Error: connection timed out
Oct 14 23:40:17 openhab node[32087]:     at Socket.<anonymous> (/etc/openhab2/scripts/tuya-mqtt/node_modules/tuyapi/index.js:337:18)
Oct 14 23:40:17 openhab node[32087]:     at Object.onceWrapper (events.js:420:28)
Oct 14 23:40:17 openhab node[32087]:     at Socket.emit (events.js:314:20)
Oct 14 23:40:17 openhab node[32087]:     at Socket._onTimeout (net.js:484:8)
Oct 14 23:40:17 openhab node[32087]:     at listOnTimeout (internal/timers.js:554:17)
Oct 14 23:40:17 openhab node[32087]:     at processTimers (internal/timers.js:497:7)
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Failed with result 'exit-code'.
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Service RestartSec=100ms expired, scheduling restart.
Oct 14 23:40:17 openhab systemd[1]: tuya-mqtt.service: Scheduled restart job, restart counter is at 385.
Oct 14 23:40:17 openhab systemd[1]: Stopped tuya-mqtt.

I am running tuya-mqtt as system service:

[Unit]
Description=tuya-mqtt
After=mosquitto.service

[Service]
#ExecStart=/usr/bin/node /etc/openhab2/scripts/tuya-mqtt/tuya-mqtt.js
ExecStart=/usr/bin/node --unhandled-rejections=strict /etc/openhab2/scripts/tuya-mqtt/tuya-mqtt.js
#ExecStartPost=/etc/openhab2/scripts/tuya-mqtt/initialize_topics.sh
Restart=always
User=openhabian
Group=openhabian
Environment=PATH=/usr/bin/
Environment=NODE_ENV=production
WorkingDirectory=/etc/openhab2/scripts/tuya-mqtt/

[Install]
WantedBy=multi-user.target

As you can see, before I started with:

ExecStart=/usr/bin/node /etc/openhab2/scripts/tuya-mqtt/tuya-mqtt.js

This caused the service to stop without a restart... Now I am running with the "--unhandled-rejections=strict"-option and at least the service will automatically be restarted. I don't really know why I see so many connection lost messages... Any chance that the timeout is set to low?

PS: I will pull the latest version over the weekend and will run the tests again with the tuya-lightbulbs and OH...

[tsightler]

Maintaining the connection is mostly the responsibility of TuyAPI, however, early betas had no real handling for lost connections. The code from this past weekend should have much improved behavior as it now monitors the heartbeats from the device (every 10 seconds) and forces a disconnect and reconnect cycle to start if it misses 3 heartbeats in a row (it should also update the status topic to offline in this case so that it's clear the device is currently not reachable).

During the reconnect cycle it will try every 60 seconds to find and reconnect to the device. I'd tested this pretty extensively by removing power from my devices, blocking them from the network, etc. The expectation is that there should be no crashes!

One challenge with the local connection is that most devices will only accept a single connection at a time. If you use the Tuya app for local control it will kick the script out. Also, if anything causes a timeout, like a simple wifi issue or something, the device will not always realize that the old connection has been reset, I've seen it take 3-5 minutes before the device allows a new connection. Some devices also seem to take a long time after they first startup (if, for example, you remove power) before they allow the local connections to work, this is even reproducible with the Tuya app so I don't think there's anything I can do about it other than keep retrying.

[tsightler]

Another issue I discovered is that some devices practically stop responding if you send them some value outside of their expected range, but it's not always obvious that the value is not valid. For example, I have several lights/dimmers that use a 255 scale for the brightness. Seems reasonable to expect that you can send any value between 0-255, or at least 1-255, right? Well, it mostly works, but for a couple of the bulbs, any value <25 causes the light to start timing out and being very slow to respond to any commands, and if I send it 0, they turns off completely. To make it behave like the Tuya app I have to limit the range of values sent to 25-255.

There was also pretty bad bug in versions prior to beta4 that caused values 0-1 to sometimes be sent incorrectly as a true/false bool so it could be something like that causing the timeouts. I think I've squashed all of those bugs at this point.

I'm currently testing with a fairly wide arrange of switches, lights, dimmers, and various other devices with no real issues and I've seen no timeouts in the logs during the last week.

[Flip76]

Thank you so much! You gave me the "missing link".... I forgot about my homebridge-installation... I've used a tuya-binding there ( https://github.com/milo526/homebridge-tuya-web ) to expose the devices to Apple homekit... But with tuya-mqtt I don't need that anymore... I just removed the homebridge-plugin and exposed the mqtt channels from OH to homekit - which works like a charme :-)

[Flip76]

So I can finally confirm that after removing the tuya-plugin from homebridge there was no further disconnect - absolutely stable!

[tsightler]

Thanks a lot @Flip76. It's been extremely stable for me as well over the last week. I feel like it is really close to release quality. I was chasing a couple of corner case bugs in the color/white switching logic last night, but I think I finally have that nailed, and was even able to simplify the overall code in the process. I need to implement some type of scaling limit for the backend code to address the issue I mentioned above where a device doesn't accept commands for the full range (I've been looking at how some other projects address this and it looks simple enough). Now time to write some docs.

I really can't think you all enough for the testing, really helped me understand the use cases and it was invaluable to have feedback outside of my own testing and devices. There are still little things I'd like to do, but I think the core functionality is quite solid at this point.

[Flip76]

I just dropped you a PM on the OH community platform as here it doesn't seem to be possible :-)

[Trauma]

@Trauma If you have a chance to test these latest minor changes, I'd really appreciate your feedback.

I've just tested, it does the trick just fine, well done.

I have timeout connections too, not sure if it's a discovery problem or real connections issues. I do not use ip's in the device.conf, so problem might be related to discovery feature. In fact it works fine for 10mins approx then timeouts happens. 3.0.0-beta4 was perfectly stable, and my setup did not changed since my previous tests, I've only rebuild my docker image this morning with latest commit "Simplify color logic".

2020-10-16T21:01:19.382Z tuya-mqtt:error find() timed out. Is the device powered on and the ID or IP correct?
2020-10-16T21:01:19.382Z tuya-mqtt:error Will attempt to find device again in 60 seconds

[tsightler]

It's a little confusing because there have been no changes to this part of the code since 3.0.0-beta4 and also, you state that it works for 10 minutes, which means the device is already found at least once and the initial connection is established. Once the initial connection is established the only way for this message to happen again is if the device disconnects, either due to the device closing the connection, or missing 3 heartbeats in a row (basically, the device is not responding to heatbeat packets). Thus there should be more details in the logs further up to show this.

The find() function is really outside of this script, as it is part of TuyAPI but it's not really clear how this could possibly go wrong as it simply relies on finding the device via the UDP packets that they all send every 10 seconds. The only way for this to happen is if the device is not sending those packets which implies that it really is off the network.

[Trauma]

Forget about the connection issues, it is indeed a disconnected device. 👍

[tsightler]

Thanks for confirming, I've been wracking my brain trying to figure out what else I could have missed. I've really put a lot of effort into testing the failure cases because I believe home automation must be near 100% reliable to not be more trouble than it's worth and to keep the wife acceptance factor high. That's one thing I learned when building ring-mqtt, it needs to handle and recover automatically from pretty much all failure cases. I'm not sure tuya-mqtt is quite there, but I've spent a lot of time testing things like powering off devices, restarting devices, rebooting APs for devices, blocking devices from APs, forcing IP changes, etc, because I expect tuya-mqtt to recover from all of those cases. The current versions pass all of my tests, but perhaps my tests are not fully comprehensive...yet. Long term reliability is more difficult to test.

[tsightler]

Squashed all the bugs I could find this weekend and finally managed to finish the documentation well enough to push 3.0.0 this weekend. Let's hope nothing bad snuck in with the last little changes but it seems really solid to me.

[Flip76]

I pulled 3.0.0 and will keep an eye on it :-)

[Flip76]

Will you push it to the master-branch soon?

[tsightler]

Will you push it to the master-branch soon?

It's done! Master branch is 3.0.0 as of ~12 hours ago. I still think the docs could use some work, but I'm not good at documentation.

[Flip76]

I'm blind... When I looked 8 hours ago I didn't see the master branch updated.... Maybe some Chrome-Cache issue :-( Now I see it! :-)

[Trauma]

It looks like gismocaster just stores it's config information in MQTT via retain, which I'm not a big fan of doing as I find people have a tendency to treat their MQTT broker as throwaway, they'll remove/reinstall it, etc.

As far as I can say tuya-mqtt v3 do not use retained messages at all ? From my perspective, i'dd prefer messages to be retained so devices availability persist across home assistant reboot. So actually I must restart tuya-mqtt after each HA restart, not convenient to me.

Don't you think HA discovery and status message should be retained ? Maybe it could be switched on/off through config file so everyone is happy.

EDIT : While looking at the code I've found that if "homeassistant/status" status message is resend every 30 sec. Which basically fits my request. Gonna try this.

[tsightler]

No retained messages and my goal is to always design solutions without any dependency on retained messages. I tried using retained messages for things in the early days of ring-mqtt and it was just a frustrating experiences of cases being opened because people couldn't delete devices (since config was retained in MQTT) or people deleting/reinstalling MQTT causing issues, or, in some cases, people not having any persistent storage configured for their MQTT broker (admittedly, I believe the last one is far less common these days).

However, the code as written does monitor for birth messages from Home Assistant which should be issued by default on Home Assistant 0.113 and newer (was finally made default exactly for this purpose). If it sees homeassistant/status message of "online" it knows that Home Assistant was restarted and, 30 second later, it will resend discovery and state data. This should require no special configuration in Home Assistant these days since birth messages are enabled by default. The code also monitors hass/status as well, since that was a common configuration for Home Assistant birth messages prior to 0.113. I have used this method since the earliest days of ring-mqtt and the ring-mqtt HomeAssistant addon and it's proven to be very functional, however, although it seems to work for me, I haven't fully tested the birth message implementation in tuya-mqtt as I didn't expect tuya-mqtt to be used often with Home Assistant since localtuya custom component provides similar functionality and is significantly easier to configure for most Home Assistant users.

[Trauma]

So I've made tests, and yes the whole behavior you describe happens, except for status message. Discovery, state and dps topics get populated just fine 30 sec after homeassistant/status turn to online, but status don't (I've deleted every topics before testing).

[tsightler]

Yeah, I noticed this bug about a week ago, status is only sent during device connect/disconnect event, not during publish/republish. I probably should force send it during republish. I have a small backlog of bug fixes/improvements that I'll probably try to group together into a 3.0.1 sometime soon which will include a fix for this.