installing MQTT bridge on ubuntu

[rhamblen]

Mosquitto is installed and working fine... using MQTT spy and seeing the tasmota devices and talking and listening to them happily.

Changed the devices to have a preface of 'smartthings' and everything still fine.

Install MQTT and MQTT bridge and these both install perfectly on same machine. configure the device.yml to only have the simple switch. configure the config.yml to have the correct ip address (with port) for MQTT broker (mosquitto). the deviceconfig to be true - so it opens the devices.yml file

everything starts but when we get to the point in Mqtt.client @ around line 583 we dont get a response to the client.on(connect,xxx) call. this implies the the bridge software is not connecting to mosquitto.

The port doesnt appear to be active, and reading through the documentation for MQTT and the code, I am suspecting we want to activate the websocket on port 8080, but I am not 100% sure if I am barking up the wrong tree.

example log.txt connectToMQTT.txt

[rhamblen]

When I run in windows, I get the same problem and stop at the same problem, but that could be for 101 reasons I've not looked at yet, so will check the setting again!

[sgupta999]

do you have username / password for your MQTT broker? IF yes have you specified them in the config file. any log form the error.log? i will read up on MQTT client too and see what is happening.

[sgupta999]

Okay I think I might know the problem specify your host as "mqtt://192.168.1.108" I do try to fix host not specified with MQTT but don't why your debug is printing Connecting to MQTT at 192.168.1.108:1883 instead of Connecting to MQTT at mqtt://192.168.1.108 (Don't worry about specifying port 1883 - I do not provide that as a configurable option.) I think definitely specifying the port is breaking the mqtt constructors host syntax.

Port 8080 is for the the bridge listening from smartthings nothing to do with MQTT and if you are using devices from devices.yml preface in config has no impact - that is for the legacy implementation

[rhamblen]

ok so I am a little closer... it is something to do with mosquitto...
either something with password and username! but my whole mosquitto service has crashed... and even mqtt_spy has stopped seeing anything. Also it looks like I have corrupted the config.ylm.

I'll rebuild everything and retest. BUT I think it was down to adding the port number.

2019-11-14 07:25:41 AM debug: client.on message complete events.js:187 throw er; // Unhandled 'error' event ^

Error: Connection refused: Not authorized at MqttClient._handleConnack (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/mqtt/lib/client.js:1076:15) at MqttClient._handlePacket (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/mqtt/lib/client.js:365:12) at work (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/mqtt/lib/client.js:283:12) at Writable.writable._write (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/mqtt/lib/client.js:294:5) at doWrite (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/readable-stream/lib/_stream_writable.js:428:64) at writeOrBuffer (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/readable-stream/lib/_stream_writable.js:417:5) at Writable.write (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/readable-stream/lib/_stream_writable.js:334:11) at Socket.ondata (_stream_readable.js:746:22) at Socket.emit (events.js:210:5) at addChunk (_stream_readable.js:326:12) Emitted 'error' event on MqttClient instance at: at MqttClient._handleConnack (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/mqtt/lib/client.js:1078:10) at MqttClient._handlePacket (/usr/lib/node_modules/mqtt-bridge-smartthings/node_modules/mqtt/lib/client.js:365:12) [... lines matching original stack trace ...] at addChunk (_stream_readable.js:326:12) { code: 5 }

[sgupta999]

Are you using the correct host format with MQTT preface?

Check your mosquitto conf file for what permissions u have. Also can u interact with tasmota devices from command console and use commands? Maybe u have some ACL rules somewhere. If you can acces tasmota device and use MQTT commands from there then this should have worked unless U have some specific MAC addresses or IP address based rules Any chance u can try without username/passwd ?

[rhamblen]

just updated the previous comment... I have had a catastropic crash of mosquitto and a corrupt config.yml

[rhamblen]

OK so I rebuilt everything! didnt use the 1883 in the address and I am now listening to localhost:8080 which means I should be able to get everything sorted. I'll document all the steps to get to this point as there is some ambiguity in some of the documentation.
I'll have it ready by sunday, as I also want to get the testing on the devices sorted.

thanks for the help and the guidance.

[sgupta999]

thats awesome - just FYI - if you make changes in devices.yml to add new devices you do not have to restart the mbs-server.

You can just go to the smartapp on the mobile app or online and hit save again in smartapp settings. That reloads all devices in devices.yml and updates all subscriptions from the IDE.

so hotswap of settings - should make it easier to add/troubleshoot device adds

[rhamblen]

update for you... had some intersting problems trying to delete the old app in smartthings so I could reinstall it and basically it wouldnt delete. I also had a problem where the app and the ide had different Device Netword IDs and it took a few attempts for me to force the value via the IDE.

Suddenly the bridge started talking!!!!
I can see the MQTT messages coming from the device (and MQTT spy) and this refreshes the switch within the time it takes for me to look from the screen to the phone on the desk!

When I turn the switch on on the phone, I see the message to say ST has turned on the switch in the bridge, but it doesnt sent a message. This is obviously a settings thing which I will play with.

I can see it isnt picking up the device name but is picking up the command

2019-11-14 23:28:21 PM debug: From ST: - switch - on

However the good news it it does work! and I am running this on Ubuntu. which means it will also work on a Pi. So we have both a windows box and a unix box both able to act a a bridge... yippeee

I'll play with the device.yml and see what devices I can get working... the iFan03 should be interesting and for me the best news is going to be getting the shelly4Pro working which isnt able to run tasmota but I should be able to make a MQTT message to read the status and turn it on... also get it to read the power for each channel.

All looking good so far.

[rhamblen]

ok so the winston.debug at line 293 was displaying topic which had just been set to "", it should have shown device which was picked up from req.body.name

then in the device.yml we missed the publish command out so it didnt have a command to do which is why it saw the ST command but did nothing.

so it all works.

[sgupta999]

sorry kind of difficult to follow - so does it work?

For any commands that you wish to send from ST - the corresponding attribute and publish topics should be in the devices.yml.

for any commands you wish to send TO ST - the corresponding attribute and subscribe topics should be in yml.

So not sure - did you get it working?

[sgupta999]

Funny I am having a similar issue with ST IDE for a new app that I am working on. "had some intersting problems trying to delete the old app in smartthings so I could reinstall it and basically it wouldnt delete. I also had a problem where the app and the ide had different Device Netword IDs and it took a few attempts for me to force the value via the IDE."

for the life of my cannot figure out how to get the old version of a smartapp with a orphan child to uninstall/delete

[rhamblen]

sorry the problems of working late and braindumping!!! yes it works!!!

BUT work needed on the devices.yml as the defaults are missing some bits and this is very confusing if you cant debug things and work it all out.

[sgupta999]

I have a detailed example with comments in the devices.example file which should cover every scenario - devices.yml is really the easier part.

Hopefully not too much debugging required - just 1 to 1 mapping that is printed in the log - i would set log status to info as debug is too verbose

[sgupta999]

Also look in the data directory for detailed and summary state information to see exactly what is getting subscribed and published to

[rhamblen]

Initially I was using the simple switch from the examples and it didnt didnt have a command to translate the ST switch control back to MQTT... was easy to identify and fix.

I can now use the sonoff mini and the sonoff ifan03 light swithc (not the fan control yet) Going to write a ST device for the 3 switch touch, and just looked at how to control the fanspeed. Actually I can control the fan speed I just need to read the status of the fan speed as only xxx/RESULT gives it when you change the speed.

Looking at the shelly pro 4 but at the moment it may not work as you cant set a preface so you have 'shellies/' forced upon you! but I think I can do a short term workaround! shame you cant subscribe to multiple prefaces.

I will write all the steps from mosquitto to up and running and send the document over to you as the code clearly works, its all the little things about the setup and configuration that are important.

[rhamblen]

Oh as an aside... I have Betts tasmota at the moment and when I turn a switch on/off in ST from his device, it instantly responds on yours. BUT when I use yours (as we both mentioned) it takes ages to refresh on Bretts... So I am going to change over all the tasmota switches in the next couple of weeks. Much happier having fast response.

[sgupta999]

Couple of points

You don’t need the smarttthings preface. You can use the Shellies if u want.

You can have infinite prefaces and remember I also support wildcards so u can even create an MQTT spy for ST if u wanted to. Just subscribe to topic # and u can have all ur MQTT traffic sent to ST.

Think about it this way The server is mapping [device name, attribute, command] [payload] from ST to any [topic][payload] you can come up with and vice versa.

So [Fan,switch,command][off] can be mapped to [shellies/whatever/blah/blah/blah][payload] as long as your MQTT broker knows where to send the topic and payload to and a device receiving it knows what to do. So this [topic][payload] will go in publish.

Similarly if u receive [shellies/whatever/blah/blah/blah1][payload] from Shelly u will put it under subscribe under the right [device name, attribute(switch,contact), command][payload] structure for subscribe.

For multi channel devices u don’t need a new device type. Say in tasmota the commands are /myprefixornone/cmnd/tasmotadevicename/power1 on on channel one and /myprefixornone/cmnd/tasmotadevicename/power2 on, on channel 2

I just create 2 individual switches. 1 switch will subscribe and publish to all /topic/power1 events and switch 2 will subscribe/publish to all /topic/power2 event

For scenarios where a generic subscription /topic/RESULT provides multiple state/device information you have the payload mapping under command. Usually tasmota has /topic/RESULT/POWER1 or such

If you send me the device attribute command from ST that u want to be mapped to specific topic to be published/subscribed for devices - I can probably do it very quickly. Maybe pick your most complex scenario and that might help

[sgupta999]

Just remember if u r using devices.yml preface has no bearing. It is for the backward compatible legacy hassio - ST integration that does not use a devices.yml

Look at my RF device using sonoff RF bridge with tasmota example where I used a rule to extract the relevant value from a JSON name/value pair payload and resend a command that I could subscribe to.

There really is no MQTT-MQTT communication that u couldn’t do it with this

[sgupta999]

Brett’s lag was my primary reason to do it - and the device handlers I wrote also let me see which tasmota switches were rebooting more often and troubleshoot a bunch of other issues with tasmota itself

Once this was up and running I did a mod on a sonoff mini to make it a 2 channel device so i could fit it in a crowded switch gang box

[sgupta999]

Living Room Light: subscribe: switch:
smartthings/stat/sonoff1/POWER: command: 'OFF': 'off' 'ON': 'on' publish: switch: smartthings/cmnd/sonoff1/POWER: retain: 'false'

So this was the simplest example in the devices.yml which should have worked for the simplest of switches. You mentioned u found something missing for your environment. Maybe if u point me to what didn’t work for you I could a better sense of ur setup. I would have thought For most scenarios a standard virtual switch mapped to a tasmota device would have worked

[rhamblen]

the reverse of the command to publish... the command wasnt pushed back to MQTT simple to fix and obvious when I saw it not working and checked the code.

Living Room Light: subscribe: switch: smartthings/stat/sonoff1/POWER: command: 'OFF': 'off' 'ON': 'on' publish: switch: smartthings/cmnd/sonoff1/POWER: command: 'off':'OFF' 'on':'ON' retain: 'false'

[rhamblen]

re lag - yep you said that to me and I agree. it is a pain. Interesting comment about reboots as I've never really been aware of the issue but must have been having it. Have you consider doing something with the LWT message?

Curious about a two channel mod on a mini! I have similar problem.

re the prefix... useful to know. I was going to ask what it was for, so it sounds like (and I'll test this now) we dont need it. It explains why there was no change when I changed from lower case to upper case.

I'll keep you informed of the testing but basically it looks like it is working. I'll send you my device examples when I've completed them. Oh and you preempted a question about why your receiver example didnt have the prefix!!!!

Thanks a lot and speak later

[rhamblen]

just done initial testing with shelly4pro on channel 0-3 and it works in both directions. i'll build the energy monitor test and then descide on which way to build the ST device handler. Yippee!

[sgupta999]

So couple of things: The command sections specifies if you need to translate the payload from one environment to another. Smartthings will only turn off the switch if it is lower case 'off' and will now work with uppercase "OFF". Since Tasmota stat commands sends uppercase OFF , in subscribe i have to translate OFF to off. Tasmota on the other hand works with both upper or lower case off so I do not need the translation and you do not really need the command section for the publish section

So if command is specified it will do the translation otherwise it will send the payload without translation and you should see it in the log info.

The devices example is old - i have since included LWT status check in both my devices.yml and in my implementation of the Tasmota devices - that already includes

so sub topics for tasmota are for e.g. smartthings/tele/pir1/LWT: smartthings/stat/pir1/STATUS: smartthings/stat/pir1/STATUS2: smartthings/stat/pir1/STATUS5: smartthings/stat/pir1/STATUS11:

you can subscribe to anything you want but your device implementation then needs to be able to do something with it when they get the message - my device handlers processes these message and display them in the ST.

When you get to the point there is a lot you can also do on the IDE side to customize display for information you receive from the server. Check one of the device handlers and you will get a good sense.

With every kind of IOT device out there you will have very limited control on what topics they publish or subscribe to - thats why i built this to really hopefully work with any MQTT device - I really think it will.

To make it more user friendly in my parseMQTT function or handlersubscription and push functions it will be easy functionality to pass controls to third party modules to do platform specific transformation.

So you could connect to something other than ST on the client and you could plug in say HassIO MQTT rules engine on the server side to process more complex rules.

[sgupta999]

What ST device handler are you trying to build. Most basic ones (switches, contact sensors etc.) should work out of the box unless you want more information than the basic transactional commands. OR you find build a composite device using controls and statuses from multiple devices. Like a turbocharged grouping functionality.

[sgupta999]

One ugly aspect to the code is - if there was no check you would get infinite loops - where ST turns a device off, device gets it, sends status update, we subscribe to it and redo the event and so on and so forth. So between ST not publishing events with no state change and some very ugly logic in the mbs smartapp there is a lot happening behind the scenes to prevent the infinite loop scenario. That path is platform dependent - couldn't figure out a way universalize that aspect

[sgupta999]

Here’s my sonoff hack

[sgupta999]

All ESP8266 chips have access to mqtt library so I will be using wemos D1 - perhaps flashed with tasmota, perhaps not - to build a garage door opener with temperature sensor - that will be an interesting project

[rhamblen]

Love the hack... but have you looked at the shelly products? The shelly 1 and 2 can be flashed with tasmota and the shelly2.5 has 2x relays. Both are fractionally smaller than the sonoff mini. Many people are using the shelly2.5 as roller shutter controller.

Size | 41mm x 36mm x 17mm

https://shop.shelly.cloud/ https://shelly-api-docs.shelly.cloud/#shelly2 https://github.com/arendst/Tasmota/wiki/Shelly-2.5

Re what am I trying to do... mostly switches with tasmota but also the shelly4pro which is a different chip set so wont work with tasmota. Moving to MQTT is not a problem for me and it will give more more things I can control as noone has done a good ST device handler for the shelly4pro.

BUT the shelly 4 also has power monitoring, so I want to do a device handler for showing power.

Also the iFan03 which is light and 3 fan speeds. I rewrote the device handler for Bretts which works with the iFan03 and is better graphics. I will modify that to work with MQTT.

Everything is in your stuff, but its getting your head around things which are second nature to you and not clear... eg dont use a port number on the MQTT host. the Prefix isnt needed. Make sure you set the deviceconfig to true. Every tip you have given me has helped me understand what is happening, so I am very greatful. And all the fault finding exercise helped me look at the code and understand what you were doing so I better understand it all.

Now I'll just write a couple of device handlers to manage the bridge to my devices. :-)

[rhamblen]

I use D1s to control my MiLights (cheap alternative to Hue lights) from smartthings... I also have a ST_anything controller connected to my smoke alarm; and one to manage door contacts and temp in the shed. Love those little things. The MiLight controler also works with MQTT but I dont see a need as the ST device handler works fine.

[sgupta999]

Shelly's do look interesting marginally smaller. I am just too vested in the sonoff universe. I am sure sonoff will come with a 2 channel mini and they will again corner the market - besides they are a fraction of the price - i have 20 of them at around $5/pop. I really want to use PIR and temperature sensors and there sonoff would work better or the D1s. IMHO D1 flashed with Tasmota are way more stable then sonoffs, but the sonoff and shellys have the full packaging. I mean sonoff already has the GPIO they just need to expose 3 pins to add a plug and play smaller footprint relay.

I might have to get them just for the shutters. How are you finding the logs. The winston upgrade was a major refactoring. but I create the info messages so one could easily follow A->B and B<-A

I am finally getting around to using my old Etekcity RF plugs with MQTT and just completed a full blown virtual thermostat device handler to integrate heaters, window AC and contact sensors.

You could literally use a D1 with tasmota or direct MQTT to anything and bring it in ST. I was seriously thinking about moving to HassIO because of MQTT but feel like now I have a workable solution. You should be able to add a new device now within 5 minutes for an existing device handler.

Its been fun playing with these - I have a sonoff controlling my 240v pool motor and a $10 spend with a temperature sensor has made my old $300 pool timer and freeze guard obsolete at 1/20th the size and 1/20 the price.

[rhamblen]

re the logs... because I am working in Ubuntu, I have the test system where I use Putty and I can read the messages live as I use the system. The messages help a lot as I am watching what happens with all the changes and also I can easily stop and start the bridge for testing. (easy to fix errors as I bring more switches online.)

I love that I can mix different systems... the shelly4pro is working with default firmware and the other devices are all using tasmota. they all work perfectly so far at just switching, though I am trying to get some other features to work but I want to play and learn before I ask.

Intestingly, I have a pool I use in the summer and one of the things I wanted to do was manage the pool heater. I was going to use a shelly 1 PM and then I could monitor the power and control the heater. I was going to just use a custom device on the D1 to monitor water temperature and air temperature and then see how efficient to heater is and only use it when water temperature is below a specific temperature.
I also wondered about building a watering system to turn on the hose pipes... you can ignore watering if it has rained but turn them on if the ground is dry.
Both these ideas would be as you say a fraction of the cost of an off the shelf device. But thats ideas for a fututre project for spring time! I've got pleanty of other things to do right now.

As an aside, in UK the shelly1 and the mini are similar priced, but mini is much cheaper when bought in bulk.

[sgupta999]

we actually build a automatic watering system for my kids school project. With the wemos outside I am always worried about weather proofing it .

I am curious you mention you have need to keep restarting the server - that means somethings must not be working correctly. As I mentioned before you can reload the devices.yml by just going to smartapp settings screen and hitting save to update the server. I hope there isnt some random crash thats causing you to do so.

I am glad in your heterogeneous system you are not getting any looping or ghosting - that was a PIA to implement

[rhamblen]

Re the waterproofing... we use whika boxes in the UK for outdoor mains waterproofing, and these are great. the 206 might be the right size for the D1

the rebooting is because sometimes I was getting the subscribe reporting an error. this was mainly because (and I think this might be a linux thing) the fist device in the examples reports errors when used. This was one of my origional problems and why I couldnt get things to work. In the example it was giving lots of different errors but the main one was incorrect indentation.

The next thing was there was a problem in the device handler. (can you add version numbers to the device handlers and the smartapp - i went to do a change and found you had done it already)
I was using the full switch handler and found the update.Tiles was missing all the code to interpret StatusSTS and so it was giving me Nulls as results for some of the items. I've fixed it now.

TasmotaSensorSwitchFix.txt

I dont appear to be getting ghosting, but I think I might be getting some nesting in processMQTT, it is doing what I expect, but it might cause a problem... ie sometimes it starts processMQTT but never finishes it. As I said I expect this, but not sure if it creates a problem. Also I can see switch bounce or maybe MQTT echo... so i see the message about duplicate MQTT message sometimes.

Ill be a little delayed in the documentation as I wont have time today... forgot I was out!
I've outlined the installation documentation and cut a load of detail and just said refer to google and youtube for this piece! (mosquitto and mqttspy - the latter has been a saviour) I want to spend time on the devices, devicehandler and smart app; as this is where all the challenges are if you want to set things up and where a simple type can make a mistake.

[sgupta999]

Ooh!!! You have a severely outdated device handler. I will try to update all device handlers later today on github - sorry still a work in progress.

So what I found was - SmartThings groovy does not do bitwise operators very well and it could also be the because of the next issue which I just discovered earlier this week working on my Thermostat DTH.

Since the status messages are coming very quickly the state object is getting corrupted - so the bitwise operator state was being updated inconsistently. I am going to change state to atomicState if possible ( might have to refactor somethings but think it should be possible) by getting 5 status one after other not working for ST

I have also added an issue for know bugs and releases and updated mbs-server.js

Official updated release versions will be tied to NPM

Have to put some thought in to versioning

[sgupta999]

Do you know how to create a docker image? When I have time I will eventually do it - but haven’t done it so have to read up about it

[sgupta999]

Like I said I have used the sever on pi before and later moved to windows so the may not necessarily be a Linux thing. I just copy and paste from an existing device to create a new one (like have a clean full example without comments and copy/paste that)

Also in the data folder state.json contains all devices in the config file written as the Jason file. Just remember the 3 dashes —- On top of the yaml file - not sure if that is the issue

On the waterproofing front here in the US anything mains is at least 30 feet away from the my pool - or where u would want to put a sensor - so either run ugly extenders or figure out how to bury wire under ground. We also have waterproof boxes attached to walls but I wanted to have a stand-alone waterproof box for the wemos - I could do it - but not set up for it right now

[rhamblen]

re docker - sort of but not well enougth to explain... I havent done one in a while and followed other people's guides.

re waterproofing... if I properly do a wiska box install, you fill it with a silicon gel (you wont ever get the stuff out again!!!) but it will be a super sealed item which is at a minimum IP67. I could put one of these a foot or so away from the pool, but run DC on a longer cable to it so the mains is out of the way.

Re device handlers (and other code) can you just put a version number and history in it so we can keep track.

Re outdate... if I wasnt outdated and I had problems, I wouldnt have learned so much about how your code works... now I understand, the rest is easy (ish).

[sgupta999]

Finally got all the device handlers and the mbs-server known issues fixed. Please download codebase and devicehandlers and example config files again form github and try.

While the MQTT wildcard works if there are overlapping subscriptions with wildcards for the same topic currently only the first one will get the update - i have a fix in mind but that will come later.

For now try these and see if things are rock solid.

thanks again for all your help.

[rhamblen]

woks on ubuntu