MQTT discussion

[visualapproach]

What is the difference between @faboaic way to connect to home automation and MQTT? Why is MQTT better? I haven't read up on this so I know hardly anything about it. Maybe it is not even a valid question :-D

[faboaic]

MQTT is "THE" protocol for embedded IoT stuff. Supported by all kinds of home automation solutions. Used for collecting measurements of sensors but managing actors, too.

You need a MQTT broker for it, e.g. mosquitto running on a raspberry PI.

Read something about it - you won't regret. It is a very lean and well supported protocol.

There could be a topic like: /outside/pool/heater/ And then you could send /outside/pool/heater/on for start heating and /outside/pool/heater/off for stopping Or subscribing /outside/pool/heater/actual or /outside/pool/heater/target to receive the temperatures...

If you have a pool inside the house you might have /inside/pool/heater/ , too.... With /+/pool/heater/off you could disable heater for all of your pools... ;-) Different wildcards are supported.

With websockets it seems a lot more difficult (that's what I did ... node red losing the connection to the websocket after every update, etc...)

So yes, MQTT is a must have feature for all embedded things. :-)

[visualapproach]

Okay, thanks for the info! Would it replace the web interface in this application or is both wanted? /Thomas

[faboaic]

Web Interface is still necessary. Not everyone has home automation. Or as a fallback you might need to set it directly on the web interface.

Do you know Tasmota ? You can control e.g. switchable ac sockets that are based on esp8266. You can use MQTT to integrate, or directly on web interface. Or via REST-API Or Alexa ;-)

REST-API could be a simple variant, too.

Defining actions based on the called websites of the pool.... E.g. http://ip/heater/on :-)

[visualapproach]

@faboaic what is the best format to send data to broker? topic - "pool", payload - "JSON data with all parameters" or topic - "pool/heater", payload - "on" repeat for every parameter.

[faboaic]

I think the second one...

See how Tasmota is doing it: https://tasmota.github.io/docs/Commands/

Pretty clever... Empty payload means query the status or value... And with payload it is used to set / execute the command.

[visualapproach]

Yeah, I'm gonna send you a test-kit by email. I preferred to keep it similar to the websocket stuff, to reuse some code.

[faboaic]

Sounds good 👍

I tried to make some little changes to control the pool via websockets. It seems to work somehow but I need to send the command twice.... Strange.

So if every necessary functionality has a dedicated function in the code we could call that function within websockets (like I tried with F1, F2 and F3 ). Same with MQTT, web request, etc.

[visualapproach]

No you should set virtualBTN to HTR and virtual_cmd to 1 to force it on. The turnheateron () function is non blocking and has to run several times until the device is unlocked.

[visualapproach]

You didn't reply fast enough by email @faboaic ;-) Maybe wrong adress. I changed my mind and made a new branch and uploaded my embarrasing attempt on mqtt as a parallel version. You have to edit the broker in a_globals. Hope for the best!

[faboaic]

Thanks so much. Will give it a try tomorrow evening. Holiday is over, now :-(

[877dev]

Hi all,

Has any progress been made on MQTT integration? I see reference to a MQTT version but can’t see the code.

I will maybe look at it myself if not :)

[faboaic]

Hi all,

Has any progress been made on MQTT integration? I see reference to a MQTT version but can’t see the code.

I will maybe look at it myself if not :)

Check the branch. Maybe you looked inside the master?

[faboaic]

I am so sorry that I still had not found the time to compile and install the mqtt-test version.

[877dev]

Check the branch. Maybe you looked inside the master?

Yup :) Same I have been busy, I will give it a try this week!

[877dev]

@faboaic what is the best format to send data to broker? topic - "pool", payload - "JSON data with all parameters" or topic - "pool/heater", payload - "on" repeat for every parameter.

Personally I prefer a JSON message, it's all send in one and easy to parse to get the required topic:

[visualapproach]

@faboaic what is the best format to send data to broker? topic - "pool", payload - "JSON data with all parameters" or topic - "pool/heater", payload - "on" repeat for every parameter.

Personally I prefer a JSON message, it's all send in one and easy to parse to get the required topic:

Hi, maybe you should hold on just a little while to save you from doing too much work. My next version is under testing now and I use JSON more which might make mqtt conversion easier. If it behaves (doubt it) I might be able to post it tonight.

[877dev]

Cool, no problems I am enjoying look at the code anyway. I know it will be superseded but maybe I will understand more then! :)

[877dev]

This is my standard template I use for all my projects, and adapt as needed....

https://github.com/877dev/MQTT_example_template

Probably this is not much use, but as you said you are not familiar with mqtt it might help. Maybe you might notice an improvement if nothing else :)

[877dev]

Hi, maybe you should hold on just a little while to save you from doing too much work. My next version is under testing now and I use JSON more which might make mqtt conversion easier. If it behaves (doubt it) I might be able to post it tonight.

So I have been setting up influxdb and grafana over the last few days, after adding it to my IOTstack setup.

Definitely json object messages are the way to go in my opinion, it's so easy to split payloads, but much harder to combine them all in node red.

Now I am going to look at how Arduino pubsub would be used to send a json payload, and finally will get around to trying to add it to the 1.27 version, if nobody beats me to it.

I've held off trying 1.27 for now as my family will be upset if the hot tub glitches out and doesn't heat for some reason! :)

[877dev]

Ok I have added MQTT to the 1.27p1 code, currently it just publishes all json data to MQTT broker.

I will look at adding callback next. But...I don't know how to submit the code for you to look at :) Edit - how about this: https://www.dropbox.com/s/hc4fao9xpr71efg/ESP12E_Bestway_controller_v1.27p1_mqtt.rar?dl=0

UPDATED CODE BELOW...

[visualapproach]

I’ll have a look at it 👍

[877dev]

I made some small changes, seems the mqtt library got updated with new features and "set buffer size" is now a function. https://www.dropbox.com/s/k1eeb61iqrw3fyx/ESP12E_Bestway_controller_v1.27p1_mqtt_v002_public.rar?dl=0

For mqtt callback, there could be a "mqttCallback" function similar to "webSocketEvent" which goes through the cases.

Or could we have : Either "webSocketEvent" or "mqttCallback" function receives the message. Gets passed to a third function which compares the message and update variables. Then "sendWSmessage" could be renamed to "sendMessage" for example, as it just regularly updates websocket and mqtt.

Whatever you think it best of course :)

PS - finally once we are happy it's working, could there be a define statement to enable the mqtt code, so it could be merged into the main branch at some point?

[877dev]

Here's my attempt at full control with mqtt, including a test flow for node red. Seems to be working on my test rig, I need to try on the real life setup.

I really need to get on with my other tasks for now :)

https://www.dropbox.com/s/hsn5vcygwdw4wyy/ESP12E_Bestway_controller_v1.27p1_mqtt_v003_public.rar?dl=0

[{"id":"6561cda5.524294","type":"mqtt in","z":"be5275e8.e06c78","name":"","topic":"BW-1.27p1_MQTT/message","qos":"1","datatype":"json","broker":"","x":250,"y":120,"wires":[["7d61d966.e6b038"]]},{"id":"7d61d966.e6b038","type":"debug","z":"be5275e8.e06c78","name":"","active":false,"tosidebar":true,"console":false,"tostatus":false,"complete":"false","x":540,"y":120,"wires":[]},{"id":"5f5aa4ec.eebdcc","type":"mqtt out","z":"be5275e8.e06c78","name":"","topic":"BW-1.27p1_MQTT/command","qos":"1","retain":"","broker":"","x":530,"y":220,"wires":[]},{"id":"3bb3478b.302cb8","type":"inject","z":"be5275e8.e06c78","name":"0xA heater","topic":"","payload":"\"0xA\"","payloadType":"bin","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":210,"y":220,"wires":[["5f5aa4ec.eebdcc"]]},{"id":"bfe22e0c.eeef2","type":"inject","z":"be5275e8.e06c78","name":"0xC air","topic":"","payload":"\"0xC\"","payloadType":"bin","repeat":"","crontab":"","once":false,"onceDelay":0.1,"x":200,"y":270,"wires":[["5f5aa4ec.eebdcc"]]}]

[visualapproach]

The Switch/Case statements in WSevents and MQTTevent can definitely be refactored to a new void executeCommand(int command){}

[visualapproach]

PS - finally once we are happy it's working, could there be a define statement to enable the mqtt code, so it could be merged into the main branch at some point?

I hope so, or better - selectable from AP portal

[877dev]

Morning, I've been testing this for a few days and it seems stable, it works with both websockets and mqtt and is very responsive. Also included a sample flow with all the commands.

https://github.com/877dev/WiFi-remote-for-Bestway-Lay-Z-SPA

[visualapproach]

Morning, I've been busy a couple of days now, and haven't been able to do much of this, but I'll check it out. Just need more free time :-) I just tested your "...mqtt_v003_public" version and it is slow when loading pages. Maybe it works better when there exists an MQTT broker and it doesn't try to reconnect all the time?

[877dev]

No worries, I have a lot of free time for some reason, it's not normally the case :) I think it would be because you have no broker, as it runs fine for me.

OK, I feel you are interested in mqtt and want to get started! :-) To dip your toes try this software I use, and use a public MQTT broker. Should take around 10mins.

1) In Arduino code make these changes: //IPAddress mqtt_server_ip myMqttIP; // Change to the IP Address for your MQTT Server... const char *mqtt_server_ip = "mqtt.eclipse.org"; //Used for public MQTT brokers Change base topic to something unique so it can be found easily later.

2) Download MQTT explorer: http://mqtt-explorer.com/ 3) Add a connection, configure like this, save then connect:

4) Use as per below:

Note this is public, so in theory someone internet wierdo could turn on your bubbles :)

[visualapproach]

Cool! I will dip my toe, but if its too cold I will jump back to the sauna ;-) I think I figured out the unresponsiveness now. I'll upload an update. It seems to work. I'll explain later, got to go now...

[877dev]

Great, catch you later!

[visualapproach]

The problem was that mqtt.connect() takes some time. Say a second - I don't know the timeouts and stuff. Which, when repeated every loop, blocks the main loop from executing wifi stuff fast enough. It needs to loop fairly quickly. So when I put that mqtt checking in the minute timer everything loaded fast. So, I'm thinking, if possible have one version with optional web interfaces by #defines as you seem to like ;-). I'll work on it another day if I don't get any good pull reuests before that ;-). I don't think I wanna scrap the mqtt web interface if it works (which I think it does). I imagine it would be nice to change for instance the IP or password to the broker from my phone instead of recompiling and uploading. One small release note on the latest version: I had to add "mqtt" in the JSON message for the web interface to update properly. I forgot that in the earlier test versions. Maybe that affect your node red voodoo... Cheers!

[visualapproach]

Note this is public, so in theory someone internet wierdo could turn on your bubbles :)

Lol, yes that would be weird, but I guess it's a risk one has to take in order to learn.

[visualapproach]

Hey - some weirdo (me :fearful:) restarted my ESP from the MQTT explorer

[877dev]

Hey again, your message from 2 days ago did not pop up in my mail for some reason, strange.

I take it you got it working with the test broker then, cool. For me though, I just uploaded BW_1.28 and it does not connect to the broker, or the web interface I'm afraid.

I'm looking now to see if there's anything I notice..

What is the reason for using checkMqttConnection logic?

[visualapproach]

Because you cannot run mqtt.connect() every loop because it stalls the rest, like wifi handling. So only check every minute.

[visualapproach]

@877dev Did you enter your credentials in user_parameters.h?

[877dev]

I know what you are saying, but I'm pretty sure mqtt.connect() has to be in the main loop. I think it's responsible for checking callback messages, publishing other connection data. So that would grind to a halt.

In my test version it is in the loop, and is working perfectly with both MQTT and the web interface.

I can only think the MQTT web interface is causing the issue. Plus I still think there is no need for that, as changing mqtt brokers is as unlikely as moving home (most people have tens if not hundreds of devices connected to the broker).

Could you try my code with the public broker and see what you think maybe?

[877dev]

Uploaded public broker version for you to try, you might want to change the base topic to something unique.

[visualapproach]

I disagree with you sir, but thats okay. Let me state what I think. I don't mean to be rude or claiming to know what you are thinking of, but here goes: It only stalls when it doesn't find a broker.
if (!MQTTclient.connected()) { MQTT_Connect(); } This means, (probably kicking in open doors here) that if there is NO connection then execute MQTT_Connect(). Doing so slows down the loop too much. The reason it works in your loop is because it finds the broker and does NOT execute MQTT_Connect(). The MQTTclient.loop() however is probably what you mean by checking callback and stuff, and that has to be executed as often as possible.

Now I am running v1.28 and it works perfectly good on my end and I can see it on MQTT explorer....

[877dev]

Oops yes you are correct! I am a little distracted at the moment and did not realise what I was typing. Yes I meant client.loop(). Kicking in open doors, lol!

I see now what you are intending with checkMqttConnection.

But still the 1.28 code did not work for me, maybe I will try after I have eaten 🍔

[visualapproach]

Haha, I know the feeling. I'm running against mqtt.eclipse.org as you showed me. I still have to test against a local IP adress to be sure, but my fresh install on the RPi is taking forever to update something... I might try a windows mqtt server if available and easy to understand.

[877dev]

If you are installing on a Pi, I cannot recommend this enough: https://github.com/SensorsIot/IOTstack https://youtu.be/KJRMjUzlHI8 It's so easy to install mosquitto broker, node red etc.. Maybe you could join the dark side :-)

[visualapproach]

My guess to why it doesn't work for you:

You defined all features to be compiled (good) Then you entered your credentials in the code. Then the program loaded an earlier version of saved mqtt settings from LittleFS.

If that's the case, just reenter the settings via mqtt webpage and it might work.

[877dev]

Maybe, I might try uploading and erasing all data, set up from fresh.

I can't upload LittleFS OTA with pasword enabled, are you able to? There's no password box.

[visualapproach]

No but you can upload an empty mqtt.txt file via IP/upload.html If your webserver works that is... Also, the topic and client ID is changed to the version name ("BW_1.28"), not only to make your life more painful but for me to be able to change version name in ONE place only.

[877dev]

Also, the topic and client ID is changed to the version name ("BW_1.28"), not only to make your life more painful but for me to be able to change version name in ONE place only.

Yes I noticed that 😢 😄

Web page not loading at all, and no mqtt. I don't understand why the files can persist, when they are being overwritten?

Maybe it's time to flash and erase all.

[visualapproach]

Flash mem on the 8266 is divided into several partitions. When uploading the sketch it never touches the file system. I updated the firmware on my wifi router today and it decided to give all units new IP numbers. You can read the pump display just to be sure...

[877dev]

I was not clear, I do understand that part. But I'm not sure why the web interface is not loading, and wonder if it's something in LittleFS partition. But I just overwritten all that...

[877dev]

Erm ok this is all I can get: Chrome says "waiting for cache".

Not connecting to broker either.. (I did change user parameter to my IP).