MQTT - internal or external server

[dvdcut]

i was thinking to switch to MQTT gateway and after fiddling through the settings and i found old pull request https://github.com/pihome-shc/pihome/pull/23 by @paulcsf but this only read temperature from Sonoff with tasmoto sketch.

i think implementing full support MQTT will add lots of devices in compatible list https://esphome.io/ what every one else think about this? how this can be implemented? i m on youtube trying to learn mqtt

[scottagecheeseandcrackers]

• following this thread with interest - I'm currently running mosquito on my pihome-pi. I have it set up to query each zone and issue the current temp every 5 mins to mqtt topics. I have also set up boost controls via mqtt. I didn't set up the internal mqtt system configuration in pihome though. No idea how it can be implanted but it certainly opens the door to a world of other connections.

[pihome-shc]

Yes it is certainly interesting topic, agree with @scottagecheeseandcrackers, implementing full MQTT will open lots of door and add unlimited number of devices to compatibility list, current python gateway.py only support mysensors serial and network gateway. my knowledge of MQTT is very very limited all i know its pub/sub system and then we have to implement that in python.

[dvdcut]

starting point is always difficult, i m going to build myself mqtt gateway tomorrow and i will do some experiments, right now i have no idea what to expect, mqtt is between gateway and pihome or sensors need new sketch as well? but for other devices i think they will communicate directly without mysensors gateway but thats all assumptions.

[scottagecheeseandcrackers]

I'm no more than a novice who's watched a few youtube vids on mqtt...so dont quote me on this...but I expected this to be completely independent of the gateway.

My understanding is...A temp sensor would subscribe to a particular temp sensor mqtt topic and publish data at a frequency determined by that sensor. I would imagine most people would run their mqtt server on their pihome. Pihome subscribes to the same topic. Reads the values as they are published and stores them in the sql database.

Maybe there is a simple way that you could subscribe to mqtt topic in the gui and this would automatically create a new node sensor and zone.

Side note - I've seen a few posts to hack the Xiaomi LCD temp sensors. If you know them they are the circular lcd displays that show temp and humidity. Typically they link to a xiaomi gateway, but I believe you can hack them for use as mqtt room sensors. Planed to test it down the line, whenever I get more time.

[dvdcut]

How to install MQTT on Raspberry pi follow this link

1. Install the mosquitto MQTT Broker sudo apt install mosquitto mosquitto-clients

2. Enable the mosquitto broker sudo systemctl enable mosquitto

My esp8266 gateway sketch, all of this copy/paste from Pihome esp8266 and from MySensors sketch. make sure you change wifi credentials to match your home wifi.

// Enable debug prints to serial monitor
#define MY_DEBUG

//Define Sketch Version 
#define SKETCH_VERSION "0.01"

// Use a bit lower baudrate for serial prints on ESP8266 than default in MyConfig.h
#define MY_BAUD_RATE 9600

// Enables and select radio type (if attached)
#define MY_RADIO_NRF24
//#define MY_RADIO_RFM69

#define MY_RF24_PA_LEVEL RF24_PA_MAX
//#define MY_DEBUG_VERBOSE_RF24

// RF channel for the sensor net, 0-127
#define MY_RF24_CHANNEL 91

//RF24_250KBPS for 250kbs, RF24_1MBPS for 1Mbps, or RF24_2MBPS for 2Mbps
#define MY_RF24_DATARATE RF24_250KBPS

//#define MY_ENCRYPTION_SIMPLE_PASSWD "pihome2019"

//#define MY_SIGNING_SIMPLE_PASSWD "pihome2019"

//#define MY_INDICATION_HANDLER

// Flash leds on rx/tx/err
// Led pins used if blinking feature is enabled above
#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED
#define MY_WITH_LEDS_BLINKING_INVERSE

// Set blinking period
#define MY_DEFAULT_LED_BLINK_PERIOD 400

#define MY_GATEWAY_MQTT_CLIENT
#define MY_GATEWAY_ESP8266

// Set this node's subscribe and publish topic prefix
#define MY_MQTT_PUBLISH_TOPIC_PREFIX "mygateway1-out"
#define MY_MQTT_SUBSCRIBE_TOPIC_PREFIX "mygateway1-in"

// Set MQTT client id
#define MY_MQTT_CLIENT_ID "mysensors-1"

// Enable these if your MQTT broker requires username/password
//#define MY_MQTT_USER "username"
//#define MY_MQTT_PASSWORD "password"

// Set WIFI SSID and password
#define MY_WIFI_SSID "MySSID"
#define MY_WIFI_PASSWORD "MyVerySecretPassword"

// Set the hostname for the WiFi Client. This is the hostname
// passed to the DHCP server if not static.
#define MY_HOSTNAME "ESP8266_MQTT_GW"

// Enable MY_IP_ADDRESS here if you want a static ip address (no DHCP)
//#define MY_IP_ADDRESS 192,168,178,87

// If using static ip you can define Gateway and Subnet address as well
//#define MY_IP_GATEWAY_ADDRESS 192,168,178,1
//#define MY_IP_SUBNET_ADDRESS 255,255,255,0

// MQTT broker ip address.
#define MY_CONTROLLER_IP_ADDRESS 192, 168, 1, 10

//MQTT broker if using URL instead of ip address.
// #define MY_CONTROLLER_URL_ADDRESS "test.mosquitto.org"

// The MQTT broker port to to open
#define MY_PORT 1883

// Enable inclusion mode
//#define MY_INCLUSION_MODE_FEATURE
// Enable Inclusion mode button on gateway
//#define MY_INCLUSION_BUTTON_FEATURE
// Set inclusion mode duration (in seconds)
//#define MY_INCLUSION_MODE_DURATION 60
// Digital pin used for inclusion mode button
//#define MY_INCLUSION_MODE_BUTTON_PIN D1

// Set blinking period
//#define MY_DEFAULT_LED_BLINK_PERIOD 300

// Flash leds on rx/tx/err
//#define MY_DEFAULT_ERR_LED_PIN 16  // Error led pin
//#define MY_DEFAULT_RX_LED_PIN  16  // Receive led pin
//#define MY_DEFAULT_TX_LED_PIN  16  // the PCB, on board LED

#include <MySensors.h>

void setup()
{
    // Setup locally attached sensors
}

void presentation()
{
    // Present locally attached sensors here
}

void loop()
{
    // Send locally attached sensors data here
}

ESP8266 Gateway serial output, as you can see as soon as mqtt broker comes online it is connected

174056 !GWT:RMQ:FAIL
174078 GWT:TPC:IP=192.168.1.6
174111 GWT:RMQ:CONNECTING...
174154 GWT:RMQ:OK
174173 GWT:TPS:TOPIC=mygateway1-out/0/255/0/0/18,MSG SENT

[dvdcut]

while googling around to find the way to communicate with mqqt gateway i came across pymysensors library

Python API for talking to a MySensors gateway. Currently supports serial protocol v1.4, v1.5, v2.0 - v2.2. Not all features of v2.x are implemented yet.

how to implement this is another question but looks very promising and need someone with bit more knowledge on python to see if we can import this or take part of code this and implement MQTT gateway

**MQTT gateway

The MQTT gateway requires MySensors serial API v2.0 or greater and the MQTT client gateway example sketch loaded in the gateway device. The gateway also requires an MQTT broker and a python MQTT client interface to the broker. See mqtt.py for an example of how to implement this and initialize the MQTT gateway.**

[dvdcut]

ok i think i m getting somewhere with this now. in python 3 as we are on python version 3. copy mqtt.py file from here and just simply run it.

error and its fix found here on stackoverflow

root@pihome:~# python /var/www/cron/mqtt_test.py
Traceback (most recent call last):
  File "/var/www/cron/mqtt_test.py", line 2, in <module>
    import paho.mqtt.client as mqtt  # pylint: disable=import-error
ModuleNotFoundError: No module named 'paho'
root@pihome:~# 

Install some other dependencies pip3 install paho-mqtt python-etcd

and now run python script

root@pihome:~# python /var/www/cron/mqtt_test.py
Start MQTT client
sensor_update 20
sensor_update 20
sensor_update 20
sensor_update 20
Invalid <Message data="20;255;3;0;0;224">: value must be integer between 0 and 100 for object value @ data['payload']. Got '224'
sensor_update 20
sensor_update 20
sensor_update 20

[dvdcut]

we already have python script that can read the mqtt data, initially worked on by @paulcsf but since then no major work dont on MQTT,

i think i m hitting my limits here on python and i could really need some help. #305 show temperature sensor readings but i can not progress any further. as you can see temperature sensors power on and python script showing data with other info.

Mon Jun 29 22:06:22 2020 - MQTT Temperature Sensors Script Started
--------------------------------------------------------------------
{'id': 1, 'name': 'Demo', 'ip': '127.0.0.1', 'port': 1883, 'username': 'mosquitto', 'password': 'mosquitto', 'enabled': 1, 'type': 0}
Connected with result code 0
Topic:    mygateway1-in/20/255/3/0/6
Payload:  b'M'
Topic:    mygateway1-in/39/255/3/0/6
Payload:  b'M'
Topic:    mygateway1-out/39/0/1/0/0
Payload:  b'21.6'
Topic:    mygateway1-out/39/0/1/0/0
Payload:  b'21.5'
Topic:    mygateway1-out/39/0/1/0/0
Payload:  b'21.6'
Topic:    mygateway1-out/39/0/1/0/0
Payload:  b'21.5'
Topic:    mygateway1-out/39/0/1/0/0
Payload:  b'21.4'

[dvdcut]

here is the total code for mysensors mqtt gateway and one temperature sensor. i will continue spend some time you need to modify mqtt.py file

def on_message_def(client, userdata, msg):
    # We are subscribed to all, just print it for viewing
    in_str = msg.topic
    if not sys.getsizeof(in_str) <= 25: 
        print(bc.ylw + "\nSize of the String Received: ", sys.getsizeof(in_str), bc.ENDC)
        print("Date & Time:                 ",time.ctime(),bc.ENDC)
        print("Full String Received:        ",in_str)
        #because with python3 the paho library is returning a byte array not the string representation of the byte array for the payload.
        msg.payload = msg.payload.decode("utf-8")
        #in_str = in_str.decode('utf-8')
        in_str = msg.topic
        statement = in_str.split("/")
        gwpub_prefix = str(statement[0])
        node_id = str(statement[1])
        child_sensor_id = int(statement[2])
        message_type = int(statement[3])
        ack = int(statement[4])
        sub_type = int(statement[5])
        payload = msg.payload.rstrip() # remove \n from payload
        print("Publish Topic Prefix:        ",gwpub_prefix)
        print("QOS:                          "+str(msg.qos))
        print("Retain:                       "+str(msg.retain))
        print("Node ID:                     ",node_id)
        print("Child Sensor ID:             ",child_sensor_id)
        print("Message Type:                ",message_type)
        print("Acknowledge:                 ",ack)
        print("Sub Type:                    ",sub_type)
        print("Pay Load:                    ",payload)
        print("-----------------------------------------------------------")

here is the output when temperature sensors is powered on.

--------------------------------------------------------------------
{'id': 1, 'name': 'Demo', 'ip': '127.0.0.1', 'port': 1883, 'username': 'mosquitto', 'password': 'mosquitto', 'enabled': 1, 'type': 0}
Connected with result code 0

Size of the String Received:  51
Date & Time:                  Tue Jun 30 02:48:47 2020
Full String Received:         mygateway1-in/20/255/3/0/6
Publish Topic Prefix:         mygateway1-in
QOS:                          0
Retain:                       1
Node ID:                      20
Child Sensor ID:              255
Message Type:                 3
Acknowledge:                  0
Sub Type:                     6
Pay Load:                     M
-----------------------------------------------------------

Size of the String Received:  51
Date & Time:                  Tue Jun 30 02:48:47 2020
Full String Received:         mygateway1-in/39/255/3/0/6
Publish Topic Prefix:         mygateway1-in
QOS:                          0
Retain:                       1
Node ID:                      39
Child Sensor ID:              255
Message Type:                 3
Acknowledge:                  0
Sub Type:                     6
Pay Load:                     M
-----------------------------------------------------------

Size of the String Received:  53
Date & Time:                  Tue Jun 30 02:48:55 2020
Full String Received:         mygateway1-out/39/255/0/0/17
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              255
Message Type:                 0
Acknowledge:                  0
Sub Type:                     17
Pay Load:                     2.3.2
-----------------------------------------------------------

Size of the String Received:  52
Date & Time:                  Tue Jun 30 02:48:56 2020
Full String Received:         mygateway1-out/39/255/3/0/6
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              255
Message Type:                 3
Acknowledge:                  0
Sub Type:                     6
Pay Load:                     0
-----------------------------------------------------------

Size of the String Received:  53
Date & Time:                  Tue Jun 30 02:48:57 2020
Full String Received:         mygateway1-out/39/255/3/0/11
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              255
Message Type:                 3
Acknowledge:                  0
Sub Type:                     11
Pay Load:                     Temperature Sensor
-----------------------------------------------------------

Size of the String Received:  53
Date & Time:                  Tue Jun 30 02:48:57 2020
Full String Received:         mygateway1-out/39/255/3/0/12
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              255
Message Type:                 3
Acknowledge:                  0
Sub Type:                     12
Pay Load:                     0.34
-----------------------------------------------------------

Size of the String Received:  50
Date & Time:                  Tue Jun 30 02:48:59 2020
Full String Received:         mygateway1-out/39/0/0/0/6
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              0
Message Type:                 0
Acknowledge:                  0
Sub Type:                     6
Pay Load:
-----------------------------------------------------------

Size of the String Received:  51
Date & Time:                  Tue Jun 30 02:48:59 2020
Full String Received:         mygateway1-out/39/1/1/0/38
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              1
Message Type:                 1
Acknowledge:                  0
Sub Type:                     38
Pay Load:                     12.10
-----------------------------------------------------------

Size of the String Received:  52
Date & Time:                  Tue Jun 30 02:48:59 2020
Full String Received:         mygateway1-out/39/255/3/0/0
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              255
Message Type:                 3
Acknowledge:                  0
Sub Type:                     0
Pay Load:                     224
-----------------------------------------------------------

Size of the String Received:  50
Date & Time:                  Tue Jun 30 02:49:00 2020
Full String Received:         mygateway1-out/39/0/1/0/0
Publish Topic Prefix:         mygateway1-out
QOS:                          0
Retain:                       0
Node ID:                      39
Child Sensor ID:              0
Message Type:                 1
Acknowledge:                  0
Sub Type:                     0
Pay Load:                     27.7
-----------------------------------------------------------

[dvdcut]

i think my exiting code can be integrated into exiting gateway.py to accommodate mysensors mqtt gateway. but for non-mysensors mqtt nodes we have to come to some agreement, how we want to manage topics, the way mysensors mqtt gateway works is little bit messy hence "#" is subscribe to all, just watch traffic. client.subscribe("#") but for other type of mqtt devices we need to set some sort of standards/rules for topics Example:

Temperature Sensors: pihome/sensors/temperature//

Boiler Controller: pihome/controllers/boiler// Boiler Controller: pihome/controllers/boiler/all/reset/ Boiler Controller: pihome/controllers/boiler/all/off/

Zone Controller: pihome/controllers/zone// Zone Controller: pihome/controllers/zone/all/off/ Zone Controller: pihome/controllers/zone/all/reset/

[pihome-shc]

@dvdcut good work, its time to build mysensors mqtt gateway, the way mysensors works is different then other mqtt devices, all sensors, controllers works on ism band to communicate to mysensors gateway and gateway pub/sub messages, whereas sonoff with tasmota sketch communicate with broker directly. i dont have any sonoff device with sensor at the moment to do any testing.

your code is still in very early stage and requires decent amount of testing before it can be integrated into gateway.py

once basic structure is in place we can define rules for topics and do some testing

[dvdcut]

i have reached my limit of python, attached python code to read node data when it it first power etc. but i can not get it to publish in loop, if i set message_out sent status to 0 and on start the sctip it works but during execution it ignores publishing part. can some one please have look and tell me what i m doing wrong.

mqtt_v2.txt

[HiQual]

Side note - I've seen a few posts to hack the Xiaomi LCD temp sensors. If you know them they are the circular lcd displays that show temp and humidity. Typically they link to a xiaomi gateway, but I believe you can hack them for use as mqtt room sensors. Planed to test it down the line, whenever I get more time.

Hello @scottagecheeseandcrackers - did you have time to make any progress with the Xiaomi sensors?

[scottagecheeseandcrackers]

@HiQual - afraid not. I have zero coding skills to figure this one out. I did come across this article which shows you how to set it up for Home Assistant. (https://banggood.app.link/UF34Mnj0Lab) The answer may lie there. Unfortunately I don't think any of my Banggood / Ali Alibaba order will be coming through anytime soon.

[scottagecheeseandcrackers]

@HiQual - just found this article...it looks to be most of the answer. Have a read. (http://lock.3dn.ru/news/podkljuchenie_datchika_t_rh_xiaomi_k_raspberry_pi/2020-10-02-194) - (there is an option on the left side bar of the site to change the language). I'm waiting on a xiaomi temperature sensor from banggood to test this out. Looks simple enough to get the data out of the sensor. Hopefully somebody here will be able to assist with a simply script to save this to the pihome database. I will keep you informed.

[HiQual]

@scottagecheeseandcrackers - Those new Xiaomi sensors look great. I would love to have them on my walls replacing my ugly thermostats. Also their live temp and humidity readout will keep the rest of my household ( those not using the app ) happy. I hope you like them and I will be watching, eagerly awaiting news of any developments. Many thanks for the update, much appreciated.

[pihome-shc]

One option that looks nice is Xiaomi sensors but it is on Zigbee and i dont think mysensors support zigbee,

[scottagecheeseandcrackers]

This maybe an option - https://www.zigbee2mqtt.io/getting_started/what_do_i_need.html .However my first port of call is to try the xiaomi Bluetooth sensors and try and communicate sensor data via the raspiberry pi ble. I'm hoping that the data could be saved to straight into the sql database as a new bluetooth sensor, via script.

[scottagecheeseandcrackers]

Side note - I've seen a few posts to hack the Xiaomi LCD temp sensors. If you know them they are the circular lcd displays that show temp and humidity. Typically they link to a xiaomi gateway, but I believe you can hack them for use as mqtt room sensors. Planed to test it down the line, whenever I get more time.

Hello @scottagecheeseandcrackers - did you have time to make any progress with the Xiaomi sensors?

Good news. Xiaomi Mijia lywsd03mmc sensor arrived. Tried a few things with the stock firmware and couldn't get far (as its encrypted). I found a very simple website (https://github.com/atc1441/ATC_MiThermometer). Thanks to this guys great work you can update the firmware using your smartphone and hey presto - its no longer encrypted. I used a spare esp32 I had lying around - and flashed it with tasmota. Using the in built esp32 ble it automatically discovered the sensor and publishing the data (temp, humidity and battery) via mqtt. If I can just figure out how to get the PiHome mqtt python cron sketch to work - this would unlock the world of Xiaomi temperature sensors.

[pihome-shc]

@dvdcut did some work on reading mqtt data but i think he was stuck at sending data to relay over mqtt. i got different model Aqara Smart Temperature Humidity Sensor didn't get time to investigate this any further

[JSa1987]

Hi all,

I have been thinking for a while that it would be nice to integrate somehow smart Thermostatic Radiator Valves to be used as zone "controllers". This way it would be really possible to regulate the temperature in each room.

I have done some researches on this and the most cost effective solution I found would be to use some Zigbee TRVs like these https://www.aliexpress.com/item/1005001865504629.html?spm=a2g0o.productlist.0.0.554d4774EYr4LF&algo_pvid=c7c81f55-7232-43d4-b351-1d082363dff3&algo_expid=c7c81f55-7232-43d4-b351-1d082363dff3-16&btsid=0b0a01f816144717074753992edba7&ws_ab_test=searchweb0_0,searchweb201602_,searchweb201603_

I see in this thread is discussed the possibility of integrating MQTT and/or Zigbee (maybe via www.zigbee2mqtt.io ?). I think it would be nice to try to integrate not only MQTT and/or Zigbee temperature sensors but also Zigbee TRVs...

I wouldn't know where to start with the coding part but I would be help if somebody can point me in the right direction...

[pihome-shc]

@JSa1987 Pihome is open source project and every one is contributing as much as they can, MQTT would be very good integration to open wide range of devices support, @dvdcut have done some ground work for MQTT but i don't think it is finished.

[dvdcut]

@JSa1987 i did some basic work but i have very limited knowledge of python and i wasnt able to send messages in loop as we have for mysensors nodes. my script send message on start and get messages in loop.