A usecase retrospective.

[sueastside]

Hello,

I don't directly have questions or issues, I just wanted to share my setup in case it could help or inspire others.

So my old setup consisted out of multiple arduino nanos in my breaker box, they drive several relay boards and read the switches around my house. They are connected to a TL-WR703N with OpenWRT which provides the networking and ran a couple of bash scripts to provide mqtt functionality. The arduinos' firmware had hardcoded mapping of inputs and outputs. So reflashing and rewriting of bash scripts was required every time I moved some stuff around. So I was looking for something more elegant.

The new setup looks like:

TL-WR703N with OpenWRT with the arduinos connected, the serial is then forwarded over the network (the wr703n has a small flash)

cat /etc/config/socat
config socat 'arduino_1'
    option enable '1'
    option SocatOptions '-d -d OPEN:/dev/ttyUSB0,b1000000,raw,echo=0,hupcl=1,cs8 tcp-listen:3002,reuseaddr'

A RPi with mqtt running and connects a local file to the networked serial with socat again. socat -d -d -d PTY,link=/dev/ttyUSB0,echo=0,raw,b1000000,perm=0666 TCP:openwrt:3002

And ofcourse has serial2mqtt running with serial2mqtt.armv7l serial2mqtt.json -m 'localhost' (Small weirdness here, if I don't specify -m it still tries to connect to tcp://test.mosquito.org ?)

{                                                                                                      
    "mqtt": {                                                                                           
        "connection": "tcp://localhost:1883"                                                            
    },                                                                                                  
    "serial": {                                                                                         
        "baudrate": 1000000,                                                                            
        "ports": [                                                                                     
                "/dev/ttyUSB0"                                                                          
        ], 
        "protocol":"jsonArray" 
    } 
}

The arduino firmware does the following;

It first subscribes to /devices/devicename/config and waits for it to be received.

This is a retained message on the mqtt broker that configures that specific device; the format looks like PIN:TOPIC:TYPE multiple entries separated by |

e.g. 5:kitchen/lights/1/control:R|6:kitchen/buttons/1/control:B|12:arduino_1/debugled/control:R

• Relay: Set pin to ouput, subscribe to topic and wait for commands.
• Button: Set pin to input, debounce it and publish to topic when state changes.

When it is configured it sets up a loopback ping to keep the connection alive.

The arduino code I wrote could probably use a lot of improvement, I wrote it on create.arduino.cc, so I tried to keep the external depencies to a minimum.

This setup has been running fine for a week now, so thank you for having made my life so much easier :)

#include <Arduino_JSON.h>
#define TASKER_MAX_TASKS 1
#include "Tasker.h"

Tasker tasker;

const short LED = 4;

volatile unsigned long lastSeen = 0; 
unsigned long pingPayload = 0; 
enum Connection {DISCONNECTED, CONFIGURING, CONNECT, CONNECTING, CONNECTED};
volatile Connection connection = DISCONNECTED;

enum Type {BUTTON = 'B', RELAY = 'R'};
typedef struct record_type
{
      short pin;
      String topic;
      Type type; 

      bool swState;
      bool swPrevState;
      bool swDebouncedState;
      bool swPrevDebounceState;
      long prevTime;
};
record_type records[8];
unsigned short numberOfRecords = 0; 

const unsigned int debounceDelay=100;

String getValue(String data, char separator, int index)
{
    int found = 0;
    int strIndex[] = { 0, -1 };
    int maxIndex = data.length() - 1;

    for (int i = 0; i <= maxIndex && found <= index; i++) {
        if (data.charAt(i) == separator || i == maxIndex) {
            found++;
            strIndex[0] = strIndex[1] + 1;
            strIndex[1] = (i == maxIndex) ? i+1 : i;
        }
    }
    return found > index ? data.substring(strIndex[0], strIndex[1]) : "";
}

class Mqtt {
  public:
    static String device;
    static void publishDevice( String topic, String message, short qos = 0, bool retained = false ) {
      Serial.print(F("[1,\"devices/"));Serial.print(device);Serial.print(F("/"));Serial.print(topic);Serial.print(F("\", \""));Serial.print(message);Serial.println(F("\"]"));
      Serial.flush();
    }
    static void subscribeDevice( String topic) {
      Serial.print(F("[0,\"devices/"));Serial.print(device);Serial.print(F("/"));Serial.print(topic);Serial.println(F("\"]"));
      Serial.flush();
    }
    static void publish( String topic, String message, short qos = 0, bool retained = false ) {
      Serial.print(F("[1,\""));Serial.print(topic);Serial.print(F("\", \""));Serial.print(message);Serial.println(F("\"]"));
      Serial.flush();
    }
    static void subscribe( String topic) {
      Serial.print(F("[0,\""));Serial.print(topic);Serial.println(F("\"]"));
      Serial.flush();
    }
    static void ping() {
      publishDevice(F("ping"), String(pingPayload));
    }
    static void subscribePing() {
      subscribeDevice(F("ping"));
    }
    static void handleLine(String& line) {
      JSONVar root = JSON.parse(line);
      onMqttMessage((const char*)root[1], (const char*)root[2]);
    }
    static void handleConfig(String& message) {
      connection = CONFIGURING;
      numberOfRecords = 0;
      int i = 0;
      String entry = getValue(message, '|', i);
      while(!entry.equals("")) {
        lastSeen = millis();
        handleEntry(entry);
        entry = getValue(message, '|', ++i);
      }
      connection = CONNECT;
    }
    static void handleEntry(String& message) {
      record_type& record = records[numberOfRecords++];
      record.pin = getValue(message, ':', 0).toInt();
      record.topic = getValue(message, ':', 1);
      record.type = (Type)getValue(message, ':', 2).charAt(0);

      if (record.type == RELAY) {
        pinMode(record.pin, OUTPUT);
        Mqtt::subscribe(record.topic);
        Serial.print(F("Relay defined on "));Serial.print(record.pin);Serial.print(F(" "));Serial.println(record.topic);
      } else if (record.type == BUTTON) {
        pinMode(record.pin, INPUT_PULLUP);
        Serial.print(F("Button defined on "));Serial.print(record.pin);Serial.print(F(" "));Serial.println(record.topic);
      }
    }
    static void onMqttMessage(String topic, String message) {
      String relativeTopic = topic;
      relativeTopic.replace("devices/" + device+"/", "");
      lastSeen = millis();
      if (relativeTopic.equals("config")) {
        handleConfig(message);
      } else if (relativeTopic.equals("ping")) {
        if (pingPayload == message.toInt()) {
          pingPayload = millis();
          Serial.println(F("pings match!"));
          if (connection == CONNECTING) {
            connection = CONNECTED;
          }
        } else {
          Serial.print(F("ping content differs!"));Serial.print(pingPayload);Serial.print(F(": "));Serial.println(message);
        }
      } else {
        bool found = false;
        for (short i = 0; i < numberOfRecords; i++) {
          record_type& record = records[i];
          if (record.type == RELAY && record.topic.equals(topic)) {
            if (record.type == RELAY) {
              digitalWrite(record.pin, !digitalRead(record.pin));
            }
            found = true;
            break;
          }
        }
        if (!found) {
          Serial.print(F(" Mqtt Message arrived "));Serial.print(topic);Serial.print(F(": "));Serial.print(message);
          Serial.println();
          Serial.flush();
        }
      } 
    }
};

String Mqtt::device = "arduino_1";

void setup() {
  Serial.begin(1000000);
  pinMode(LED, OUTPUT);
  while (!Serial) {
    ;
  }

  digitalWrite(LED, HIGH);
  tasker.setInterval(connectionCheck, 1000);
  digitalWrite(LED, LOW);
}

unsigned int timeout = 5000;
void connectionCheck() {
  unsigned long currentMillis = millis();

  if (connection == DISCONNECTED || (currentMillis - lastSeen) > timeout) {
    connection = CONFIGURING;
    digitalWrite(LED, LOW);
    Mqtt::subscribeDevice("config");
    lastSeen = currentMillis;
    timeout = 5000;
  } else if (connection == CONFIGURING) {
    digitalWrite(LED, !digitalRead(LED));
    Mqtt::ping();
  } else if (connection == CONNECT) {
    Mqtt::subscribePing();
    connection = CONNECTING;
    timeout = 2000;
  } else if (connection == CONNECTING) {
    digitalWrite(LED, !digitalRead(LED));
    Mqtt::ping();
  } else if (connection == CONNECTED) {
    digitalWrite(LED, HIGH);
    Mqtt::ping();
  }
}

String line;
void loop() {
  while (Serial.available()) {
    char ch = Serial.read();
    if ( ch == '\r'){}
    else if ( ch == '\n' ) {
      Mqtt::handleLine(line);
      line = "";
    } else
      line += ch;
  }
  tasker.loop();
  readButtons();
}

void readButtons(){
  for (short i = 0; i < numberOfRecords; i++) {
    record_type& record = records[i];
    if (record.type == BUTTON) {
      record.swState = digitalRead(record.pin) == HIGH;
    }
  }
  debouncePins();
  checkStateChange();
}

void debouncePins(){
  unsigned long currentMillis = millis();

  for (short i = 0; i < numberOfRecords; i++) {
    record_type& record = records[i];
    if (record.type == BUTTON) {
      if(record.swState != record.swPrevState){
        record.prevTime = currentMillis;
      }
      if(currentMillis - record.prevTime > debounceDelay){
        record.prevTime = currentMillis;
        record.swDebouncedState = record.swState;
      }
      record.swPrevState = record.swState;
    }
  }

}

void checkStateChange(){
  for (short i = 0; i < numberOfRecords; i++) {
    record_type& record = records[i];
    if (record.type == BUTTON) {
      if(record.swPrevDebounceState != record.swDebouncedState){
        if(record.swDebouncedState == 1){
          Mqtt::publish(record.topic, F("OFF"));
        }
        if(record.swDebouncedState == 0){
          Mqtt::publish(record.topic, F("ON"));
        }
      }
      record.swPrevDebounceState = record.swDebouncedState;
    }
  }
}

[vortex314]

Hi, thanks for the feedback. Interesting setup you have established, I guess it's used as a Home Automation system. Thanks for sharing your experience , it always gives some satisfaction if something I created can be of use to somebody else. Just wondering on your setup how many arduinos you can connect to the TL-WR703N, it looks like having only a single serial port. Looking at your setup I was also wondering if the TL-WR703N couldn't be replaced by a Raspberry Pi Zero. I'll certainly look into the the weirdness/bug of the command line arguments in serial2mqtt. Lieven

[sueastside]

Hey,

No thank you for sharing Lieven!

It is indeed an home 'automation' system. Not so much automated yet though, but the potential is there. Current uses are; turning everything off when everyone is away(wifi presence of phones), turning hallway lights on when someone comes home, toggling things off/on with MQTT Dash on phone, opening the front door with electric lock, turning lights on in the morning and bathroom heater in the winter and a self-destruct.

All outlets and lights power runs directly to the breaker box connected to the relays, all switches are cat5 cable that also runs to the breaker box. Both connect to the nanos (22pins at less than 2euros a piece, make them cheap and capable multiplexers(and easy to replace))

There are 3 nanos connected to an usb hub connected to the TL-WR703N, so not to the TP's serial port directly. (so theoretically 127 :) ) Sure I can replace everything with a Pi zero :D (the TP is connected over ethernet though, not wifi) But it is an existing home automation system I build a while back, with 'affordable' hardware that was available at the time. I might replace the TP with a RPi that also runs Home Assistant or something, but not sure how the faster hardware will be affected by the noisy RF environment of the breaker box. (My current RPi 3 is on my desk, hence the socat trick) We will see.

[banier1]

Sueastside, I'd be very interested to understand how you set-up Serial2Mqtt for your home automation project.I'm trying to do something very similar to you by connecting Arduino to Raspberry Pi, for the Arduino to read button statuses (input) and drive relays (output).  Would you be able to help, as I cannot understand the instruction on the Git? Kind Regards,Dan On Wednesday, 29 July 2020, 22:14:33 BST, sueastside notifications@github.com wrote:

Hello,

I don't directly have questions or issues, I just wanted to share my setup in case it could help or inspire others.

So my old setup consisted out of multiple arduino nanos in my breaker box, they drive several relay boards and read the switches around my house. They are connected to a TL-WR703N with OpenWRT which provides the networking and ran a couple of bash scripts to provide mqtt functionality. The arduinos' firmware had hardcoded mapping of inputs and outputs. So reflashing and rewriting of bash scripts was required every time I moved some stuff around. So I was looking for something more elegant.

The new setup looks like:

TL-WR703N with OpenWRT with the arduinos connected, the serial is then forwarded over the network (the wr703n has a small flash) cat /etc/config/socat config socat 'arduino_1' option enable '1' option SocatOptions '-d -d OPEN:/dev/ttyUSB0,b1000000,raw,echo=0,hupcl=1,cs8 tcp-listen:3002,reuseaddr'

A RPi with mqtt running and connects a local file to the networked serial with socat again. socat -d -d -d PTY,link=/dev/ttyUSB0,echo=0,raw,b1000000,perm=0666 TCP:openwrt:3002

And ofcourse has serial2mqtt running with serial2mqtt.armv7l serial2mqtt.json -m 'localhost' (Small weirdness here, if I don't specify -m it still tries to connect to tcp://test.mosquito.org ?) {
"mqtt": {
"connection": "tcp://localhost:1883"
},
"serial": {
"baudrate": 1000000,
"ports": [
"/dev/ttyUSB0"
], "protocol":"jsonArray" } }

The arduino firmware does the following;

It first subscribes to /devices/devicename/config and waits for it to be received.

This is a retained message on the mqtt broker that configures that specific device; the format looks like PIN:TOPIC:TYPE multiple entries separated by |

e.g. 5:kitchen/lights/1/control:R|6:kitchen/buttons/1/control:B|12:arduino_1/debugled/control:R

• Relay: Set pin to ouput, subscribe to topic and wait for commands.
• Button: Set pin to input, debounce it and publish to topic when state changes.

When it is configured it sets up a loopback ping to keep the connection alive.

The arduino code I wrote could probably use a lot of improvement, I wrote it on create.arduino.cc, so I tried to keep the external depencies to a minimum.

This setup has been running fine for a week now, so thank you for having made my life so much easier :)

include

define TASKER_MAX_TASKS 1

include "Tasker.h"

Tasker tasker;

const short LED = 4;

volatile unsigned long lastSeen = 0; unsigned long pingPayload = 0; enum Connection {DISCONNECTED, CONFIGURING, CONNECT, CONNECTING, CONNECTED}; volatile Connection connection = DISCONNECTED;

enum Type {BUTTON = 'B', RELAY = 'R'}; typedef struct record_type { short pin; String topic; Type type;

  bool swState;
  bool swPrevState;
  bool swDebouncedState;
  bool swPrevDebounceState;
  long prevTime;

}; record_type records[8]; unsigned short numberOfRecords = 0;

const unsigned int debounceDelay=100;

String getValue(String data, char separator, int index) { int found = 0; int strIndex[] = { 0, -1 }; int maxIndex = data.length() - 1;

for (int i = 0; i <= maxIndex && found <= index; i++) {
    if (data.charAt(i) == separator || i == maxIndex) {
        found++;
        strIndex[0] = strIndex[1] + 1;
        strIndex[1] = (i == maxIndex) ? i+1 : i;
    }
}
return found > index ? data.substring(strIndex[0], strIndex[1]) : "";

}

class Mqtt { public: static String device; static void publishDevice( String topic, String message, short qos = 0, bool retained = false ) { Serial.print(F("[1,\"devices/"));Serial.print(device);Serial.print(F("/"));Serial.print(topic);Serial.print(F("\", \""));Serial.print(message);Serial.println(F("\"]")); Serial.flush(); } static void subscribeDevice( String topic) { Serial.print(F("[0,\"devices/"));Serial.print(device);Serial.print(F("/"));Serial.print(topic);Serial.println(F("\"]")); Serial.flush(); } static void publish( String topic, String message, short qos = 0, bool retained = false ) { Serial.print(F("[1,\""));Serial.print(topic);Serial.print(F("\", \""));Serial.print(message);Serial.println(F("\"]")); Serial.flush(); } static void subscribe( String topic) { Serial.print(F("[0,\""));Serial.print(topic);Serial.println(F("\"]")); Serial.flush(); } static void ping() { publishDevice(F("ping"), String(pingPayload)); } static void subscribePing() { subscribeDevice(F("ping")); } static void handleLine(String& line) { JSONVar root = JSON.parse(line); onMqttMessage((const char)root[1], (const char)root[2]); } static void handleConfig(String& message) { connection = CONFIGURING; numberOfRecords = 0; int i = 0; String entry = getValue(message, '|', i); while(!entry.equals("")) { lastSeen = millis(); handleEntry(entry); entry = getValue(message, '|', ++i); } connection = CONNECT; } static void handleEntry(String& message) { record_type& record = records[numberOfRecords++]; record.pin = getValue(message, ':', 0).toInt(); record.topic = getValue(message, ':', 1); record.type = (Type)getValue(message, ':', 2).charAt(0);

  if (record.type == RELAY) {
    pinMode(record.pin, OUTPUT);
    Mqtt::subscribe(record.topic);
    Serial.print(F("Relay defined on "));Serial.print(record.pin);Serial.print(F(" "));Serial.println(record.topic);
  } else if (record.type == BUTTON) {
    pinMode(record.pin, INPUT_PULLUP);
    Serial.print(F("Button defined on "));Serial.print(record.pin);Serial.print(F(" "));Serial.println(record.topic);
  }
}
static void onMqttMessage(String topic, String message) {
  String relativeTopic = topic;
  relativeTopic.replace("devices/" + device+"/", "");
  lastSeen = millis();
  if (relativeTopic.equals("config")) {
    handleConfig(message);
  } else if (relativeTopic.equals("ping")) {
    if (pingPayload == message.toInt()) {
      pingPayload = millis();
      Serial.println(F("pings match!"));
      if (connection == CONNECTING) {
        connection = CONNECTED;
      }
    } else {
      Serial.print(F("ping content differs!"));Serial.print(pingPayload);Serial.print(F(": "));Serial.println(message);
    }
  } else {
    bool found = false;
    for (short i = 0; i < numberOfRecords; i++) {
      record_type& record = records[i];
      if (record.type == RELAY && record.topic.equals(topic)) {
        if (record.type == RELAY) {
          digitalWrite(record.pin, !digitalRead(record.pin));
        }
        found = true;
        break;
      }
    }
    if (!found) {
      Serial.print(F(" Mqtt Message arrived "));Serial.print(topic);Serial.print(F(": "));Serial.print(message);
      Serial.println();
      Serial.flush();
    }
  } 
}

};

String Mqtt::device = "arduino_1";

void setup() { Serial.begin(1000000); pinMode(LED, OUTPUT); while (!Serial) { ; }

digitalWrite(LED, HIGH); tasker.setInterval(connectionCheck, 1000); digitalWrite(LED, LOW); }

unsigned int timeout = 5000; void connectionCheck() { unsigned long currentMillis = millis();

if (connection == DISCONNECTED || (currentMillis - lastSeen) > timeout) { connection = CONFIGURING; digitalWrite(LED, LOW); Mqtt::subscribeDevice("config"); lastSeen = currentMillis; timeout = 5000; } else if (connection == CONFIGURING) { digitalWrite(LED, !digitalRead(LED)); Mqtt::ping(); } else if (connection == CONNECT) { Mqtt::subscribePing(); connection = CONNECTING; timeout = 2000; } else if (connection == CONNECTING) { digitalWrite(LED, !digitalRead(LED)); Mqtt::ping(); } else if (connection == CONNECTED) { digitalWrite(LED, HIGH); Mqtt::ping(); } }

String line; void loop() { while (Serial.available()) { char ch = Serial.read(); if ( ch == '\r'){} else if ( ch == '\n' ) { Mqtt::handleLine(line); line = ""; } else line += ch; } tasker.loop(); readButtons(); }

void readButtons(){ for (short i = 0; i < numberOfRecords; i++) { record_type& record = records[i]; if (record.type == BUTTON) { record.swState = digitalRead(record.pin) == HIGH; } } debouncePins(); checkStateChange(); }

void debouncePins(){ unsigned long currentMillis = millis();

for (short i = 0; i < numberOfRecords; i++) { record_type& record = records[i]; if (record.type == BUTTON) { if(record.swState != record.swPrevState){ record.prevTime = currentMillis; } if(currentMillis - record.prevTime > debounceDelay){ record.prevTime = currentMillis; record.swDebouncedState = record.swState; } record.swPrevState = record.swState; } }

}

void checkStateChange(){ for (short i = 0; i < numberOfRecords; i++) { record_type& record = records[i]; if (record.type == BUTTON) { if(record.swPrevDebounceState != record.swDebouncedState){ if(record.swDebouncedState == 1){ Mqtt::publish(record.topic, F("OFF")); } if(record.swDebouncedState == 0){ Mqtt::publish(record.topic, F("ON")); } } record.swPrevDebounceState = record.swDebouncedState; } } }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[sueastside]

@banier1,

If you want to install it on a RPi:

• Download https://github.com/vortex314/serial2mqtt/raw/master/build/serial2mqtt.armv7l.zip
• Unzip the serial2mqtt.armv7l from the Debug directory (this is the serial2mqtt executable)
• Possibly do a chmod +x serial2mqtt.armv7l to make it executable
• In the same directory as the executable create a serial2mqtt.json (with the contents above, make note of the port of your arduino)
• Run with serial2mqtt.armv7l serial2mqtt.json -m 'localhost' (assuming you run your mqtt broker on the RPi as well)

Then ofcourse you need a firmware on your arduino that can talk to serial2mqtt (my code is above) Or you can write your own or look a the other repo https://github.com/vortex314/mqtt2serial which has client code for several devices. Good thing to know is that all serial output of any sketch will be published to the mqtt topic "src/raspberrypi.USB0/serial2mqtt/log" (or something similair, check serial2mqtt log ouput! ) so you could test with any sketch that writes to serial out.

mq2tt has two communication modes jsonObject and jsonArray (my sketch uses jsonArray) So from your arduino sketch you write out json to the serial and mq2tt will see these as 'commands'(publish or subscribe) You could read from serial too if you subscribed to certain topics which will also be json that you will have to parse.

[vortex314]

Thanks for exchanging info and documenting the deployment. 👍

[halfbakery]

@sueastside,

My setup is quite similar, built from four major parts:

• the real HVAC controller is an Arduino Mega
• connected to a TL-MR3220 with OpenWRT forwarding the serial port of the Mega :-)
• to a JSON-RPC server had been written in perl
• and a HTML5/AJAX frontend

I decided to eliminate my aging code, and switch to a future-proof solution. My goal is to

• keep most of my Arduino HVAC controller
• drop my frontend and use Home Assistant
• get rid of my JSON-RPC server and connect the HVAC controller directly to Home Assistant by serial2mqtt :-)

BTW with my latest addition you can eliminate the socat running on your RPi :-)

[PizzaProgram]

@sueastside This project is getting better and better! Thank you very much for sharing all these infos. Keep us up to dated in the future too ;-)

I'm currently learning C language to be able to help / enhance this project by:

• keepalive / re-connect function
• i2c support for Arduino Nano boards. (So it could handle many kinds of sensors and expanded IO ports too.)

[halfbakery]

FYI: How to build as an OpenWRT package