Wanted: Clean shutdown for deep sleep

[clough42]

I'm working with the 2.0 codebase, and I'm looking for a good way to shut down for deep sleep.

I've tried disconnecting the MQTT broker and then using the event to sleep after that happens, but this causes the LWT message to not be sent, so the retained status message continues to show the device on-line.

I've tried just going into deep sleep abruptly, and this causes the LWT message to be sent, but sometimes the most recent property values are not sent--especially in a slow, low-signal environment.

I think what I really want is a call that will flush all of the messages currently queued, send the offline message and then shut down the connection gracefully, giving me a chance to sleep the processor before it tries to reconnect.

Is there a graceful way to do that today?

Is this something you would entertain having in the 2.0 code?

[benzino77]

Yesterday I have prepared POC of deepSleep (haven't test it yet). But I have same doubts:

1. Is it a good idea to do all the stuff in setupHandler? All I need is to report status of sensor and go to sleep.
2. Is it a good idea to publish node property via mqttClient handler? I need to be sure that the message was published (ACK from broker) before go to deepSleep.
3. Gentle disconnetion from broker will not leave LWT.
4. Will disconnection from broker cause automatic reconnection?

Here is code example:

#include <Homie.h>

const uint8_t DOOR_PIN = D4;
bool sleepFlag = false;
const char* property = "open";
uint16_t packetId = 0;
HomieNode doorNode("door", "status");

void onHomieEvent(HomieEvent event) {
  switch(event) {
    case HomieEvent::MQTT_DISCONNECTED:
      sleepFlag = true; //disconnected from broker, give a signal that I'm ready to deepSleep
      break;
  }
}
void onMqttPublish (uint16_t pId) {
  if (pId == packetId) { //I'm waiting for publish aknowledge for specific packet ID
    //payload is published, gently disconnect from broker
    Homie.getMqttClient().disconnect();
  }
}
void loopHandler() {

}
void setupHandler() { //called once in normal mode
  int doorValue = digitalRead(DOOR_PIN);
  // build topic based on Homie.cpp, nice to have a function returning full topic for a node
  char* topic = new char[strlen(Homie.getConfiguration().mqtt.baseTopic) + strlen(Homie.getConfiguration().deviceId) + 1 + strlen(doorNode.getId()) + 1 + strlen(property) + 1];
  strcpy(topic, Homie.getConfiguration().mqtt.baseTopic);
  strcat(topic, Homie.getConfiguration().deviceId);
  strcat_P(topic, PSTR("/"));
  strcat(topic, doorNode.getId());
  strcat_P(topic, PSTR("/"));
  strcat(topic, property);
  Homie.getMqttClient().onPublish(onMqttPublish); 

  //I need packetId to know my data was published (aknowldged by broker), before deepSleep
  packetId = Homie.getMqttClient().publish(topic, 1, true, doorValue ? "true" : "false"); //QoS = 1, retained = true
  delete[] topic;
}

void setup() {
  Serial.begin(115200);
  pinMode(DOOR_PIN, INPUT_PULLUP);
  Homie.disableLedFeedback();
  Homie_setFirmware("door_sensor", "1.0.0");
  Homie.setSetupFunction(setupHandler);
  Homie.setLoopFunction(loopHandler);
  Homie.onEvent(onHomieEvent);
  doorNode.advertise(property);
  Homie.setup();
}

void loop() {
  Homie.loop();
  if (sleepFlag == true) {//let's deepSleep
    ESP.deepSleep(60 * 1000UL);
  }
}

[benzino77]

Maybe it's a good idea to have one more property for a device:

devices/12345678/$online [true|false]
devices/12345678/$sleep [true|false]

So in deepSleep there will be: $online: false, $sleep:true in other scenario: $online:true, $sleep:false

[benzino77]

Short update. I've added few debugging informations to the sketch and additional flag which indicates that Homie.getMqttClient().disconnect() is called by me (there is always chance that I lost connection to the broker)

#include <Homie.h>

const uint8_t DOOR_PIN = D4;
bool sleepFlag = false;
bool preSleep = false;
const char* property = "open";
uint16_t packetId = 0;
HomieNode doorNode("door", "status");

void onHomieEvent(HomieEvent event) {
  switch(event) {
    case HomieEvent::MQTT_DISCONNECTED:
      if (preSleep == true) {
         sleepFlag = true; //disconnected from broker, give a signal that I'm ready to deepSleep
         preSleep = false;
         Serial.println("Disconnected.");
      }
      break;
  }
}
void onMqttPublish (uint16_t pId) {
  if (pId == packetId) { //I'm waiting for publish aknowledge for specific packet ID
    //payload is published, gently disconnect from broker
    preSleep = true;
    Serial.println("Publish aknowledged.");
    Serial.println("Disconnecting from broker.");
    Homie.getMqttClient().disconnect();
  }
}
void loopHandler() {

}
void setupHandler() { //called once in normal mode
  int doorValue = digitalRead(DOOR_PIN);
  // build topic based on Homie.cpp, nice to have a function returning full topic for a node
  char* topic = new char[strlen(Homie.getConfiguration().mqtt.baseTopic) + strlen(Homie.getConfiguration().deviceId) + 1 + strlen(doorNode.getId()) + 1 + strlen(property) + 1];
  strcpy(topic, Homie.getConfiguration().mqtt.baseTopic);
  strcat(topic, Homie.getConfiguration().deviceId);
  strcat_P(topic, PSTR("/"));
  strcat(topic, doorNode.getId());
  strcat_P(topic, PSTR("/"));
  strcat(topic, property);
  Homie.getMqttClient().onPublish(onMqttPublish); 

  //I need packetId to know my data was published (aknowldged by broker), before deepSleep
  Serial.println("Publishing value.");
  packetId = Homie.getMqttClient().publish(topic, 1, true, doorValue ? "true" : "false"); //QoS = 1, retained = true
  delete[] topic;
}

void setup() {
  Serial.begin(115200);
  pinMode(DOOR_PIN, INPUT_PULLUP);
  Homie.disableLedFeedback();
  Homie_setFirmware("door_sensor", "1.0.0");
  Homie.setSetupFunction(setupHandler);
  Homie.setLoopFunction(loopHandler);
  Homie.onEvent(onHomieEvent);
  doorNode.advertise(property);
  Homie.setup();
}

void loop() {
  if (sleepFlag == true) {//let's deepSleep
    Serial.println("Going sleep.");
    ESP.deepSleep(60 * 1000000UL);
  }
  Homie.loop();
}

The result is:

** Booting into normal mode **
{} Stored configuration:
  • Hardware device ID: 245c0cef
  • Device ID: 245c0cef
  • Boot mode: normal
  • Name: Door sensor
  • Wi-Fi
    â—¦ SSID: myssid
    â—¦ Password not shown
  • MQTT
    â—¦ Host: 192.168.0.100
    â—¦ Port: 1883
    â—¦ Base topic: devices/
    â—¦ Auth? yes
    â—¦ Username: muuser
    â—¦ Password not shown
  • OTA
    â—¦ Enabled? yes
↕ Attempting to connect to Wi-Fi...
âś” Wi-Fi connected
Triggering WIFI_CONNECTED event...
↕ Attempting to connect to MQTT...
Sending initial information...
âś” MQTT ready
Triggering MQTT_CONNECTED event...
Calling setup function...
Publishing value.
Sending Wi-Fi signal quality (100%)...
Sending uptime (4s)...
Publish aknowledged.
Disconnecting from broker.
âś– MQTT disconnected
Triggering MQTT_DISCONNECTED event...
Disconnected.
↕ Attempting to connect to MQTT...
Going sleep.
↕ Attempting to connect to MQTT...
âś– Wi-Fi disconnected
Triggering WIFI_DISCONNECTED event...
↕ Attempting to connect to Wi-Fi...

As you can see just after Homie.getMqttClient().disconnect() Homie is trying to connect to broker again. Then it goes to sleep for 60 sec..

[clough42]

It seems Homie could handle all of this internally with a disconnectMqtt() call. It's in a much better position to handle the details of the mqtt publish and wait for ACK.

It would do the following:

• Set a callback for onMqttPublish() (if it doesn't already have one)
• Send the $online: false message and remember the ID
• When onMqttPublish() is called with the ID of the offline message, disconnect the broker
• My code catches the Homie::MQTT_DISCONNECTED event and goes into deep sleep

One could even imagine it setting a flag to prevent the reconnect, though this probably doesn't matter. It doesn't end up happening--the sleep interrupts it.

Maybe I'll take a shot at implementing it.

I would like @marvinroger to weigh in on whether it's something he wants in homie-esp8266. No sense putting together a pull request if he's planning to do something else.

[euphi]

Not related to "deep sleep", but regarding $online: After startup, Homie should first send $online false, immediately followed by an$online true. This way a listener is informed about a reset, even if the startup is fast enough to not trigger the LWT.

[clough42]

The listener is informed with the repeated $online: true. If there is any doubt, $uptime will also indicate the reset. It's probably not a good idea to be firing out false retained messages just to try to get a state transition.

On Tue, Aug 30, 2016 at 6:32 AM Ian Hubbertz notifications@github.com wrote:

Not related to "deep sleep", but regarding $online: After startup, Homie should first send $online false, immediately following by an$online true. This way a listener is informed about a reset, even if the startup is fast enough to not trigger the LWT.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/marvinroger/homie-esp8266/issues/138#issuecomment-243423528, or mute the thread https://github.com/notifications/unsubscribe-auth/ADK5HOu8b4F8mcuUEeX7uPlmyRwh3l7Lks5qlCLsgaJpZM4JwBAF .

[marvinroger]

@euphi definitely, an $online false followed by a true is a good idea. Oh or yes, as @clough42 said you can simply check if new $uptime < old one... It's a matter of taste, I guess.

@clough42 Regarding LWT: the LWT is sent in the connection packet, it is not sent before disconnecting. So a clean disconnect, just like a connection error, should trigger the LWT on the broker side anyway. I'll take a look.

@benzino77 the setupHandler is a nice place to do so. It's right that the current way of exposing the raw mqttClient is not best, as if you're trying to disconnect, it will try to reconnect anyway, so there must be a built-in method prepareShutdown() or something.

What if send() returns an uint16_t indicating a packetId? Then, a second struct parameter could be added to the event handler, with a new event indicating if a message was acknowledged.

[clough42]

@marvinroger I'm working on changes to add a Homie.disconnect() method that sends the $online:false message, waits for ACK, then shuts down the mqtt client and triggers a new HomieEvent::DISCONNECTED once it disconnects. It also prevents the mqtt reconnect if the disconnect was requested.

I'll put in a pull request once I've done some testing. This seems like a reasonable way to do it, but it's up to you whether you want it in homie-esp8266. If you'd like to do it a different way, I'm happy to make changes.

[marvinroger]

@clough42 don't bother to do that, LWT is done for that, but there's a bug somewhere. No need to send $online false manually.

[clough42]

@marvinroger As I understand it, LWT is only sent on an ungraceful disconnect. On a graceful disconnect, some code somewhere will have to send the $online false message.

The other reason I think Homie-esp8266 needs to have this function is because it's necessary to wait for the last message to be acknowledged before disconnecting the client, and there's no good way to do that externally.

I guess I could just send a message through the raw MQTT client myself (outside of Homie) and wait for it to ACK and then kill the client ungracefully. That would trigger the LWT. It just seems like it would be much better to do it gracefully.

[benzino77]

@clough42 is right. Only ungracefyll disconnect trigger LWT: http://www.hivemq.com/blog/mqtt-essentials-part-9-last-will-and-testament

[benzino77]

@marvinroger maybe you can consider to put this feature prepareShutdown or prepareSleep in future version?

+1

What if send() returns an uint16_t indicating a packetId?

[clough42]

@marvinroger I have a branch on my fork with the graceful disconnect working. Do you want a pull request, even if only for discussion?

[marvinroger]

My bad, indeed, only ungraceful disconnects trigger LWT!

Yes @clough42, please.

[marvinroger]

Oh, looks like you went faster than me! :wink:

[clough42]

I just sumitted #139 for you to look at. If a different name, like prepareForSleep() is better, I'll happily change it. I think that probably makes more sense anyway.

I just want to be sure Homie-esp8266 stays general-purpose. No need to fill it up with application-specific code.

[clough42]

Here's an example of how I would use this:

https://github.com/clough42/homie-eventsensor/blob/master/src/main.cpp

This is an event sensor only. An event (like opening a mailbox or dumping a garbage can) resets the ESP8266. It boots, sends a triggered:1 message and then goes back into deep sleep.

[marvinroger]

For deep sleeping, you need to make sure the messages sent are received from the other side. The current event system implementation does not allow any other data to be sent, other than the type of event itself. Due to the "limitations" of C++, the global event handler will disappear in favor of something like this (really just like ESP8266WiFi):

void onSuccessfulPublish(const HomieEvent::SuccessfulPublish& event) {
  Serial.print("packetId ");
  Serial.print(event.packetId);
  Serial.println(" successfully published");
}

void setup() {
  Homie.onSuccessfulPublish(onSuccessfulPublish);
  // Homie.onWiFiConnected(); and so on
}

Sounds good?

[clough42]

I think that will work fine if I know the packet ID.

Is this instead of prepareForSleep() or in addition to it?

[marvinroger]

Packet ID will be known as an uint16_t returned by the send method.

No, in addition to it. You will want to call prepareForSleep() when you are sure your messages are received.

[benzino77]

How do I know that all "internal" Homie properties ($uptime, $localip, $signal, etc.) are received before I call prepareForSleep()? Do I have to worry about that?

[clough42]

Thanks, @marvinroger. I just merged down trunk to my fork, and it looks good!

[marvinroger]

@benzino77 this is TCP anyway, and prepareForSleep() needs to send some packets to succesfully disconnect, so the previous packets should be sent. In other words, don't worry about that!