search

home-assistant / addons

:heavy_plus_sign: Docker add-ons for Home Assistant
https://home-assistant.io/hassio/
Apache License 2.0
1.51k stars 1.47k forks source link

Mqtt official broker problems with ESP sending messages ? #1747

Closed zambujal closed 3 years ago

zambujal commented 3 years ago

The problem

From maybe 2018, since I start using hassio, many/many sd card ago... :D I had allways some collapse ... and consequence sd card getting corrupted... sometime it survived 3-4 weeks... not good enough. The card get corrupted, because at the distance, I make an off/on with an switch remote from tuya... normally nothing happens, all went well... sometimes corruption... Several times I thought I was getting hacked or something... but this happens even without connection with the outside... none whatsoever... mosquitto getting of the line, not doing anything ... not doing the work of broker... that is. ## Environment

I use the ESP 8266 coded with the normal libraries to connect to the mqtt broker of hassio the mosquitto one... to send the data from my arduino (connected via serial with an arduino)... and zigbee2mqtt on top, I had many weeks of good service... though... this mqtt broker never was stable enough. several times it survived weeks... sometimes just hours.

Then, just a few days ago, Again...I even had to the change the sdd card.... again... and said... enough... 1 terabit hard disc to the raspberry with a nicely 5 amp power source... with an usb extension charged from outside...

The MQTT disconnects from everything, don't restart (zigbee doesn't do anything anymore it needs the broker)... but the hassio enviroment, works nicely.

could be the ESP connected though the mqtt that works bad? (ite sends some weird stuff to the broker that makes him... sleep?!?) inconsistent? not every time, but sometimes... it's weird that it's only me?

I never was able to put more than 30 zigbee sensors (I have a lot more mind you not only zigbee or mqtt... the system after 2-3 weeks get corrupted, and I had to do an fresh resinstall... I was lasy to make an copy of this... my system never progressed like I wanted. But now it's a lot more "consistent" (with the failures)... I even was able to replicate this with an i386 machine with hassio (virtual machine with 8 gigas of ram ...more than the riny raspberry)

Problem-relevant configuration

When I put just the mosquitto in there with receving both zigbee2mqtt and the esp messages ( I don't believe it's the zigbee2mqtt... but my ESP... ```yaml # Configure a default setup of Home Assistant (frontend, api, etc) default_config: # Text to speech tts: - platform: google_translate group: !include groups.yaml automation: !include automations.yaml script: !include scripts.yaml scene: !include scenes.yaml http: server_port: xxxxx octoprint: host: xxxxxx api_key: xxxxx mqtt: discovery: true discovery_prefix: homeassistant sensor: - platform: mqtt name: "Temperature sump" state_topic: "sensor/Temperatura Sump" unit_of_measurement: '°C' - platform: mqtt name: "Humidade sump" state_topic: "sensor/Humidade sump" unit_of_measurement: '%' icon: mdi:water-percent - platform: mqtt name: "Temperatura água" state_topic: "sensor/Temperatura água" unit_of_measurement: '°C' icon: mdi:coolant-temperature - platform: mqtt name: "Condutividade água osmose" state_topic: "sensor/tdsValue" unit_of_measurement: '°' icon: mdi:water - platform: mqtt name: "Estado" state_topic: "sensor/estado" - platform: mqtt name: "Potência Violeta" state_topic: "sensor/Potência luz" unit_of_measurement: '%' - platform: mqtt name: "Potência Branco" state_topic: "sensor/Potência luz2" unit_of_measurement: '%' - platform: mqtt name: "Potência Azul" state_topic: "sensor/Potência luz3" unit_of_measurement: '%' - platform: mqtt name: "Horas" state_topic: "sensor/Horas" - platform: mqtt name: "Minutos" state_topic: "sensor/Minutos" - platform: mqtt name: "Segundos" state_topic: "sensor/Segundos" - platform: mqtt name: "Bomba de reposição" state_topic: "sensor/Bomba de reposição" - platform: mqtt name: "Frasco escumador" state_topic: "sensor/frasco escumador" - platform: mqtt name: "pH água" state_topic: "sensor/pH água" unit_of_measurement: 'º' icon: mdi:test-tube - platform: enphase_envoy name: enphase_new # username: xxxxxx # password: xxxx ip_address: xxxxxxx scan_interval: 30 monitored_conditions: - production - consumption - daily_production - seven_days_production - lifetime_production - inverters # Calculate Remaining Power - platform: template sensors: remaining_power: value_template: > {{ '%0.1f' | format(states('sensor.enphase_newenvoy_current_energy_consumption') | float - states('sensor.enphase_newenvoy_current_energy_production') | float) }} unit_of_measurement: 'W' friendly_name: Energia importada switch: - platform: mqtt name: "Tasmota" state_topic: "stat/tasmota/RESULT" value_template: "{{ value_json.POWER }}" command_topic: "cmnd/tasmota/POWER" payload_on: "ON" payload_off: "OFF" availability_topic: "tele/tasmota/LWT" payload_available: "Online" payload_not_available: "Offline" qos: 1 retain: false ``` ## Traceback/Error logs

Additional information

And then I start using the mqtt that you are going to remove from the add-on ... the mqtt server and web client addon... everything FINE ok, just after 2-3 days... (fingers crossed) but after I gained courage to put the mosquitto... all went... bad again! So it's the mosquito, and if I am one of the few, I bet something from the ESP, that this mosquito doesnt't like... I could put also the code of the esp for you to see... this is really weird... why the other mqtt server works nicely... I don't really now...

I have an similar post that Frenk closed, but I was on wrong assumptions... now I tested it a lot more... My idea is that with this "old" mqtt broker, I will never be bad... if you erased this... I might have to go to the non hassio enviroment... to get an mqtt that works...

zambujal commented 3 years ago

here what my system is doing now:

https://www.youtube.com/watch?fbclid=IwAR1LRrVsBofgW1vXMuoFFY9Ijq0awAGf65THqaaH-lQxjU5nvAJSm1_zlVM&v=jV822y7u2x0&feature=youtu.be

After an fresh install with nothing but 1 zigbee sensor and the arduino stuff (the code on yaml I put above) I get the same collapse... even in i386.

Now I can add... and Add I will... even going to change my texas sinffer cc2531 for one more powerfull ...

The post I was talking about that I made with wrong assumptions... https://github.com/home-assistant/addons/issues/1740

zambujal commented 3 years ago

Maybe it's relevant I will post my esp code (arduino based) here... it's an work in progress... and I am really an amateur...

many copied code... from here and there...

I am now trying to get mqtt messages from it... so half baked code: **_#include

include

include

include

// change next line to use with another board/shield

//const char ssid = "xxxxx"; //const char password = "xxxxxx";

const char ssid = "xxxx"; const char password = "xxxxxx";

const char mqttServer = "xxxxx"; const int mqttPort = xxxx; const char mqttUser = "xxxxx"; const char* mqttPassword = "xxxxx";

unsigned long lastTick = 0; unsigned long clic = 0; int refresh = 100000;

WiFiUDP ntpUDP;

// By default 'pool.ntp.org' is used with 60 seconds update interval and // no offset //NTPClient timeClient(ntpUDP); // You can specify the time server pool and the offset, (in seconds) // additionaly you can specify the update interval (in milliseconds). //

NTPClient timeClient(ntpUDP, "europe.pool.ntp.org", 7200, 60000);

// Example 5 - Receive with start- and end-markers combined with parsing

const byte numChars = 64; char receivedChars[numChars]; char tempChars[numChars]; // temporary array for use when parsing

// variables to hold the parsed data int state; int hour; int minute; int second; int humido; float paridade; float floatFromtemp; float floatFromhumidity; float floatFromtempEau; float floatFrompH ; float tdsValue; int intFromlight; int intFromlight2; int intFromlight3; boolean newData = false; const char producao = "1"; WiFiClient espClient; PubSubClient client(espClient); void callback(char topic, byte* payload, unsigned int length) {

Serial.print("Message arrived in topic: "); Serial.println(topic);

Serial.print("Message:"); for (int i = 0; i < length; i++) { producao=((const char* )payload[i]); }

Serial.println(); Serial.println("-----------------------");

} //============

void setup() {

Serial.begin(4800); WiFi.begin(ssid, password);

while (WiFi.status() != WL_CONNECTED) { delay(500); //Serial.print("Connecting to WiFi.."); }

//Serial.println("Connected to the WiFi network");

client.setServer(mqttServer, mqttPort); client.setCallback(callback);

while (!client.connected()) { // Serial.println("Connecting to MQTT...");

if (client.connect("ESP8266Client", mqttUser, mqttPassword )) {

  //   Serial.println("connected");

} else {

  //      Serial.print("failed with state ");
  //     Serial.print(client.state());
  delay(2000);

}

}

// Serial.println("This demo expects 6 pieces of data - integer and a floating point value"); \ // Serial.println();

timeClient.begin(); }

//============

void loop() {

reconnect(); recvWithStartEndMarkers(); if (newData == true) { strcpy(tempChars, receivedChars); // this temporary copy is necessary to protect the original data // because strtok() used in parseData() replaces the commas with \0 parseData(); if ((paridade,1) == ((state+floatFromtemp + floatFromhumidity + tdsValue+floatFromtempEau+intFromlight+intFromlight2+intFromlight3+hour+minute+second+humido),1)) {

  showParsedData();
}
newData = false;

}

ntp(); client.subscribe("sensor/producao");

}

//============

void reconnect() { if (WiFi.status() != WL_CONNECTED) { WiFi.begin(ssid, password); while (WiFi.status() != WL_CONNECTED) {

  //Serial.print("Connecting to WiFi..");
}

}

if (!client.connected()) {

client.setServer(mqttServer, mqttPort);

while (!client.connected()) {
  //  Serial.println("Connecting to MQTT...");

  if (client.connect("ESP8266Client", mqttUser, mqttPassword )) {

    //     Serial.println("connected");

  } else {

    //      Serial.print("failed with state ");
    //      Serial.print(client.state());

  }
}

} } void recvWithStartEndMarkers() { static boolean recvInProgress = false; static byte ndx = 0; char startMarker = '<'; char endMarker = '>'; char rc;

while (Serial.available() > 0 && newData == false) { rc = Serial.read();

if (recvInProgress == true) {
  if (rc != endMarker) {
    receivedChars[ndx] = rc;
    ndx++;
    if (ndx >= numChars) {
      ndx = numChars - 1;
    }
  }
  else {
    receivedChars[ndx] = '\0'; // terminate the string
    recvInProgress = false;
    ndx = 0;
    newData = true;
  }
}

else if (rc == startMarker) {
  recvInProgress = true;
}

} }

void ntp() { clic = millis(); if (clic - lastTick > refresh) { lastTick = clic; //duração do segundo

timeClient.update();
Serial.print("<");
Serial.print(timeClient.getHours());
Serial.print(",");
Serial.print(timeClient.getMinutes());
Serial.print(",");
Serial.print(timeClient.getSeconds());
Serial.print(",");
Serial.print(producao);

Serial.print(">");

} }

void parseData() { // split the data into its parts

char * strtokIndx; // this is used by strtok() as an index

strtokIndx = strtok(tempChars, ","); // get the first part - the string state = atoi(strtokIndx); // copy it to messageFromPC

strtokIndx = strtok(NULL, ","); // get the first part - the string floatFromtemp = atof(strtokIndx);// copy it to messageFromPC

strtokIndx = strtok(NULL, ","); // this continues where the previous call left off floatFromhumidity = atof(strtokIndx); // convert this part to an float

strtokIndx = strtok(NULL, ","); floatFromtempEau = atof(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ","); floatFrompH = atof(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ","); intFromlight = atoi(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ","); intFromlight2 = atoi(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ","); intFromlight3 = atoi(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ",");

tdsValue = atof(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ","); hour = atoi(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ",");

minute = atoi(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ","); second = atoi(strtokIndx); // convert this part to a float

strtokIndx = strtok(NULL, ","); humido = atoi(strtokIndx); // convert this part to a float strtokIndx = strtok(NULL, ","); paridade = atoi(strtokIndx); // convert this part to a float- }

//============ void receiveMqtt() {

} void showParsedData() { if (state == 0) { client.publish("sensor/estado", "Ciclo", true); }

if (state == 1) { client.publish("sensor/estado", "Blue", true); } if (state == 2) { client.publish("sensor/estado", "White", true); } if (state == 3) { client.publish("sensor/estado", "Violeta", true); } client.publish("sensor/Temperatura Sump", String(floatFromtemp,1).c_str(), true); client.publish("sensor/Humidade sump", String(floatFromhumidity,0).c_str(), true); client.publish("sensor/Temperatura água", String(floatFromtempEau,1).c_str(), true); client.publish("sensor/pH água", String(floatFrompH).c_str(), true);

if (state == 0) { client.publish("sensor/Potência luz", String(intFromlight).c_str(), true); client.publish("sensor/Potência luz2", String(intFromlight2).c_str(), true); client.publish("sensor/Potência luz3", String(intFromlight3).c_str(), true); } if (state == 3) { client.publish("sensor/Potência luz", String(intFromlight).c_str(), true); client.publish("sensor/Potência luz2", "0", true); client.publish("sensor/Potência luz3", "0", true); }

if (state == 2) { client.publish("sensor/Potência luz", "0", true); client.publish("sensor/Potência luz2", String(intFromlight2).c_str(), true); client.publish("sensor/Potência luz3", "0", true);

} if (state == 1) { client.publish("sensor/Potência luz", "0", true); client.publish("sensor/Potência luz2", "0", true); client.publish("sensor/Potência luz3", String(intFromlight3).c_str(), true); }

client.publish("sensor/tdsValue", String(tdsValue,0).c_str(), true);

client.publish("sensor/Horas", String(hour).c_str(), true); client.publish("sensor/Minutos", String(minute).c_str(), true); client.publish("sensor/Segundos", String(second).c_str(), true); if (humido == 0) { client.publish("sensor/Bomba de reposição", "Seca", true); client.publish("sensor/frasco escumador", "vazio", true); } if (humido == 1) { client.publish("sensor/Bomba de reposição", "Submersa", true); client.publish("sensor/frasco escumador", "vazio", true); } if (humido == 3) { client.publish("sensor/Bomba de reposição", "Seca", true); client.publish("sensor/frasco escumador", "cheio", true); } if (humido == 4) { client.publish("sensor/Bomba de reposição", "Submersa", true); client.publish("sensor/frasco escumador", "cheio", true); }

}_**

zambujal commented 3 years ago

I believe this is the same problem as I have: https://github.com/home-assistant/addons/issues/1329

zambujal commented 3 years ago

I updated the libraries for compiling the esp, and also the publish mqtt libraries... hope now it works fine... testing... the biggest problem is it works... then... caput... after disconnecting the client nothing can connects...

zambujal commented 3 years ago

still happens... (not with the other mqtt broker), just the mosquitto one. too much communication? Mosquitto should handle each second new messages... I can drop 95% of the meassage, just better code... maybe it's this... so the other mosquitto is better at handling more "correspondence"? I do not know if this is the problem mind you... could be... compiling the code... only sending info each minute... and after something important happens .

zambujal commented 3 years ago

too much instability...disconnects from mqtt then nothing... I made an debian Virtual machine install (on my PC)... and went for it... everything fine... I made the same thing configured as in the HASSIO work great. so, it's something else. What doesn't work is hassio in the virtual machine... well it works like in raspberry.

I will make an rasperry debian fresh install... My guess is that it will work fine. so ... I really don't understand.

zambujal commented 3 years ago

Never mind, I actually got it working nicely within an VM in windows... and then I bought an intel nuc.,.. time to move on from raspberry. 👍 Maybe this is just too much for the raspberry... .D

zambujal commented 3 years ago

Using ubuntu 20.10 and docker with homeassistant... everything nice...

zambujal commented 3 years ago

I get it right... homeassistant stable with mqtt mosquitto official, nuc or raspberry... or it was due to some update? it was the database ?!? the normal database?

even ubuntu 20.10 had the same problem... and it was not official...it just took more time... with mariadb I don't get the stall that I used to. so after weeks everything is working fine... I click on history and it doesn't take 1 hour to show the history it's instantaneous...

I even had the same problem in nuc, in virtual machine i386, and raspberry.... could be just too much stuff for the "normal" database. using mariadb is like ... from hell to heaven! the machines (even the nuc) get into the knees with the normal database? slowing down, producing the problem with the mosquito.

Homeassistant should have mariadb as standard!

zambujal commented 3 years ago

this problem got me into buying an nuc for homeassistant... and it was not the raspberry fault for sure... :D I am happy now, so eveything is working as it should, with mariaDB ...

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.