Closed ezar closed 5 years ago
Ok, so Publish failed!
usually indicates that the internal TCP buffer is filling up and so MQTT messages cannot be published until the previous messages have been dealt with. Due to that reason the error message is also quite brief because it increases the chances of it still fitting into the buffer, but also by sacrificing context.
Ok, so for fixing this warning I would recommend decreasing the log level so that fewer MQTT messages are published and thus the TCP buffer doesn't fill up as quick.
Hello @OttoWinter,
I have the same problem as described in the first posts, but for me it is not warning, because the mqtt messages are never reaching my server.
I set the loglevel to NONE
, but not only one mqtt message has ever reached my mqtt server, but if I set the loglevel to VERBOSE
it is working like a charm, which is the opposite of that what you mentioned.
@tringler Well that would be because if you set the log level to NONE
no log messages will be produced. So you will never see log messages. However, when the log level is NONE
all other (more important) messages like sensors announcing their state have a higher probability of being transmitted correctly.
Maybe I understand it wrongly, but just to make sure:
Loglevel NONE
: No mqtt messages are received by my mqtt server
Loglevel VERBOSE
: All mqtt messages are received by my mqtt server
This is expected? I understand the issue as it should exactly the opposite.
Please specify what you mean by "no mqtt messages" and "all mqtt messages".
What I meant was that if you set the log level to NONE
, you shouldn't see any MQTT messages on the topic <NODE_NAME>/debug
, as no log messages are ever produced at that log level. However, all other MQTT messages (for example sensors publishing their state on <NODE_NAME>/sensor/.../state
), should be sent correctly. Are really no MQTT messages (even sensor state messages) sent with NONE
log level?
(If really no messages whatsoever are sent, then that could be because of the MQTT server aborting the connection due to inactivity, the keepalive
parameter for MQTT might help in that case)
Are really no MQTT messages (even sensor state messages) sent with
NONE
log level?
Indeed.
I'm not talking about the debug output. At that moment I set the loglevel to NONE
I see the Published failed!
messages in the debug log and no messages are transferred to my mqtt server. I see the HomeAssistant state as unavailable
then.
At that moment I set the loglevel to VERBOSE
I see no Published failed
anymore and get the right state in HomeAssistant.
Ok then, so it is really the NONE
log level. Unfortunately, after looking at the code again, I can't figure out what it could be. In fact, I just tested it myself using the NONE log level and it is working as expected.
At that moment I set the loglevel to NONE I see the Published failed! messages in the debug log
This is weird, when you set the log level to NONE
you shouldn't see publish failed
at all, as publish failed
is a log message and is not transmitted with log level NONE.
You're right. Let me reproduce this again in the next days. It is a few days ago and maybe a bit wrong in my mind. I will provide a list of the behaviour of the different log levels on my mqtt server and what I get from the debug esphomelib logging.
So I'm seeing many "Publish failed" too. Lots of logs here: https://gist.github.com/thubot/d770f6a14676de5f8e51d2e2228f577b This is my yaml: https://gist.github.com/thubot/ea5c523f0ce3393a8db7c3320aef8e8e
The problem is persistent identically over reboots of the same flash, but differs per esphomeyaml run of same config (sometimes the adc config message fails, sometimes only the temperature config message, sometimes both, notably both long HA discovery messages)
In my test runs, the problem did only occur on the INFO DEBUG log level and above, not on VERBOSE or VERY_VERBOSE. I'm not sure if this is significant or just a coincidence.
The only part of the config messages that changes over reruns is the version string, so that could be the only thing that might change the length of the message, but it seems to be constant length.
Does setting VERBOSE or VERY_VERBOSE raise the buffer size somehow?
I have the same problem....I only have 2 sensors with 1 message/sec each, so I don't think It's buffer related. And in my case, the Publish failed! msg comes after MQTT Connected, so I suppose the buffer doesn't even have time to fill up. In the meantime, I added the sensors manually in configuration.yaml to view its info, but the node doesn't read mqtt messages (eg, the one I did to no enter in deep sleep). Can I help with some test or info?
@thubot The severity level does not change the allocated buffer size. It does however send out more messages which I guess could end up improving the probability that more important messages come through.
@HFsi I recently did some improvements for 1.10.0 concerning this issue. In 1.10.0, the messages will be auto-republished at a later point if something goes wrong. You can test the new system using (might not be 100% stable, but please do report bugs)
esphomeyaml:
esphomelib_version:
branch: custom-sensor-platform
I think I found my problem....If I define:
mqtt: birth_message: will_message:
For retain the messages, the node doesn´t update the sensors anymore. I´m on 1.9.3, and without birth and will, works fine.
I get the same error on 1 sensor on 1 board. If I set logger to very_verbose I still get the publish failed error but the missing sensor shows up in HA.
@HFsi I recently did some improvements for 1.10.0 concerning this issue. In 1.10.0, the messages will be auto-republished at a later point if something goes wrong. You can test the new system using (might not be 100% stable, but please do report bugs)
esphomeyaml: esphomelib_version: branch: custom-sensor-platform
I think the main problem is that the device tries to send a lot of messages right after boot especially if you use home assistant discovery. This might cause the problem of filling the TCP buffer. The biggest problem however is that the messages that fail most often are the discovery messages and when they are failing Home Assistant wont even show the device in the GUI and also won't listen to any following messages.
Sadly thg custom-sensor-platform
branch doesn't seem to exist anymore. I tried the dev
branch which prints the message "[...] will retry later.." - but how much later exactly? I did wait 10+ minutes or so and didn't see any retry message in the (DEBUG) log.
@markusressel Yes that branch has been merged. will retry later
is good, unless you set the log level to verbose you won't see when the message is actually published successfully. But if it would fail repeatedly, you would see a lot of continuous will retry later
messages.
I'm also seeing the "Publish failed!", mainly on the discovery data. Spent a lot of time thinking it was something with the broker, swapping it our and changing stuff.
Biggest issue with this, like posted above, is that it's the publish of the discovery data that was failing. The status binary_sensor got through, but the light didn't. It's probably not random what messages fail, or rather which message in the queue fails.
After a short night with too little sleep: commenting status binary_sensor, status led and web server, discovery data got through.
I don't know if it's possible to correlate the "Publish failed!" and the message itself? Knowing that all logs are also going over mqtt, it might only add more bytes to the TCP buffer.
When the TCP buffer gets full, it's because of too many messages, but what is the underlying issue here?
Anyway, thanks again @OttoWinter for all your hard work. Looking forward to improvements.
[07:52:58][I][mqtt.client:115]: MQTT Connected!
[07:52:59][V][mqtt.client:239]: Publish(topic='wemos_bandit_scrappy/status' payload='online' retain=1)
[07:52:59][V][mqtt.component:067]: 'bandit_scrappy status': Sending discovery...
[07:52:59][V][mqtt.client:239]: Publish(topic='homeassistant/binary_sensor/wemos_bandit_scrappy/bandit_scrappy_status/config' payload='{"device_class":"connectivity","payload_on":"online","payload_off":"offline","name":"bandit_scrappy status","state_topic":"wemos_bandit_scrappy/status","unique_id":"ESPbinary_sensorbandit_scrappy_status","device":{"identifiers":"84f3eb9fc081","name":"wemos_bandit_scrappy","sw_version":"esphomelib v1.9.3 Dec 22 2018, 07:49:35","model":"PLATFORMIO_D1_MINI","manufacturer":"espres
[07:52:59][W][mqtt.client:249]: Publish failed!
[07:52:59][V][mqtt.client:214]: Subscribing to topic='wemos_bandit_scrappy/light/bandit_scrappy_one/command' qos=0 with JSON...
[07:52:59][W][mqtt.client:249]: Publish failed!
[07:52:59][W][mqtt.client:249]: Publish failed!
[07:52:59][V][mqtt.client:239]: Publish(topic='wemos_bandit_scrappy/light/bandit_scrappy_one/state' payload='{"effect":"None","state":"ON","brightness":255,"color":{"r":255,"g":36,"b":36}}' retain=1)
@DavidDeSloovere Try with latest dev branch - that has fixed MQTT discovery publishing.
The issue is device-related. The ESP has a fixed-size TCP buffer (can be changed by some platformio commands I think). Once it's full, any data that's written in there will be lost - that's why you see publish failed.
Anyway, most of this will be solved anyway with the next release. The new native HA<->esphome API is much more efficient with data size.
I'm now running version 1.10.1 with the native api via Home Assistant and so far it works great. Apart from some setup issues (installing the new firmware and hooking it up to home assistant) I had no issues whatsoever getting it to run on first try.
So: damn great work there @OttoWinter!
Fails many times to publish :(