Unfortunately, only sporadic data from a Shelly Pro3EM to the Borker

[GeThi255]

Hello,

Currently I use a Raspi with Mosquito as a broker to distribute the data from the Ahoy DTU and a Shelly Pro3EM. This also works very well.

Now I want to replace the Raspi with the broker with an ESP32 with PicoMqtt. The Ahoy DTU also connects to the ESP32, but the Shelly Pro3EM doesn't connect very well.

The Shelly Pro3EM publishes a data set about every 3 seconds. With the Raspi there is no problem. With the ESP32, most of them are lost.

In the diagnostics window of the Shelly there is a message with "MQTT0 queue overflow!".

shelly_notification:163 Status change of em:0: 
{"id":0,"a_act_power":4.1,"a_aprt_power":66.2,"a_current":0.291,"a_pf":0.52,"a_voltage":227.9,"b_act_power":177.4,"b_aprt_power":263.5,"b_current":1.153,"b_pf":0.76,"b_voltage":228.6,"c_act_power":93.0,"c_aprt_power":104.1,"c_current":0.455,"c_pf":0.91,"c_voltage":228.9,"n_current":null,"total_act_power":274.478,"total_aprt_power":433.737,"total_current":1.899} 
shos_mqtt_conn.c:857 MQTT0 queue overflow!
shos_mqtt_conn.c:857 MQTT0 queue overflow!

Any ideas?

BR Gerd

[mlesniew]

It's hard to say based on the information you shared, but it appears that Shelly buffers unsent MQTT messages in a queue. It wasn't able to send the messages, so the queue filled up. It attempts to add new messages to the queue, but fails -- that's the queue overflow.

Try the following:

• Maybe your device just can't reach the broker at all
  • Make sure the device tries to connect to the right address.
  • Try to see if PicoMQTT receives and accepts the connection
• Make sure your device uses MQTT 3.1.1
• From what I found on the internet, your device will try to send messages with QoS 1. The PicoMQTT broker only supports QoS 0 messages. Still, things should still work as expected when QoS 1 or QoS 2 messages are published. The broker will reply with the right MQTT messages to comply with the protocol. It will not do the extra processing required for QoS 1 & 2 messages, but that shouldn't impact clients. Perhaps this bit is not working correctly.

[GeThi255]

I did some tests but i can't figure out very much.

The WiFi connection ist good and device is connected. The PicoMQTT received some messages form the Shelly device with "overflow" in the Shelly log console. At the same time the connection to my Ahoy-DTU works fine.

I don't know which MQTT standard the Shelly device uses.

May be you are right with your lasst point but I can't change the QoS in the settings. I tried the "alexCajas EmbeddedMqttBroker" for ESP with the same result. Ahoy-DTU works fine and Shelly had the same issue.

Everything works fine on my Raspi with mosquito.

Than you for your help.

[mlesniew]

If Ahoy-DTU connects correctly, but Shelly can't, then there must be something unusual that Shelly is trying to do or it is not configured correctly. Unfortunately, I don't have a Shelly device to do testing myself.

You can try doing the following:

• Double check the MQTT configuration on the Shelly device. Make sure the broker address and port is correct (should be 1883 unless you configured PicoMQTT explicitly to use a different port). Make sure TLS or SSL is disabled. Don't provide a username or password.
• For testing, flash the ESP32 using this example. Use a serial monitor to see what messages it receives -- they should all be dumped to the serial console.
• Try using the example mentioned above, but with the PICOMQTT_DEBUG_TRACE_FUNCTIONS preprocessor macro defined. This will make PicoMQTT print all function calls and returns. Based on that you can try to analyze what's happening in the broker. This will slow down the sketch significantly, but is useful for debugging.
• Configure both devices to connect to Mosquitto on your PC or on a Raspberry. Use Wireshark to capture TCP traffic going to the Mosquitto broker. Try to see if there are any differences in the way the devices connect. This way you can also find out what MQTT version Shelly tries to use.

[mlesniew]

@GeThi255 any luck?

[GeThi255]

Hello,

sorry for my late answer. I was in vacation and afterwards I had a lot to do on my job.

I've tried a few things but not really much. So I had no progress.

I checked the first point on your list, without any changes. I did the second point without receiving any messages. Same setup with my raspi works. No success on the esp3.

That's all i did by now. I'll try some additional test in the next days.

Thank you very much

BR Gerd

[GeThi255]

Hello,

I did some changes yesterday and may be I found a reason.

First I did a firmware update on my Shelly device (Version1.0.3). But this did not help

Afterwards I changed the Shelly standard client ID (shellypro3em-c8f09e87ef1c) to something shorter like x12345.

Now it seems to work. Could it be a issue with the length of the client ID?

BR Gerd

[mlesniew]

You've hit the nail on the head!

PicoMQTT follows the MQTT 3.1.1 standard, which specifies that client IDs should be between 1 and 23 characters long and can only contain letters and digits. Your client ID "shellypro3em-c8f09e87ef1c" is just a tad too long at 25 characters and includes a dash. (see http://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html#_Toc385349242).

At the same time the standard states that MQTT broker implementations have the flexibility to relax these restrictions. They may accept longer client IDs or different characters, which they often do.

In my PicoMQTT library, I tried to play by the standard's rules. If the client ID goes over 23 characters, it's rejected and the connection is closed.

Technically, this isn't a bug because it's in line with MQTT's standard. However, as your experience shows, strict adherence to the standard is impractical. I have now increased the maximum allowed client ID length to 64 chars.

Version 0.3.8 will have this change included and should be available for installation soon.

Thanks for your feedback and for diving deep into this!