Delayed and unreliable delivery

[B3N4K]

Hi,

First of all, thank you for this wonderful plugin! It's just wonderful! The options it provides are really great, thank you especially for the SSL support with custom certificates. Really great job!

However, I have a little problem. I am using a NodeMCU MQTT client to publish messages to a few channels (readings from humidity+temperature+pressure sensor & free heap space). The messages are being pushed to the latest version of Mosquitto MQTT server. I am subscribed to this channel with mosquitto_sub client and all the messages are seen reliably and in a timely manner in this client.

However, the logstash running at the same server as Mosquitto with your plugin used as only input plugin (and outputting to stdout) subscribed to the same topics is very unreliable at receiving the messages. Moreover, the messages received are often out of order and incredibly delayed even by several minutes all the while new messages are published every 5 seconds! This is the case no matter what QOS (in plugin settings) I use.

Here is an example - first the readings received by mosquitto_sub (https://pastebin.com/F4eAczQc) and then readings received by the plugin (https://pastebin.com/pjU5gTrh).

As you can see, the first time the temperature value is above 27 in the plugin's readings is after about 10 minutes from when it was published by the NodeMCU (and then immediately read by mosquitto_sub). This is a huge and unacceptable delay for me. Moreover the plugin is also for some reason reading in 5 second intervals (even delayed by 10 minutes!) and i have cases when it's simply not working pretty much at all - reading only very small subset of messages with incredible delays. For example https://pastebin.com/104aLRNm - very large delays between messages (almost a minute), only one or two of the channels are being received and rest is simply lost, all with huge delay of course. The NodeMCU is still transmitting on all 4 channels (heap, humidity, temperature, pressure) every 5 seconds.

Do you have any idea what could be causing these problems and are there any options to address this? I am thinking there has to be some issue with the plugin since different MQTT client (mosquitto_sub) has no issues reading all the values reliably and in timely manner consistently.

Thank you very much for any help and tips!

Also here is my config:

input {
  mqtt {
    host => "localhost"
    port => REDACTED
    topic => "sensor/sensor_4178452b898bfd19b95e8fae4ab5fd48/#"
    qos => 0
    client_id => "REDACTED"
    username => "REDACTED"
    password => "REDACTED"
    certificate_path => "REDACTED"
    key_path => "REDACTED"
    root_ca_path => "REDACTED"
    logfile => "/var/log/logstash/mqtt.log"
    ssl => true
  }
}

filter {
}

output {
  stdout {}
}

(qos is set to 0 here but again, I tried all the values (0, 1, 2) and nothing changed)