Timestamps for retained messages

[itspa1]

Feature Request: It would be good to have timestamps in the retained messages being sent since a retained WILL message has no option of sending exact timestamp cause it would be set during the connection.

[ralight]

I'm afraid that I don't think this is appropriate for inclusion. Even as an option, it would be impossible to cater for formatting the message in all the ways that people might want.

[itspa1]

Hey, Thanks for the reply, yes indeed it would be impossible to cater for formatting the message in all the ways but, the basic idea behind the request for a timestamp in the retained messages is it would be better to get the arrival time if it has a timestamp being set either by the publisher and if not, the broker could add a UNIX timestamp to the payload so that it would be easy for the subscriber to make sense if it is a stale/old-retained message or indeed a new one. also, this feature request is not just on the fly but some brokers like flespi provide this functionality. so, was hoping this would be a good addition to the most used open source broker.

[ratara]

@Impeekay Isn't it the idea of retained messages that they contain valid data no matter when they were published? What would that scenario look like where you need the publish time stamp to verify? - If you are looking for something that needs an alive information, an event(not retained message) could be a more direct approach.

[itspa1]

@ratara yes absolutely right, if something that needs alive information, an event might be the best approach but, certain use cases would go a step far and would need timestamps sometimes, or at least in my use case tho. Hence the request so that if at least added optionally, many use cases requiring this might be taken care of. and some brokers "flespi" might have added it for a reason. again as said earlier this would be a good addition to the feature list

[ilgrank]

+1 for this request. I have battery-powered sensors that report temperature only when it changes over a defined threshold. (and publish a retained message when this happens) In this way, I can query the broker and know the value of the last reading. Alas, I don't know when it was published. Most of the times the temperature hasn't changed and so the last reading is still valid, but the battery could be dead and the retained message could be weeks old. No way of knowing. (and these devices do not set a LWT by the way)

[ralight]

The most compliant way to solve this would be to use MQTT v5 user-property properties.

To simulate this you could use:

mosquitto_pub -t topic -m message -r -V 5 -D publish user-property timestamp "yesterday"

[itspa1]

mosquitto_pub -t topic -m message -r -V 5 -D publish user-property timestamp "yesterday"

@ralight here yesterday means adding a new timestamp on our own, or can it be a stream of new timestamps which will be generated on its own?

[ilgrank]

The most compliant way to solve this would be to use MQTT v5 user-property properties.

To simulate this you could use:

mosquitto_pub -t topic -m message -r -V 5 -D publish user-property timestamp "yesterday"

This if you can alter the behaviour of the devices that are publishing. My IoT devices use MQTT v3 and are closed-source, so I can't alter their posts.

[ralight]

@Impeekay That would be a per-message timestamp generated by the publishing client in whatever format is desired for that particular project. This is exactly what the user-property property is for, adding in application specific information.

[ralight]

@ilgrank Would you be happy with a solution that allowed you to attach a user property to messages at the broker, so your subscriber would have to support MQTT v5, but your publishers would go untouched, and no modification of the payload would be required?

[ilgrank]

@ralight I do currently use Paho javascript client (I'm developing a web-app) that according to the docs https://www.eclipse.org/paho/clients/js/ does not support MQTT 5 alas :(

[itspa1]

@ralight thanks, this actually seems like a suitable workaround. But, it would require all the clients that I currently have to be ensured that they send out the user-property of timestamps.

[Sumatt92]

Could you run an application on the same host as the broker that uses a client library to subscribe to the topic and append the timestamp before publishing to a different topic? Like if original topic is /mysensor1/temp , client application subscribes to /mysensor1/temp, appends timestamp to payload or reforms as mqttv5 with user property and publishes to /mysensor1/tempandtime ?

[ilgrank]

@Sumatt92 I've already a workaround in place: for battery powered sensors a client inserts the values (with timestamp) in a database.. but it is a clunky workaround. As it is the one you're suggesting :)

[ilgrank]

@ralight :any hope to have it as an MQTT v3 / 3.1 solution (as PAHO JS doesn't support currently MQTT v5)?

[ckrey]

You may include the timestamp in the payload as CSV:

‘1591979245,25.9’

or JSON:

‘{“tst”:1591979245,”temp”:25.9}'

On 11. Jun 2020, at 18:29, ilgrank notifications@github.com wrote:

@ralight https://github.com/ralight :any hope to have it as an MQTT v3 / 3.1 solution (as PAHO JS doesn't support currently MQTT v5)?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/eclipse/mosquitto/issues/1225#issuecomment-642790703, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6XHGN5P322A377WACXMV3RWEA6NANCNFSM4HE3TEIA.

[ilgrank]

@ckrey : I really don't understand how, since the payload is generated by a closed-source device whose messages I can't alter

[ralight]

@ilgrank I don't really want to have the broker modifying the payload, it's a bit of a minefield for configuration. Probably the best way to do this would be to have a plugin type interface where you could write some code to modify the payload to your requirements. Would that work?

[ilgrank]

@ralight this seems a great idea to me, thanks!

[rcomellas]

+1 for this request Normally our sensors in the field don't have a reference of time (RTC, NTC) so it would be very useful that the server kept the timestamp of the moment it receives a published message

[thg2k]

I also think it is a missing feature in the MQTT spec, adding timestamps in the payload is plainly wrong, also you might not have an accurate clock on the device. The broker should timestamp retained messages, user property would be the way to go until they fix the MQTT spec.

[Sumatt92]

I believe that the payload is and should remain a black box to the broker. Modifying the payload with the broker breaks the network layer model. The client application(s) are the producers and consumers of the payload, while the broker simply routes the packets from a producing client to a consuming client.
This may seem like a snooty response but I think there is a lot of value in maintaining a simple operating model for the broker where its concern is only routing a payload from publisher to subscriber(s).

I also don't feel like using a second client to publish a modified payload is a hack. If you had two temp sensors in a room, you probably wouldn't consider it a hack to have a client application on the broker machine subscribe to both sensor's topics, average the two and publish it back under another topic.

[ilgrank]

@ralight in fact was proposing a plugin to do that. I do agree that it is not the broker's job to do alter a message, but in my scenario, the clients are closed source and do not have (being battery-powered devices) NTP functionality. Adding a subscriber that merely reads those topics and re-publishes them with an added timestamp would effectively double the network traffic and severely decrease the equipment lifetime (Being the broker too battery powered).

To clarify: I have several IoT sensors, and a raspberry zero wich does the 'heavy lift' of collecting and storing the sensor data. Sensors are battery-powered, while the Raspberry is Solar+Battery powered. There are constranints on the equipment size, so I can't use larger batteries or solar panels. Mosquitto was chosen (among other softwares) because of its lightweight operation. The 'plugin' scenario would allow genral users to have a very clean MQTT brocker, while allowing rare case scenarios like mine to have the opportunity to have a solution (please pardon my english, it's not my native language) Thanks

[karlp]

Why would it double the network traffic?!

[ilgrank]

@karlp: because the listening client would have to re-publish every message that is published.

[karlp]

but that doesn't have to go over any network... that can be just looped back at the point that does have time. you don't need to send untimestamped and timestamped over your network. (but this is OT for this ticket now really)

[ilgrank]

..anyway, as you can see this is a much requested feature: in this issue alone there are at least 3 people requesting it. Many more such requests can be found by simple googling, such as: https://www.eclipse.org/lists/mosquitto-dev/msg00903.html

[ralight]

I've just pushed an example plugin which does this to the develop branch, let me know what you think.

You'll need the develop branch version of Mosquitto, and to compile the plugin in plugins/message-timestamp.

[ilgrank]

Thanks @ralight ! sorry, I'm not that an advanced user.. Is there a guide on how to compile the plugin? Thanks!

[ralight]

@ilgrank What system are you compiling on? Linux / Windows / Mac / ?

[ilgrank]

Linux, on a Debian-based distro (dietpi) but so far I've not compiled, but downloaded from the semi-official repo

[ralight]

sudo apt install make gcc git libssl-dev
git clone https://github.com/eclipse/mosquitto mosquitto-with-plugin
cd mosquitto-with-plugin
git checkout devel
make -C src
make -C plugins/message-timestamp

That should give you src/mosquitto and plugins/message-timestamp/mosquitto_timestamp.so. You can copy mosquitto_timestamp.so to /usr/local/lib, and copy src/mosquitto over the top of the existing one at /usr/sbin/mosquitto (this means the existing service control will work the same). Then add to your config file:

plugin /usr/local/lib/mosquitto_timestamp.so

Note that this version of Mosquitto changes the way listeners are configured, so if you don't specify any listeners then the default ones will only be available on the local machine. Coupled with that, the default behaviour for listeners is to deny anonymous access so if you haven't configured some form of authentication then you will find your clients unable to log in. You can of course allow anonymous logins if you need, but it has to be an explicit choice on your part.

[ilgrank]

Hello @ralight , thanks! I banged my head a little bit, there's a typo in your instructions, the branch is (I think) develop while you wrote devel Trying the new binaries now, I'll report my findings! Again, thanks!

[ralight]

My mistake, that should have been develop. Sorry for the confusion.

[ilgrank]

Alas, it seems that GLIBC 2.29 is now required mosquitto: /lib/x86_64-linux-gnu/libm.so.6: version 'GLIBC_2.29' not found (required by mosquitto) while my distro latest version is 2.28 10:

ldd --version
ldd (Debian GLIBC 2.28-10) 2.28

I've asked in my distro (dietpi) support group and I've been told it is not trivial to update GLIBC :/

[ralight]

If you're compiling directly from source, then it should only be using libraries that you have installed and available. The message you have suggests it was compiled against glibc 2.29. I have come across this before when cross compiling and it is possible to force the linker to use a particular glibc version for a symbol, but I think the easiest thing for you to try is compile with:

make clean
make WITH_SYS_TREE=no

That should stop the maths function being used, and hence the error should disappear.

[ilgrank]

@ralight You're right, I cross-compiled on Ubuntu 20 since my distro (DietPi 6.30) compile was erroring out:

make: Entering directory '/temp_mosquitto/mosquitto-with-plugin/src'
cc  -I. -I.. -I../include -I../../include -I../src/deps -I../lib -DWITH_TLS -DWITH_TLS_PSK -DWITH_BRIDGE -DWITH_PERSISTENCE -DWITH_MEMORY_TRACKING -DWITH_SYS_TREE -DWITH_EC -DWITH_CONTROL -DWITH_UNIX_SOCKETS -DWITH_EPOLL -Ideps -Wall -ggdb -O2 -DVERSION="\"2.0-dev0\"" -DWITH_BROKER -c mosquitto.c -o mosquitto.o
mosquitto.c:21:12: fatal error: unistd.h: No such file or directory
 #  include <unistd.h>
            ^~~~~~~~~~
compilation terminated.
make: *** [Makefile:89: mosquitto.o] Error 1
make: Leaving directory '/temp_mosquitto/mosquitto-with-plugin/src'

(yes, there's no unistd.h in this debian-based distro) I've re-compiled with the options you suggested, replacing just my current (1.68) binary with the freshly compiled one.

After several errors (I had to enable Websockets) I still have this problem mosquitto: error while loading shared libraries: libwebsockets.so.15: cannot open shared object file: No such file or directory even if I compiled with: WITH_STATIC_LIBRARIES:=yes I'm missing something? thanks for your kind support

[habazut]

I am testing this as well but where do I see the timestamp then?

1602529417: Loading config file /etc/mosquitto/conf.d/auth.conf
1602529417: Loading config file /etc/mosquitto/conf.d/network.conf
1602529417: Loading config file /etc/mosquitto/conf.d/plugins.conf
1602529417: mosquitto version 2.0-dev0 starting
1602529417: Config loaded from /etc/mosquitto/mosquitto.conf.
1602529417: Loading plugin: /usr/local/lib/mosquitto_timestamp.so
1602529417: Opening ipv4 listen socket on port 1883.
1602529417: mosquitto version 2.0-dev0 running
...
1602530340: New connection from 192.168.0.83 on port 1883.
1602530340: New client connected from 192.168.0.83 as esp8266-6da003 (p2, c1, k15).
1602530340: No will message specified.
1602530340: Sending CONNACK to esp8266-6da003 (0, 0)
1602530350: Received PUBLISH from esp8266-6da003 (d0, q1, r1, m1, 'tingfast45/tankESP/temperature/0', ... (5 bytes))
1602530350: Sending PUBACK to esp8266-6da003 (m1, rc0)

$ /usr/bin/mosquitto_sub -h 192.168.0.17 -t 'tingfast45/tankESP/+/0' -F "%j"
{"tst":1602530664,"topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"23.31"}

Sorry if this is a stupid question, but I am new to MQTT.

Harald.

[ilgrank]

@habazut : isn't {"tst":1602530664 the timestamp you're looking for? :) (in unix time format, that is)

date -d @1602530664
Mon 12 Oct 21:24:24 CEST 2020

[ralight]

@ilgrank I presume you've installed the libwebsockets library on your system. You probably need to update the runtime linker cache, which you would typically do by running ldconfig.

[habazut]

@habazut : isn't {"tst":1602530664 the timestamp you're looking for? :) (in unix time format, that is)

Nope. That's just "now" and not when the message came in. In addition to that, the plugin adds "timestamp" as string and not "tst" as seconds-since-epoch.

   clock_gettime(CLOCK_REALTIME, &ts);
    ti = gmtime(&ts.tv_sec);
    strftime(time_buf, sizeof(time_buf), "%Y-%m-%dT%H:%M:%SZ", ti);

    return mosquitto_property_add_string_pair(&ed->properties, MQTT_PROP_USER_PROPERTY, "timestamp", time_buf);

and btw, I know that this fridgeESP value is from yesterday and the tankESP is from today:

$ date -u +%s
1602535047
$ /usr/bin/mosquitto_sub -h 192.168.0.17 -t 'tingfast45/+/+/0' -F "%j"
{"tst":1602535051,"topic":"tingfast45/fridgeESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"27.50"}
{"tst":1602535051,"topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"23.13"}

So I had hoped that the new value of tankESP would have the timestamp added when it comes in, but nope.

1602535203: Received PUBLISH from esp8266-6da003 (d0, q1, r1, m2, 'tingfast45/tankESP/temperature/1', ... (5 bytes))
1602535229: Sending PUBLISH to mosqsub|13178-habitat (d0, q0, r1, m0, 'tingfast45/tankESP/temperature/0', ... (5 bytes))
{"tst":1602535229,"topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"23.19"}

So the tst is when the mosquitto broker did send it out. But I am looking for the value added by the plugin when the message came in.

Is there some more thing that must be configured than

# cat /etc/mosquitto/conf.d/plugins.conf 
plugin /usr/local/lib/mosquitto_timestamp.so

and I can see that the plugin is loaded from the log?

Thanks, Harald.

[ralight]

@habazut This only works with MQTT v5, because it adds a user-property to the message. You should add -V 5 to the mosquitto_sub command:

mosquitto_sub -t \# -V 5 -F "%j"

This would give you an output like this:

{"tst":"2020-10-12T21:54:21Z","topic":"topic","qos":0,"retain":0,"payloadlen":7,"properties":{"user-properties":{"timestamp":"2020-10-12T20:54:21Z"}},"payload":"message"}

In your own client you would access the properties from the on_message callback.

[habazut]

Thanks, the -V5 did show the extra data where it was present (tank and not fridge, see below).

$ mosquitto_sub -V 5 -h 192.168.0.17 -t 'tingfast45/+/+/0' -F "%j"
{"tst":"2020-10-12T23:29:26.082246Z+0200","topic":"tingfast45/fridgeESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"payload":"27.50"}
{"tst":"2020-10-12T23:29:26.082545Z+0200","topic":"tingfast45/tankESP/temperature/0","qos":0,"retain":1,"payloadlen":5,"properties":{"user-properties":{"timestamp":"2020-10-12T21:10:23Z"}},"payload":"23.25"}

Here one can see as well how the timestamp value != tst value.

Harald.

[ilgrank]

@ilgrank I presume you've installed the libwebsockets library on your system. You probably need to update the runtime linker cache, which you would typically do by running ldconfig.

@ralight I have libwebsockets installed in the system where I compiled mosquitto, but not on the system where mosquitto will run. Having statically linked libraries shouldn't allow for this? (my current mosquitto 1.68 runs websockets without having libwebsockets after all) I also tried

sudo ldconfig
make clean
make -C src

and ldconfig lists libwebsocket. but I get the same error: mosquitto: error while loading shared libraries: libwebsockets.so.15: cannot open shared object file: No such file or directory by compiling with WITH_STATIC_LIBRARIES:=yes shouldn't these libraries be included in the executable?

[ralight]

@ilgrank Sorry I missed that part. I'm afraid that WITH_STATIC_LIBRARIES:=yes just builds the libmosquitto.a static library, nothing more.

You could try just compiling the broker: make -C src

[ilgrank]

@ralight : Thanks, there's one thing I still don't understand: The mosquitto 1.68 I have downloaded from the semi-official repo does come with websockets support, and does so without any external library. What flags/configs I need to compile a mosquitto binary that has the same capabilities? (ie: websockets support without the need of an external library)? Thanks

[ralight]

@ilgrank The packages from Debian direct, and those from repo.mosquitto.org both rely on the libwebsockets package provided by Debian. In other words they do have an external library.

[ralight]

If you only have a static libwebsockets.a file on your system and no libwebsockets.so then I believe the linker will just use the static library without any other changes needed.

[ilgrank]

@ralight : so I really can't explain this:

root@DietPi:/usr/sbin# mosquitto_200
mosquitto_200: error while loading shared libraries: libwebsockets.so.15: cannot open shared object file: No such file or directory
root@DietPi:/usr/sbin# mosquitto_168
1602580715: mosquitto version 1.6.8 starting
1602580715: Using default config.
1602580715: Opening ipv4 listen socket on port 1883.
1602580715: Opening ipv6 listen socket on port 1883.

Update (just seen your second answer): on the building computer I have:

/usr/lib/x86_64-linux-gnu/libwebsockets.a
/usr/lib/x86_64-linux-gnu/libwebsockets.so
/usr/lib/x86_64-linux-gnu/libwebsockets.so.15

please pardon me, I'm really new at building mosquitto, and most likely I'm just missing some basic thing. I've googled since yesterday in order to understand how the binary in the repo was built (what make flags, what config.mk and so on)

[ralight]

Don't worry about it. So both your building device and the dietpi device are x86_64 architecture? The difficulty you're having seems to be the mismatch between what you have on the building device and on the target device.