search

nats-io / nats-server

High-Performance server for NATS.io, the cloud and edge native messaging system.
https://nats.io
Apache License 2.0
15.91k stars 1.41k forks source link

Allow setting message timestamp #5310

Open GeorgeGkinis opened 6 months ago

GeorgeGkinis commented 6 months ago

Proposed change

Allow setting message timestamp when pushing to a jetstream.

Use case

From my understanding so far, setting the timestamp of a message being pushed to a jetstream is not currently possible and the message gets its timestamp based on the receive time.

I would like to be able to set the timestamp when pushing to a jetstream, so I could upload historical event data (i.e. from csv using benthos), setting the timestamp based on the original event timestamp and then use replay policy: original.

If I do this now, then replaying happens in the ingestion speed when pushing to the jetstream, not the original speed.

It would be even more awesome if the messages still get ordered in the jetstream based on their timestamp, even if they are pushed out-of-order.

This is possible with influxdb (which is a DB and not a message distributor like nats.)

Contribution

Yes, if qualified.

kfollesdal commented 6 months ago

This would also be nice feature to have if you make your own "tired storage"

Zetanova commented 6 months ago

I tried this now over the header Nats-Time-Stamp but with bad results at least on nats 2.9.20

The server persist the header but still takes its own received timestamp and the c# NATS.Client 1.1.4 does silently throws an parse exception on msg.Time.

I thin the current only workaround is use a custom header for the timestamp.

derekcollison commented 6 months ago

This is not supported at this time but is being considered.

geekox86 commented 2 months ago

This is a use case that we also need to migrate historic messages to NATS

Zetanova commented 2 months ago

My workaround is to add a header ts to a relayed message that has presidency on read.