amqp-node / amqplib

AMQP 0-9-1 library and client for Node.JS
https://amqp-node.github.io/amqplib/
Other
3.69k stars 474 forks source link

No support for streams timestamp offset? #722

Closed snird closed 1 year ago

snird commented 1 year ago

The example for streams: https://github.com/amqp-node/amqplib/commit/4993dee433665e9e6fe7729f3eaeacca06b46cb7

says we can specify a timestamp. But by specifying a number for the x-stream-offset header argument, RabbitMQ uses it as the offset number, not timestamp.

I looked at RabbitMQ source code, and their test suite for the case makes it clear: https://github.com/rabbitmq/rabbitmq-server/blob/ddb9fbd12cc8e7c38740d48a226f770d4878dcf9/deps/rabbit/test/rabbit_stream_queue_SUITE.erl#L954

For timestamp, the argument type must be "timestamp". If it is "long", as it seems to be the case with this lib, it uses it as the offset directly, not as timestamp.

Is there a way to make timestamp offsets work?

snird commented 1 year ago

I have tried to use the coded of this lib: https://github.com/amqp-node/amqplib/blob/4993dee433665e9e6fe7729f3eaeacca06b46cb7/lib/codec.js

but so far I was not succesful. I guess that I have to use it, to encode a timestamp correctly.

Can someone shed some light on how to do that?

snird commented 1 year ago

I'm now using https://github.com/coders51/rabbitmq-stream-js-client instead. It required some work as it is still experimental, but mostly merging locally the open PRs was enough. I only consume through the rabbitmq-stream-js-client lib, to publish and create I still do through amqplib.

Just letting people here know in case they try to use streams.

Maybe worth noting in the "streams example" in this repo that timestamp offsets don't work currently.

cressie176 commented 1 year ago

Hi @snird,

Thank you. I did have a look over the weekend but couldn't get the streams example working either. Do you have a public repo with the working version of rabbitmq-stream-js-client? I'd like to see what the x-stream-offset timestamp looks like with Wireshark and compare with amqplib

Thanks

Steve

snird commented 1 year ago

I don't have it public, as it is still on testing phase on my side.

Anyway, that won't help your case: the https://github.com/coders51/rabbitmq-stream-js-client lib uses the binary format of streams in RabbitMQ. That's not amqp 0.9.1 at all, so there are no possible comparisons.

cressie176 commented 1 year ago

OK, thanks anyway

cressie176 commented 1 year ago

You can force a timestamp using the following syntax

x-stream-offset’: { ‘!’: ‘timestamp’, value: 1686519750 }

Where 1686519750 is the number of seconds (as opposed to miliseconds) since 00:00:00 UTC, 1970-01-01

cressie176 commented 1 year ago

I'll leave this issue open until I've added something to the streams example & documentation

cressie176 commented 1 year ago

Done. Thanks for opening this issue @snird, it was interesting.