Closed aricart closed 1 year ago
The value is not json encoded either, nor does it do any API pass through or accounting. So much more efficient.
Feature landed in the server!
While I opted not to add direct get as a feature in the CLI for getting messages I did add --allow-direct as a option when adding streams, but since its so niche its not something being prompted for.
Will report such a stream in s info though
ok for now, but once settled this will be the way for KV etc and possible individual message retrieval directly from a stream.
Note that subject has changed to $JS.API.DIRECT.GET.${stream}
Note that there's a next_by_subj
option
@derekcollison @aricart From a client library perspective, should we return a "raw" NATS message with the incoming headers, or should we make it a "JS" message where users can access Sequence, Time, etc..?
If the latter, are the headers copied over as-is (including all the Nats-xxx
ones)? I am guessing that the original message could have its own headers, so we can't simply strip the whole headers section...
Are we re-using the existing schema response, stream_msg_get_response
, the same as the current get messages? That message object could be expanded to have a reply-to field which is where we pull the js meta data, or just have a js-meta data field.
These are delivered on INBOX (or reply) and are friendly to Request Muxing. They have headers that denote stream, subject, sequence and timestamp.
These are raw messages, not JSON encoded.
These also participate in a queue group so all members, optionally including mirrors, can participate.
@derekcollison But regarding my question: should the library return them as-is (that is a raw NATS message) or as a JetStream message? For instance in Go the normal GetMsg() returns a RawStreamMsg
that looks like this:
type RawStreamMsg struct {
Subject string
Sequence uint64
Header Header
Data []byte
Time time.Time
}
As I was asking if we return as a RawStreamMsg, do we need to suppress the Nats-xxx
headers that were return by the "direct get" API response?
I think JetStream message. Apologies for the confusion. But these can not be ack'd and are not part of a consumer of course.
Likely they shouldn't be a JetStream message as the API shouldn't introduce ambiguity on how it was obtained. If the client provides a different "type" of msg, it should do so if it can. It is possible that in the current Go client these are the same as the Go client doesn't differentiate.
But it is a JetStream message, you got it out of a stream. I think client libs should present a consistent view upwards regardless of low level mechanism to retrieve, e.g. consumer vs stream get vs direct stream get.
But it is a JetStream message, you got it out of a stream
I would agree here. Of course, it can't be ack'ed (since it was not received through a subscription), but it is still a JetStream message.
in the case of Go, external functions like ack,etc are not bound to the type. In other languages they are, so typing it as a "JetStreamMsg" would also imply functions that are not applicable, thus the comment that languages should do what is appropriate.
when doing a last_by_subject
you can now simply append this subject to the request for the message without any payload. This allows things like KV to be clamped for permissions on GET when using the Direct API. See https://github.com/nats-io/nats.deno/pull/341
Note that the original description of this issue omittedmirror_direct
configuration property on the stream, which should also be available in order to enable mirror to use the direct get api
Right now it gets set to true if the origin stream has AllowDirect, just FYI.
@aricart We should update the description that the server does NOT set allow_direct to true when max_msgs_per_subject is > 0, this change in the server has been reverted here: https://github.com/nats-io/nats-server/pull/3441
Yes. I marked that on @bruth release worksheet
Removed initial comment where the initial server behaviour of setting allow_direct
was set to true by the server by default
Stream docs were updated to document the AllowDirect
and MirrorDirect
options as of https://github.com/nats-io/nats.docs/pull/489.
Overview
Retrieve messages from a stream in a more efficient way than the get message API (
$JS.API.STREAM.MSG.GET.<stream>
)allow_direct
allow_direct
andmirror_direct
Direct Get Api
When the property is enabled, clients can lookup messages using the direct JetStream API:
$JS.API.DIRECT.GET.<stream>
with a payload the same as the get message API:A regular message is returned but will not contain the JS meta data (the reply-to will be empty).
Examples
{last_by_subj: string}
- get the last message having the subject{seq: number}
- get a message by sequence{next_by_subj: string}
- get the first message (lowest seq) having the specified subject{seq: number, next_by_subj: string}
- get the first message with a seq >= to the input seq that has the specified subjectThere is a shortcut form for
last_by_subj
$JS.API.DIRECT.GET.<stream>.<last_by_subj>
This for is preferred over the payload version as it can be secured and the subject level.If not an error (404, 408, etc), the request will return the value. 404 is returned if the request is valid but cannot resolve a message 408 is returned if the request is empty or invalid.
Headers constants
When getting a message directly, Nats specific headers are added:
Nats-Stream
Nats-Sequence
Nats-Time-Stamp
Nats-Subject
Nats-Last-Sequence
is also possibleClient Behavior
next_by_subj
field.max_msgs_per_subject
), the stream should be created withmax_msgs_per_subject
of 1. Also as a reminder, 64 is currently the current maximum allowed history per key that clients should allow.References
Clients and Tools
Other Tasks
Implemented
Client authors please update with your progress. If you open issues in your own repositories as a result of this request, please link them to this one by pasting the issue URL in a comment or main issue description.
Example unit test
allow_direct
set to trueThen test and validate headers for expected subject and sequence. I checked stream also and that a timestamp was present.
1
s1
s2
0
, next_by_subj =s1
0
, next_by_subj =s2
1
, next_by_subj =s1
1
, next_by_subj =s2
2
, next_by_subj =s1
2
, next_by_subj =s2
5
, next_by_subj =s1
5
, next_by_subj =s2