search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
17k stars 1.47k forks source link

jetstream support in nats sink #10534

Open abh opened 2 years ago

abh commented 2 years ago

As mentioned in #8112 the nats sink was merged without jetstream support. At least for my use case, jetstream is an essential feature for the log pipeline.

(see also #10533 for authentication and TLS)

sghazaryan commented 1 year ago

Nats sink still beta and without Jetstream support. Any plans to support Jetstream ?

jszwedko commented 1 year ago

Hey @sghazaryan ! This isn't on our near term roadmap, but we are happy to help support a contribution here for anyone so motivated.

sghazaryan commented 1 year ago

Hey @jszwedko ! Thank you for answer. Hope after async_nats update jetstream suppot will be implemented.

Jarema commented 1 month ago

I can work on this somewhere in Q4. Happy to be asigned to it.

jszwedko commented 1 month ago

I can work on this somewhere in Q4. Happy to be asigned to it.

That'd be great!

Jarema commented 3 weeks ago

We realized that the current sink can also flood the JetStream as it sends all messages as Core NATS publisher.

Changing to JetStream publish would not only allow actual acks to be available, but also introduce new sets of possible errors (non existing streams for published subjects etc).

How would you like that to be handled:

jszwedko commented 3 weeks ago
  • a option to use either JetStream or Core NATS? (Core NATS sink can make sense too)?

I think this would make sense. If it diverges enough we may want a separate sink, but if it is only minor modifications to the implementation and user-configuration structures then just updating the nats sink seems sensible.

  • is it ok to introduce new errors to the sink?

Yes, I think this would be ok. Again it really depends on how much the Jetstream sink behavior diverges from the existing NATS sink.

Jarema commented 3 weeks ago

It diverges quite a bit in one sense @jszwedko :

Current sink can publish into Core NATS subjects, meaning very performant at-most-once without persistence (and NATS does not have the concept of creating subjects, its all based on interest graph).

With JetStream, there is at-least-once, and a persistence layer.

JetStream publish will fail if stream for given subject does not exist, which would be a breaking change. If there are use cases in Vector for non-persistent sink, then it would break those. If Vector should always be used with JetStream - then it's fine, though I suppose it's the former. Then, a separate sink sounds more sensible?

jszwedko commented 3 weeks ago

Thanks for the additional detail @abh . JetStream still relies on the NATS protocol, though, right? Even though the messages are handled differently.

Jarema commented 3 weeks ago

Yes. JetStream is built on top of Core NATS.

jszwedko commented 3 weeks ago

👍 in that case what exactly is missing from the existing nats sink to be able to publish to JetStream?

Jarema commented 2 weeks ago

@jszwedko It is able to publish to JetStream, but:

  1. It is sending publish, instead of requet, or rather - jetstram.Publish, that allows it receive ack
  2. Because of not using jetstream.Publish, it is publishing without any throughput control, flooding JetStream with messages.