NATS & NATS Streaming support in an end-to-end STREAM BASED scenario

vectordotdev / vector

A high-performance observability data pipeline.

https://vector.dev

Mozilla Public License 2.0

18.28k stars 1.61k forks source link

NATS & NATS Streaming support in an end-to-end STREAM BASED scenario #11865

Open djpable opened 2 years ago

djpable commented 2 years ago

A note for the community

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Use Cases

Useful to connect vector agents and vector datacenter instances.

Attempted Solutions

I tried to use NATS Sink component in the agent and a NATS Source component in the datacenter instance, failing in transmitting Metrics data-type. It seems that those components supports only Log data-type.

Proposal

No response

References

No response

Version

vector 0.20.0 (x86_64-pc-windows-msvc 2a706a3 2022-02-10)

jszwedko commented 2 years ago

Hi @djpable !

We are actually working on a feature right now that would enable this with #9930 . Feel free to follow that issue for updates.

djpable commented 2 years ago

NATS supports perfectly JSON encoding of Metrics and Logs, but that native encoding sounds very interesting. Thank you. You are doing a great product.

A NATS Source output:metrics and the relative NATS Sink would be very appreciated.

martinpales commented 2 months ago

Hi @jszwedko, is there any update on this? The NATS sink still doesn't accept metrics, only logs. Thank you.

jszwedko commented 2 months ago

No updates yet. I think the change is relatively straightforward though, if anyone wants to take a shot at it. It would mean relaxing:

https://github.com/vectordotdev/vector/blob/ca0fa057eaa128beb7777428f79cec9924f1d396/src/sinks/nats/config.rs#L121

to be

 Input::new(self.encoding.config().input_type())

and doing some testing to make sure metric events are handled correctly.