search

vectordotdev / vector

A high-performance observability data pipeline.
https://vector.dev
Mozilla Public License 2.0
18.04k stars 1.59k forks source link

Splunk S2S source #2537

Closed MadsRC closed 2 years ago

MadsRC commented 4 years ago

I'd love to see Vector support the Splunk S2S v3 protocol (S2S = Splunk 2 Splunk) as a source (A sink would be nice to have).

The S2S v3 protocol is the protocol in use by all newer versions of Splunks products whenever they ship data in between each other.

It should allow Vector to receive data from Universal Forwarder deployments, allowing Vector to receiving data from Splunk deployments (Using S2S) and forwarding it to Splunk using the HEC sink.

I'd love to contribute some code, however my Rust is... very rusty... I can, however, contribute with a series of traffic dumps of Universal Forwarders speaking to other Universal Forwarders in a controlled environment.

binarylogic commented 4 years ago

Thanks @MadsRC, I agree, we'd love to have this feature. If you want to take a crack at contributing feel free. Would this break backward compatibility with our current Splunk integration? Or are you thinking these would be new sinks/sources?

satellite-no commented 3 years ago

I want to plus 1 this as a very useful feature request similar to #3848 & #4562.

MadsRC commented 3 years ago

@binarylogic - I can't believe that I never got around to answer, how rude of me. Please do accept my apologies.

For the questions: It shouldn't break backward compatibility as this is an entirely new/separate sink/source. S2S is an inhouse proprietary protocol that have gone through quite a few iterations at Splunk. I believe they are at version 7 or 8 right now (Could be wrong, haven't looked at the protocol for a year or so).

I did some reversing of the protocol last year and eventually came up with an implementation that would convince the sender that they where speaking to a Splunk Universal Forwarder. The benefit of the S2S protocol, compared to a regular Splunk HEC, is that the S2S protocol includes metadata, such as original filename and source type.

I have a feeling I'll be digging in to the protocol soon again, and if I'm able I'll try to contribute some code back to you guys ;)

jszwedko commented 3 years ago

Thanks for all of the comments @MadsRC ! If you do dig back into the protocol, we'd love to see what you come up with.

jszwedko commented 2 years ago

Closing this as a duplicate of https://github.com/vectordotdev/vector/issues/3848