New `logstash` sink

[binarylogic]

A note for the community Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request If you are interested in working on this issue or have submitted a pull request, please leave a comment Use Cases Vector already accepts Beats inputs, but it's missing the ability to output data in this format. That would allow for:

reliable data delivery to secured Elasticsearch clusters that use Logstash input as a security layer easier migration path from Beats + Logstash to Vector, as only one of components would need to be replaced at a time Attempted Solutions No response

Proposal Add a Beats (Logstaqsh) output, supporting

TLS compression multiple target addresses with load-balancing acknowledgements References No response

Version No response

[zoulja]

I have to mention, Logstash supports loadbalancing, so in my case of Filebeat+Graylog cluster I don't have to use dedicated load balancer, just set loadbalance:true in Filebeat configuration

[tshepang]

I think it would be smart to separate the concerns of trying to be a swap in replacement for Filebeat and sending data to Logstash. For example, a filebeat_record_former transformer could be paired with the logstash sink to create a replacement for filebeat.

What would filebeat_record_former mean?

Anyways, was just about make use of this functionality (logstash sink), and then remembered that it doesn't exist :)

[tshepang]

Having this would ease a transition away from Filebeat, by removing the need to coordinate the transition between senders, who are all behind different firewalls (separate customers), and receivers. Implication of this is that the senders are not easily accessible (restrictive policies), and a gradual migration is the only practical thing, where opening additional network ports would require lots of coordination.

[jszwedko]

Duplicate of https://github.com/vectordotdev/vector/issues/16397

[DSmithVA]

We need to push to Logstash, so having a more versatile and durable option than Filebeat would be phenomenal.

I hope that the proposal's "multiple target addresses with load-balancing" implies there will be timeout/failover settings exposed, as well as allowing multiple connections per target host:port, as those have been critical in our experience.

[haiwu]

@jszwedko : Any plan to add support for this one?

[jszwedko]

Not planned, but I think we'd be open to a contribution. Alternatively you can integrate Vector and Logstash via the Vector HTTP sink and the Logstash HTTP input.