search

elastic / logstash

Logstash - transport and process your logs, events, or other data
https://www.elastic.co/products/logstash
Other
14.2k stars 3.5k forks source link

Create a logstash output beats #5867

Open ph opened 8 years ago

ph commented 8 years ago

Since LSF is now end of life it makes sense to logstash to have a logstash-output-beats, this plugin could leverage the java rewrite and use the encoder in the test. The current ruby implementation doesn't work when you have an intermediate ca in the chain, it will refuse to complete the handshake.

What need to be done:

fjiang212 commented 7 years ago

This plugin will be very useful for our cases. We have many logstash shipping instances collecting data from different the business unit and then send to kafka. The architecture is very similar to the once describe in this document(https://www.elastic.co/guide/en/logstash/5.0/deploying-and-scaling.html). Now we have to drop message queue because infrastructure issue. Then we need to find another output to build the bridge between the logstash shipping instances and logstash indexing instance. This new plugin will be perfect fit for our case. And I guess it will send the data to logstash indexing instance beat input and support the following features like the filebeat have:

Could you implement it in the logstash v5.3 since I found it has been rescheduled couple times already.

And logstash v5 have a persistent queues now. Hope this plugin + persistent quest can kind of replace the message queue to simply the whole Elastic stack architecture. https://www.elastic.co/guide/en/logstash/current/persistent-queues.html

Thanks, Feng

ph commented 7 years ago

@fjiang212 We don't currently have a timeline for it, buts its certainly one of our priorities.

ph commented 7 years ago

@cgough is the SQS input/output an option for you until we add a beats outputs?

jordansissel commented 7 years ago

[2017-02-20T18:39:07,085][ERROR][logstash.outputs.lumberjack] Client write error, trying connect {:e=>#<RuntimeError: Whoa we shouldn't get this frame: >, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/1.9/gems/jls-lumberjack-0.0.26/lib/lumberjack/client.rb:146:in `ack'",

The above is a strange error. THe log message code:

https://github.com/elastic/ruby-lumberjack/blob/master/lib/lumberjack/client.rb#L146

raise "Whoa we shouldn't get this frame: #{type}" if type != "A"

Your log message doesn't show anything for the 'type' (value read from the socket). It might be a nil value, which indicates the socket was read, but closed, so it returned nil.

If so, this indeed is probably an indication that Logstash (the receiver, beats input) is overloaded or stalled and is rejecting connections. Your Logstash with beats input should log when it is rejecting connections, but I might be remembering incorrectly.

THe output-lumberjack plugin should still work because the protocol hasn't changed in a way that rejects it, if I'm remembering right.

Anyway, we have plans to make a beats output, and will do it when we can. Hopefully for Logstash 5.5.0 time frame.

alexef commented 7 years ago

We are also interested in this.

jordansissel commented 7 years ago

overall much more reliable than using lumberjack

The purpose of the lumberjack (now beats) protocol is to provide reliable transport. If you are finding anything unreliable about this transport, it is a bug, not a feature, and if you have the energy, I invite you to file an issue about what you experience with these bugs.

joshFive commented 6 years ago

I would like to add my interest in the logstash-beats-output.

lsoumille commented 6 years ago

Any progress on this ?

widhalmt commented 5 years ago

Is this still on your list?

AnthonyJClink commented 4 years ago

We leverage beats and logstash in an IOT project where our instance sits between low connected devices and an enrichment process we have in our data center. I feel that our product would gain quite a bit if we could push this IOT data via beats.

I would love to help with this issue if the LS team can help guide me to a possible implementation. I believe I could give it a stab.

allamiro commented 1 year ago

Any progress on this issue ?

tanganellilore commented 1 year ago

Team, any news on this?

Lumberjack output have a loto of "bugs" in CA, not allow the usage of CA, not allow to use private key for mtls and use ip address for socket instead fqdn.

I can submit some PR to add all of these features o lumberjack output and in the ruby_lumberjack, but seems that all of them are not supported and maintabed by anyone.

Let me know if I can supoort