search

logstash-plugins / logstash-input-beats

Apache License 2.0
86 stars 81 forks source link

Setting logstash.inputs.beats: TRACE does not produce any additional logging #204

Open ppf2 opened 7 years ago

ppf2 commented 7 years ago 
curl -XPUT "http://localhost:9600/_node/logging" -d '{
    "logger.logstash.inputs.beats" : "TRACE"
}'
  "loggers" : {
    "logstash.agent" : "INFO",
    "logstash.api.service" : "INFO",
    "logstash.codecs.jsonlines" : "INFO",
    "logstash.codecs.plain" : "INFO",
    "logstash.filters.date" : "INFO",
    "logstash.filters.grok" : "INFO",
    "logstash.filters.mutate" : "INFO",
    "logstash.filters.ruby" : "INFO",
    "logstash.inputs.beats" : "TRACE",

When using the logging API to increase the logging of the beats input, it produces no additional logging. It can be helpful to log event content at debug or trace level so admins can turn this on dynamically for troubleshooting (to see what is flowing through the beats input).

jordansissel commented 7 years ago

What are you trying to diagnose?

jordansissel commented 7 years ago

to see what is flowing through the beats input

What do you want to see? What action would you take once you see it?

allenmchan commented 7 years ago

This is the scenario we were going through. You can make determination if an enhancement is needed. We were seeing events that was not properly formed in kibana. Multiline messages were cut in half and partial messages were shown. Support was working with me to trying to isolate where the problem is. We added a logstash file output to see the events leaving the logstash instance before being inserted into kafka. We confirmed the events were already partial at that point and therefore can rule out kafka and all downstream parts of the pipeline. We are now turning our attention between filebeat and beats input.

I believe that a debug or trace level in beats input that show the events coming through the input, admins can quickly isolate where to focus troubleshooting attention. Having this info in the input stage rules out the filter stage and gives admins another way to isolate issues.

Hopefully that gives you the reasoning behind this request.

andrewvc commented 6 years ago

This is a great idea, but I think it makes more sense to build this sort of sampling directly into Logstash core rather than as one-offs in individual plugins.

I think what would be generally more useful would be a feature in logstash whereby a plugin could be referenced by ID and some of its output sampled. WDYT?

andrewvc commented 6 years ago

I've setup https://github.com/elastic/logstash/issues/8581 to discuss a design solution that is more general to logstash here.

andrewvc commented 6 years ago

I'll add that I wouldn't reject a patch to the beats input that does this, but it doesn't solve the problem in a general enough way to warrant special focus.

jakelandis commented 6 years ago

In original comment, only the Ruby side of logging was is working. There are a couple trace messages coming from the Java side of things, but the Java logging had issues in earlier versions.

The Java logging didn't work until 4.0.5 + LS 5.6.2...but since LS 5.x is pinned to 3.x of beats, Java logging still doesn't work by default on any released version. The 3.x backport PR to fix the java logging stalled.

I agree with the sentiment here of more logging, just wanted to clarify the the current state of Ruby vs. Java logging.