In order to identify which events are not being parsed by logstash and are being dropped, we should enable the logging of anything that reaches the end of the pipeline and is still tagged as a _grokparsefailure. ie In /etc/logstash/conf.d/99-output.conf add the following to the output section:
if "_grokparsefailure" in [tags] {
file { path => "/var/log/logstash/grokparsefailure.log" }
}
This may also require that logstash is started with the --verbose flag. ie In /etc/default/logstash add --verbose to LS_OPTS.
To ensure that this doesn't cause support issues the log file should be log rotated.
In order to identify which events are not being parsed by logstash and are being dropped, we should enable the logging of anything that reaches the end of the pipeline and is still tagged as a
_grokparsefailure. ie In/etc/logstash/conf.d/99-output.confadd the following to the output section:This may also require that logstash is started with the
--verboseflag. ie In/etc/default/logstashadd--verbosetoLS_OPTS.To ensure that this doesn't cause support issues the log file should be log rotated.