search

coolacid / GettingStartedWithELK

Getting Started with ELK
50 stars 7 forks source link

Added example of the checksum filter for deduplication of events. #15

Closed shurane closed 10 years ago

jordansissel commented 10 years ago

FYI: checksum filter will be removed soon. Recommend using fingerprint filter instead; it does similar but performs better.

shurane commented 10 years ago

@jordansissel Looks like it otherwise works the same way.

Hm, I'm kind of confused with trying to use fingerprint {}. Why is the key field required for sha1 but not for murmurhash3? Is this related to the anonymize word elsewhere?

https://github.com/elasticsearch/logstash/blob/v1.4.2/lib/logstash/filters/fingerprint.rb#L44

shurane commented 10 years ago

Alright, updated configuration with usage of both the fingerprint and checksum filters. Note that with logstash 1.4.2, the @timestamp field doesn't work with the fingerprint filter. Find out more about it at: https://github.com/elasticsearch/logstash/issues/1572