search

freedomofpress / ansible-role-elk

Installs a turnkey ELK stack for log aggregation and analysis, with optional Riemann support for alerting
18 stars 13 forks source link

Add UFW type and filter for /var/log/ufw.log #13

Closed ageis closed 8 years ago

ageis commented 8 years ago

@conorsch how's that?

conorsch commented 8 years ago

Excellent, that's exactly what I was talking about.

Before we merge: can we just ditch the syslog part of the ufw config entirely? Looks like ufw logs the exact same events to both places.

ageis commented 8 years ago

@conorsch Yup. Also note the slight difference in the grok filters. Should we stick with fine-grained fields, or greedydata for TCP options?

conorsch commented 8 years ago

The key/value arg parsing that the syslog pattern uses is a bit weak, but I think it's more valuable than having everything in one combined string field. Haven't ever run a search against any of those attributes, though, so it may be wasted effort. There's a kv filter for logstash that I haven't used before, and may be useful here.

Either way, let's use one pattern and one pattern only. My vote would be for the one without GREEDYDATA, but your call. If you store the pattern in a file called files/logstash-patterns/ufw, you can then reference it inside the ufw filters—DRY! Just make sure to include the patterns_dir parameter in the grok stanza when using custom patterns.

ageis commented 8 years ago

@conorsch I actually like GREEDYDATA for TCP options and think we should go with that, since I agree they are not useful for our searches, and if you look at a ufw.log with various BLOCK events, some lines don't have all those fields (ex. WINDOW, RES, RST, UGRP are not always included), so this additionally ensures that we'll always match.

conorsch commented 8 years ago

some lines don't have all those fields (ex. WINDOW, RES, RST, UGRP are not always included), so this additionally ensures that we'll always match.

Color me convinced. If we drop the syslog parsing, then we only have one pattern, which means we don't need to create a custom patterns file to reference it reliably. I like that simplicity!

conorsch commented 8 years ago

+1 This is great, merging