Maybe use csv filter instead of grok?

[jordansissel]

Howdy!

I was helping someone with bro/logstash stuff and I found brostash. This is a great idea!

I saw it outputs grok filters with tab separations, but the csv filter is super nice for this kind of data:

Example fields input:

capture_loss.log:#fields    ts  ts_delta    peer    gaps    acks    percent_lost

Possible logstash filter config:

filter {
  csv {
    columns => ["ts", "ts_delta", "peer", "gapsacks", "percent_lost"]
    separator => "\t"
  }
}

Should work? Also should be more readable than the grok patterns.

Any thoughts? :)

[jordansissel]

I also wonder if we could do a 'bro' filter instead, something that would look for this '#fields' line and keep a cache of known csv filters and just automatically do everything that brostash does, but doing it in logstash in real-time instead of reequiring user intervention. I'm happy to help you build that if you want, and I'm sure Bro+Logstash users would love it.

[jeffgeiger]

Short term, I'll swap in the CSV filter.
As to a 'bro' filter, which is more productive: Creating a filter for logstash to parse bro logs, or modifying Kibana to read bro records written into ES by Bro's native ES output? I'm guessing there are use cases where people may not be writing directly from Bro in to ES.

[jordansissel]

If you use a logstash filter, you'll get structured events before hitting ES, this lets you output to ES and any other outputs (or doing extra filtering!). Useful! Also, because it's already structured in ES (due to logstash), you can do structured queries and aggregations to analyze data.

If you leave it unstructured, you cannot reasonably do structured queries in Elasticsearch nor can you use most facets/aggregations on it.

If all you want to do is look at individual log records, it probably doesn't matter what format it is in. If you want to do structured queries (query by time range, query by peer or analyze the distribution of the percent_lost field, or any other fields, you must have it in a structured form in Elasticsearch, so I strongly recommend using a logstash filter, not implementing some trickery in Kibana simply to render the output.

[dcode]

Has there been any luck with this? I'm willing to write code, but I can't wrap my head around how Logstash might handle this. It looks like the "event" passed to a filter is stateless. How can we track a pattern to a file when we're just given a line? Or does Logstash create an instance of each filter per entry in the config? If that's the case, then it's easy.

EDIT: I'm tracing through the Logstash source, and I think I'm starting to get it. The filter entry can specify the config for a given "type" field that gets added, which is accessible to the filter method.

[dcode]

I'm outta time for the day, but I came up with this untested, blindly coded filter. I think it's at least in the ballpark, but I'm neither a Ruby expert nor LogStash expert (though I've written a similar parser in Python and awk). Thoughts? It may not even work yet, syntactically

https://gist.github.com/dcode/60ac10d8f584a17dc77f