search

Cyb3rWard0g / HELK

The Hunting ELK
GNU General Public License v3.0
3.75k stars 680 forks source link

How to integrate custom grok filters in this stack? #540

Open ordin2342 opened 3 years ago

ordin2342 commented 3 years ago

Hi,

thanks for providing this great boilerplate. I've just installed everything 2 days ago, regarding versions.

I've managed to forward syslogs from my log-aggregator to Kafka - and I'm able to see my logs in Kibana under "indexme-*"

But they're not pretty and I'm not sure how to use any predefined dashboards, like for syslog events or how to actually go Hunting using Mitre-Att&ck rulesets.

I'd like to use my own grok filter cause right now the data is not very usable. My filter would be: %{SYSLOGTIMESTAMP:timestamp} %{HOSTNAME:hostname} %{SYSLOGPROG:servicename}.*?(%{IP:remoteip})?.*?(%{POSINT:remoteport})? for starters.

Usually I would make it somewhere in logstash.conf but this doesn't even exist here.

I'm aware that you can't provide basic ELK user support, but would be very thankful for a little bit of guidance, as I said Kafka within the stack makes it a little more complicated than I'm used to.

Many thanks!

Cyb3rWard0g commented 3 years ago

Hey @ordin2342 !

The folder pipeline is mounted to the logstash docker container. Therefore, you can add your .conf files there with your grok filter right?

ordin2342 commented 3 years ago

Thanks, that all worked out. Got another issue, gonna post it right here... if it's not ok, I'll open a new issue.

I heavily modded the HELK to work with Linux logging and public auth. Now I wanted to connect internal Windows clients again. At first I send winlogbeat to Kafka nothing seemed to arrive at Elastic, even though I couldn't find an error. When I send winlogbeat directly to Elastic it worked but it arrived at the wrong index: winlogbeat- <- so now the HELK sysmon dashboards don't work anymore. So I went back to Kafka, and again logs seem to arrive at winlogbeat-

Can you help me find the Kafka config file where I can decide the index for winlogbeat? Maybe I'm missing a "winlogbeat setup" or I accidently modded something else. I could setup a new HELK and know it would work out-of-the-box but I rather have it in one machine and I wanna learn this system.

I know it should arrive at logs-endpoint-winevent-* but not sure where that's decided, at Kafka or at Logstash? Made lots of mods to Logstash, never used Kafka in the stack till this point.