search

Cyb3rWard0g / HELK

The Hunting ELK
GNU General Public License v3.0
3.73k stars 675 forks source link

Kafka enrichment for MITRE tagging via Event List #523

Open priamai opened 3 years ago

priamai commented 3 years ago

Hi there, I believe your colleague Miriam has an interesting project called EventList from Microsoft Security Baselines that can be used to enrich directly in Kafka the windows event logs. What you could do is to take the posgtresql schema from her repository: https://github.com/miriamxyra/EventList register it as a Kafka table and then join the incoming events based on the Event id.

Happy to provide some more info.

priamai commented 3 years ago

I can see there is a Logstash pipeline here: https://github.com/Cyb3rWard0g/HELK/blob/master/docker/helk-logstash/pipeline/0003-attack-input.conf where it does a join on the MITRE databse. Would love to have some documentation on that to see if it has been done already.

priamai commented 3 years ago

More pointers: Miriam talk about EventList Windows Logging Cheat Sheet Github repo but outdated malwarearcheology

Cyb3rWard0g commented 3 years ago

Very interesting yeah that would be great to add to events . I have to take a look at Miriams mapping. I am working with the ATT&Ck team on this document to have a methodology behind the events mappings. We have come up with all this so far https://ossemproject.com/dm/mitre_attack/security_events_mappings.html

The good thing about this is that we have all those mappings as a YAML file (one file only). Easy to use in the HELK deployment.

Regarding using Kafka, I believe we would need to use KSQL for that right> (JOIN). Maybe we can add the YAML file as a dictionary and do the enrichment at the Logstash level. Maybe? What do you think?

priamai commented 3 years ago

Ah that OSSEM mapping is a gem, yes we should check whether Miriams is a super set of the OSSEM or whether they are just identical. I will see if I can convert the table I have into YAML and let you know. Yes that is a fair point we would need to add KSQL dependency and yes is best to do it via Logstash but I am not very good at it to be honest ahah! I think in terms of performance since is a dictionary lookup there shouldn't be a big improvement KSQL vs Logstash.