search

OTRF / OSSEM

Open Source Security Events Metadata (OSSEM)
MIT License
1.22k stars 212 forks source link

Thoughts about Elastic Common Schema (ECS) #34

Closed wesleyraptor closed 4 years ago

wesleyraptor commented 5 years ago

Seems like the Elastic Common Schema (ECS) is seeking to solve the same problem: establishing a naming convention with consistent field names across any data source. Just curious what your perspective is on ECS, where you see OSSEM addressing areas missed by ECS, etc. Thanks!

neu5ron commented 4 years ago

First I want to preface my comments/views are not speaking for @Cyb3rWard0g but I wanted to provide some insight I have seen.

Thoughts on ECS It is a great idea and a big step for the community. Schema for disparate/different datasets is something regardless of SIEM/database technology that is very important. Not only are they just creating a schema, they are actually writing many of the parsers and things in order to bring it to life for people without having to perform the manual tasks of parsing / ETL modifications. However, ECS is pretty young and missing a few things at the moment - but seems the things are already on the radar/road-map.

Thoughts on ECS & OSSEM I don't think there is competition/overlap. The best way I can try to explain is OSSEM is more granular. It's so specific that it gets down to the EventID of a Windows log, but not even just that... It is built based on deep understanding of the specific log and each individual field and value, how it applies from a threat hunting / data analysis perspective, and has multiple input/recommendations from people. Additionally, even though OSSEM is very granular - it still has the flexibility to support “broader” schemas like ECS. This is because the logs are mapped with the whole OSSEM schema in mind to provide common fields were appropriate. Also, elasticsearch has field alias types, this leaves the possibility for people to use OSSEM yet add compatibility for ECS... or vise versa... people using ECS to add compatibility for OSSEM.

Finally, it's a little known secret :see_no_evil: - IPs and Zeek network community ID and a few other things were introduced into HELK as ECS aliases... However, it didn't change any core HELK functionality or more importantly the greatness of OSSEM. I mention this, because I think this shows OSSEM & ECS in "harmony" in a live implementation/project.

Cyb3rWard0g commented 4 years ago

I agree with @neu5ron 💯 !!! What are your thoughts @wesleyraptor on ECS and what do you think when you go over OSSEMs repo? I would like to know what you find important or useful for your research. I also would like to say that OSSEM is not tied to any specific commercial tool so it is flexible enough to provide suggestions that would fit any use cases. I like the integration with ECS per @neu5ron comments above.

sp4ce-hUtch commented 4 years ago

I think the OSSEM CIM pretty much does the same thing as ECS, and then for mapping specific data sources/logs to the overall schema, OSSEM Data Dictionaries are similar to Elastic Beats Modules and their corresponding ingest pipelines, e.g. Zeek module for Filebeat. I do think the OSSEM Detection Data Model and ATTACK Data Sources efforts make this project more distinct and provides value to security practitioners/detection engineers - this is probably the more "granular" aspect that @neu5ron mentioned previously.

Also, may be better in a new issue/question, but any thoughts on pitching/requesting the OSSEM CIM to be the go to schema for Sigma fields used to write detection rules? The Sigma project sort of has a taxonomy, but it's not as developed as OSSEM, and it would be great to be able to simply write all Sigma detection rules using a generic vendor-agnostic schema like OSSEM. In such a case, I would be able to write a single Sigma Converter config file, mapping OSSEM-defined Sigma fields to my chosen backends - in my case, ECS based fields for Elasticsearch/ElastAlert-related backends.