opensearch-project / observability

Visualize and explore your logs, traces and metrics data in OpenSearch Dashboards
https://opensearch.org/docs/latest/observability-plugin/index/
Apache License 2.0
52 stars 96 forks source link

[FEATURE] Create an ECS integration to SS4O #1474

Open derek-ho opened 1 year ago

derek-ho commented 1 year ago

Security Dashboards is tightly following Elastic's ECS schema - if we have an integration for ECS it will help with security analytics integration use case, as well as folks who are transferring data from elastic to opensearch

YANG-DB commented 1 year ago

are you referring for AWS - ECS integration ?

derek-ho commented 1 year ago

Updated description - I am talking about elastic's ECS https://www.elastic.co/guide/en/ecs/current/index.html

YANG-DB commented 1 year ago

I think we should approach this in a gradual manner - Integration by Integration ... As soon as we are presented with a new log-based Integration request - it should also be accompanied with the appropriate catalog-component representing its fields content...

Currently our schema catalog approach supports incrementally adding new log type support which formulate a components sub categories under the category section of the schema.

Each of these component (sub-categories) represent some scope of fields that address the relevant domain they are associated to (cloud, registry, file-system, Network ...)

NginX example: For the first Integration of the NginX web server, we added the http.mapping support to reflect the needs for the Nginx integration for http based log fields

In a similar manner, once we will be introduced with a requirement for K8 Integrations - for example, we will add the appropriate compound fields mapping to the Observability logs folder and thus allow the Integration template to explicitly declare its dependency on that component.

This mechanism helps the Integrations loading process to know which compound template it need to add to the root log template sso_logs_mapping template .

Once the template generation is complete - it will be able to support the integration's data-stream ingested documents