search

elastic / ecs

Elastic Common Schema
https://www.elastic.co/what-is/ecs
Apache License 2.0
1.01k stars 418 forks source link

Field event.type should not be reserved #447

Open vbohata opened 5 years ago

vbohata commented 5 years ago

I found event.type, event.category, ... are planned for some canonical categorization with permitted set of values. From my perspective I do not think it is really possible and could make things very unclear. From our logs I see many sources where event.type, event.category is appropriate to use but sometimes contains very different values. We already use event.type, event.category, ... with our org prefix because it fits to the use case. We also defined another field to help us with categorization.

For example we have system which generates multiple feed types, each feed type has logs of multiple categories and types and kinds. Such as:

This is perfectly clear and easy to understand during search. Also very usable for different event soruces. You can see TUNNEL_EVENT category of events perfectly fits to event.category but I really do not thing it would be in ECS the permitted list values for this field. For example for exchange tracking the event.type can be NOTIFYMAPI ...

I think the effort for defining permitted values .... is great but there are a lot of use cases where it is impossible. So maybe all the event fields should be allowed to use (just document how without permitted values list) and something like "event.canonical" with multiple subfields would be great to introduce.

willemdh commented 5 years ago

Kind of agreeing on this... I have made similar remarks in https://github.com/elastic/ecs/issues/316

webmat commented 5 years ago

Hey folks, I agree we also need a place to put the "raw" event types, categories & so on. It's one of the things we'll be addressing next.

You're also right that coming up with a broad enough list of expected values for these fields will be quite a thing. We're working on it :-)

ptinsley commented 5 years ago

If the idea is to get broad adoption of this schema, which would be awesome, restricting field values in such a ubiquitous concept as "event" is shooting that idea in the foot. I would understand if it were a narrowly focussed idea like a firewall log but every app will have events that they need to emit and there is just no way you can encompass all of the ideas each type of app could represent.

One of our apps has these possible dispositions that I need to fit into an event logging model: served-cdn-cached served-cdn-uncached served-origin-cdn-fail served-origin-direct failed-origin-direct failed-origin-cdn blocked

And that's just one of our apps, one that I happen to be trying to use this schema with currently

If I have to avoid the event model in ECS honestly I'm just going to take ECS and make my own thing, which completely defeats the purpose but I have to get my work done and being forced into custom logging where this event concept is exactly what I need just won't work.

MikePaquette commented 5 years ago

@ptinsley The intent behind temporarily marking certain event.* fields as "reserved" is not to "restrict" field values, nor to try to categorize all the world's custom apps.

Instead, this is a temporary measure to allow us time to define a set of ECS categories without causing early ECS users to experience breaking changes within their mappings, once these enumerated category values become available.

A primary goal of defining such a set of ECS categories is to enable analysis content (searches, aggregations, visualizations, dashboards, alerts, ML jobs, etc.) for common use cases to be shared and re-used by ECS community members.

I expect that ECS event.category will be defined to enumerate perhaps 30-50 categories into which events from common use cases like networking, security, IT operations, IT infrastructure, APM, metrics, cloud infrastructure, monitoring, etc. can be reasonably placed.

Applications that fall outside these common use cases will be considered custom apps, and should be able to use the event.* categorization fields without conflict, although there may be some conditions placed on them.

As @webmat mentions above, another immediate option would be to create a place for "raw" event categorization fields, and rely on them in the interim.

Question for you: For the app you describe here, would you consider this to be a common app category for which you'd expect to be sharing analysis content with other users? Or would you consider this a custom app for which your analysis content is not likely to be shared?

vbohata commented 4 years ago

Or would you consider this a custom app for which your analysis content is not likely to be shared?

In many use cases this one.