ECS Safety template for field types needed, so different versions can be supported

[Randy-312]

So that new filebeat versions that send into our 701 ECS template do NOT break our kibana index patterns mapped to filebeat-7.0.1*, I'd like to have a base template that factors in all field types, per the naming standards.

i.e, everying ending in ip is geo and all other conventions for naming standards (which I cannot find)

[webmat]

I'm not sure I understand what you're asking here, can you elaborate?

Please also note that while the ECS team tries to consider & contribute to Beats, the right place to report specific Beats problems is on the Beats repository, not here.

ECS is meant to be a schema people can use regardless of their ingestion pipeline (Beats, Logstash, etc.) Therefore ECS is independent from Beats.

[Randy-312]

If we do NOT have a template setup for a 7.3.0 release and someone starts sending in from a 7.4.0 filebeats, then it will go INTO the filebeats-7.4.0-yyyymmdd index.

Since our Kibana has an index pattern mapped to filebeats-*.... , then elastic will do its best to identify the field types, and inevitably get some wrong. Yes, we DID have this happen when someone sent in 7.2 filebeats to our environment that did NOT have the 7,2 templates loaded.

In OUR schemas, we force all unknowns to 'strings', and then have a naming standard for fields. _ip _int _date, _timestamp, _time

This lowers the likelihood of someone sending in something that can Break existing visualizations. Because when they do break, they need my team to fix it..

Having an ECS safety template would allow us to lessen how often visualizations break.

i.e., we could have a 7.x template or, depending on your response toECS-1246, could have that at 7.4.x.

[webmat]

Thanks for the clarification. I think I understand what you're asking.

I don't think providing a minimalist template would solve the problem. ECS defines a few hundred fields, but Beats populates thousands of fields (in addition to ECS fields).

So even with a template catching some of the common naming patterns & types, most of the time, someone starting a Beat without having run setup first will require the same work to fix the situation.

Note however that a Kibana index pattern is a transient artifact. If a Beat starts and messes it up, all you need to do is delete the incorrect index, refresh your index pattern and you're back in business.

If you object to how Beats handles the creation of indices, templates etc, please open an issue on the Beats repository.

[Randy-312]

So, there never will be a standard around naming for non-string fields.
Which causes problems in Kibana when people do inevitably use the wrong version of elasticsearch that we don't have a template loaded for. The existing index patterns tend to break, requiring a call out to my team to identify the problem, find the culprit index, then delete it.

This extra work / risk due to lack of formatted fields, leaves us with a bit of challenge in using ECS.

We can't simply say send anything that's using ECS 1.2, unless we a) understand all possible senders (Filebeat, winlogbeat, functionbeat) and have loaded templates for those, since each one has it's own index pattern / template tied to it. b) handle (reject) anything NOT on the defined list, and versions. Otherwise we'll have a lot of 'guesses' that elasticsearch does for formats. See original concern on index patterns

Then, when a 7.4.1 comes around, we need to 1) Take the time to board each template for each product into elasticsearch. 2) Migrate pipelines (grok patterns) into all relevant logstash nodes. (At least ECS isn't changing at the patch level) 3) Let people know they can now start using the new version of the products.

Does that all sound right, as a somewhat manual way to address this lack of naming standard / field types?

I'm just trying to illustrate what it's going to take to use ECS and handle the evolving templates in each patch level, if we do NOT have a safety template.

I suppose alternatively, the products could just have templates at the Minor version, and not the Patch level. This would at least cut down on how often we have to touch the code.