[REQUEST]: Default index settings and mappings

[joshdover]

Description

Fleet & Agent integrations provide a number of global index settings and mappings defaults which are designed to strike a good balance for a typical use cases between indexing performance, search performance, and storage size.

We should document what these defaults are, the rationale for choosing them, and also consider documenting how to tweak them for optimizing for certain use cases.

Should include at least the following:

• index.codec
• number_of_shards
• number_of_replicas
• number_of_routing_shards
• refresh_interval
• index: false / doc-only-value
• mapping.total_fields.limit
• _source: synthetic
• TSDB-related settings

Resources

• https://github.com/elastic/kibana/issues/104620
• https://github.com/elastic/obs-infraobs-team/issues/405
• (not yet implemented) https://github.com/elastic/integrations/issues/3419
• Slack thread about compression recommendations
• Related Elasticsearch Docs:
  • Tune for indexing speed
  • Tune for search speed

Collaboration

TBD. The docs and product team will work together to determine the best path forward.

Point of contact.

Main contact: @joshdover

Stakeholders: @jsvd @amitkanfer

[amitkanfer]

@mukeshelastic can we please prioritize with @karenzone ?

[amitkanfer]

This is becoming more important as we think about introducing data fidelity modes.