search

elastic / rally

Macrobenchmarking framework for Elasticsearch
Apache License 2.0
1.91k stars 314 forks source link

[Metrics store] Add support for custom templates #1854

Closed inqueue closed 1 month ago

inqueue commented 1 month ago

This change adds the ability to customize indices in the metrics store. The goal is to support ILM policies and eventually shift to using data streams.

gbanasiak commented 1 month ago 
TypeError: put_index_template() got an unexpected keyword argument 'ignore_missing_component_templates'

We would need to modify assumption on metric store ES version to support this approach. We're still maintaining compatibility with 6.8 for target cluster which I think is something we could drop, but 7.17.x must stay for now. Unfortunately ignore_missing_component_templates was added in 8.7. Do we want to bifurcate and have different requirements for metric store and target cluster?

inqueue commented 1 month ago 
TypeError: put_index_template() got an unexpected keyword argument 'ignore_missing_component_templates'

We would need to modify assumption on metric store ES version to support this approach. We're still maintaining compatibility with 6.8 for target cluster which I think is something we could drop, but 7.17.x must stay for now. Unfortunately ignore_missing_component_templates was added in 8.7. Do we want to bifurcate and have different requirements for metric store and target cluster?

Thanks, Grzegorz. I hadn't realized ignore_missing_component_templates was so new. The parameter is not even supported in elasticsearch-py until 8.14 via https://github.com/elastic/elasticsearch-py/pull/2525 (Rally uses 8.6.1 currently).

Rally would need to load an empty *@custom component template to support this, if one did was not already in the metrics cluster:

{
  "template": {}
}

The ultimate goal was to slowly move toward using data streams, or at least provide the option for it, to allow for better management of metrics retention, and pushing metrics to serverless. We are not there yet on any of these fronts, and therefore, closing this PR.