search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.82k stars 8.21k forks source link

[Ingest Manager] Support custom ILM policies #72986

Open neptunian opened 4 years ago

neptunian commented 4 years ago

Currently all index templates point to the corresponding type of data ILM policy which comes installed in ES calls logs or metrics. We need to support a custom ILM policies, either per package or per dataset or both. @ruflin recently made a change to a dataset manifest file to override index template mappings or settings: https://github.com/elastic/package-registry/issues/497. But I think the ILM policy file can be added per dataset in the ilm_policy directory before the index template is created and referenced there. The index template should reference the most specific policy. Most specific being the dataset and least the global). Policies will need to be tracked and removed like other installation assets.

Note: There is a stack monitoring package that @ycombinator will be introducing in the future that will need ILM policies per dataset package.

elasticmachine commented 4 years ago

Pinging @elastic/ingest-management (Team:Ingest Management)

ycombinator commented 4 years ago

There is a stack monitoring package that @ycombinator will be introducing in the future that will need ILM policies per dataset.

Strictly speaking it will be per package but we could define the same policy per dataset too, if that's all we decide to support.

ph commented 4 years ago

@mostlyjason I don't think we need that with data tier? correct?

mostlyjason commented 4 years ago

Yes we are planning a data tiers feature that will allow users to set retention and other settings. @mukeshelastic is helping to lead that