elastic / elasticsearch

Free and Open Source, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
69.95k stars 24.74k forks source link

Ability to Read-Only a Data Stream #100439

Open BenB196 opened 1 year ago

BenB196 commented 1 year ago

Description

Enhancement

It would be nice that a data stream could exist in a strictly historical/archival form, similar to how an index can be made read-only, but at a data stream level (data stream mappings can't be updated, data stream can't be rolled over, even if "forced").

Use Case

Because data streams mainly target timeseries data, there are cases where a data stream is no longer actively needed, but needs to be kept around because it contains historical data. An issue that I've observed is that there is no way to mark a data stream as archived and that it shouldn't receive things like mapping updates or rollovers.

An example of where this is an issue is with Elastic Fleet. If I have a data stream with a specific namespace value, that I eventually stop using, Elasticsearch knows not to roll it over because the leading index has 0 docs, but Fleet doesn't seem to know this, and if the Fleet integration gets an update that changes mappings, Fleet will see this data stream and force a rollover, causing 0 doc indices to exist.

It would be nice to have the ability to mark a data stream in a way that things like Fleet (or other similar products), no-op changes on archived data streams.

elasticsearchmachine commented 1 year ago

Pinging @elastic/es-data-management (Team:Data Management)