search

elastic / elasticsearch

Free and Open, Distributed, RESTful Search Engine
https://www.elastic.co/products/elasticsearch
Other
68.56k stars 24.35k forks source link

Make reindexing a tsdb data stream easier #99177

Open martijnvg opened 10 months ago

martijnvg commented 10 months ago

Re-indexing a tsdb data stream is more challenging than re-indexing a regular tsdb data stream. This is because when a new data stream is created, then the new backing index start and end time settings are blindly set to $now-2h to $now+2h. The backing indices of the existing tsdb data stream may overlap with this, or not at all. Directly re-indexing from the old tsdb data stream into the new tsdb data stream only works for documents that have a timestamp that matches with $now-2h to $now+2h.

Reindex a tsdb data stream should be made easier. Reindexing a tsdb data stream should be similar to as reindex a regular data stream.

Maybe under the hood the reindex api should understand the layout of the different backing indices of the source backing indices and their respective start and end times and automatically create tsdb data stream with that in mind.

elasticsearchmachine commented 10 months ago

Pinging @elastic/es-analytics-geo (Team:Analytics)

elasticsearchmachine commented 3 months ago

Pinging @elastic/es-storage-engine (Team:StorageEngine)