CCR auto_follow to follow both newly created and existing indices

Leaf-Lin commented 2 years ago

Description

When users creates a ccr auto_follow pattern, users may intuitively expect all indices matching the pattern to be replicated on the follower. Unfortunately, as CCR auto_follow API works as a separate process that picks up new indices to follow ^1, one requires to additional steps to replicate existing indices:

use individual _ccr/follow to follow existing indices or
use the snapshot to restore existing indices on the follower.

It would be great if ccr auto_follow can replicate both newly created and existing non-searchable snapshot indices. This is to simplify the recovery after losing a cluster when using CCR.

User Story I -- set up new cluster:

Assuming a cluster is set up without CCR and has been running for a long time containing some data. If users wish to set up another cluster to replicate the existing cluster, what are the steps required to do so?

Presently, users need to implement complex logic (either combining restoring from snapshots and reinstalling auto-follow, or iteratively starting replication of existing shards).
Ideally, users can simply use some regexp to replicate certain index patterns without caring which ones are newly created vs existing.

User Story II -- disaster recovery:

Assuming two clusters are set up with CCR. When one cluster (production) is lost, the other cluster (backup) can take over.

Presently, all follower indices on the backup cluster are required to be converted to regular indices using pause_follow and unfollow. Once the production cluster is back, more steps are required to replicate both new and existing indices from the backup cluster.
Ideally, the recovery steps can be simplified. Elasticsearch should be able to calculate the differences in two clusters and only replicate missing data without users manually promoting, following, unfollowing ccr steps.

Linking to a related issue on restored [^2] scenario where it's not clear if auto_follow should be performed.

[^2]: https://github.com/elastic/elasticsearch/issues/87055 raise the discussion on whether restored indices should be automatically replicated by auto follow patterns.

elasticsearchmachine commented 2 years ago

Pinging @elastic/es-distributed (Team:Distributed)

Janbong commented 7 months ago

Any discussion on this one? I am in a similar situation where a 2 cluster setup has issues replicating because the leader cluster creates the indices right before the second cluster comes up. This results in the indices not being replicated by the auto-follower as it only auto-follows newly created indices (after the auto-follower pattern was added).

Thanks in advance!

dbrimley commented 3 weeks ago

elastic / elasticsearch