[BUG] Restore snapshot on a production index while configuring replication between production and dr

[zalseryani]

Describe the bug

I am assuming the following scenario:

• I have index at production, and created a replication rule for it on dr (follower index)
• configured a snapshot management on production index.
• due to any reason, I needed to restore a snapshot on the same production index, so I have closed the index, restore the snapshot, and when I tried to add data on the index, and then check the replication rule status, it shows the below error

  {
  "status": "PAUSED",
  "reason": """AutoPaused:  + [[scalability-test-all-messages][2] - org.opensearch.index.seqno.RetentionLeaseNotFoundException - "retention lease with ID [replication:opensearch-cluster:UjLLAqK6RQ-p6ccqh0KPSA:[scalability-test-all-messages][2]] not found"], [[scalability-test-all-messages][0] - org.opensearch.index.seqno.RetentionLeaseNotFoundException - "retention lease with ID [replication:opensearch-cluster:UjLLAqK6RQ-p6ccqh0KPSA:[scalability-test-all-messages][0]] not found"], """,
  "leader_alias": "my-connection-alias-vs",
  "leader_index": "scalability-test-all-messages",
  "follower_index": "scalability-test-all-messages"
  }

Is there a best practice for such a case, because it might occur at the production site I appreciate your time and support.

Related component

Indexing:Replication

To Reproduce

• mentioned in description

Expected behavior

• Continuing the replication as long as the data on prod index is the same as the data on dr index (follower), noting that when restoring a snapshot, the follower index will have a checkpoint number that is higher than the leader index at production site after restoring a snapshot.
• Also, restoring the data into a new index is not a viable/practical solution because if the snapshot is large, (since restoring it into a new index) , the replication needs to replicate the data all over again from the prod index into the dr index.

Additional Details

• opensearch/opensearch-dashboards version 2.11.1
• deployed on kubernetes cluster

[ankitkala]

With snapshot restore, follower can't track the delta changes. You'll need to delete the follower index & restart the replication.