[BUG] Cross Cluster Search is not returning partial result in case of skip_unavailable is true

[nisgoel-amazon]

Describe the bug

Cross Cluster Search is not returning partial result in case of skip_unavailable is true. This is happening in the case when one of the destination node is replaced with another node.

For running cross cluster search, we have to add destination cluster seed nodes in source cluster. So when cluster connection alias seed node setting is updated with new node of destination cluster. Cross cluster search(CCS) started failing even skip unavailable flag is true in source cluster setting.

Doc for cross cluster search setup.

Related component

Search

To Reproduce

1. Create 2 clusters using cross cluster replication plugin setup
2. Add following setting in source cluster "cluster.remote.remote-alias.skip_unavailable" : "true"
3. Add below data in destination cluster

   curl -k -XPOST -u $CREDS "http://$DESTINATION/_bulk?pretty" -H 'Content-Type: application/json' -d '
   { "index": { "_index": "test-index", "_id": "1" } }
   { "description": "new ice cream", "name": "test1", "created": "2024-03-25T06:08:14Z" }
   { "index": { "_index": "test-index", "_id": "2" } }
   { "description": "new croissant", "name": "test2", "created": "2024-03-25T06:08:16Z" }
   { "index": { "_index": "test-index", "_id": "3" } }
   { "description": "new tennis shoe", "name": "test3", "created": "2024-03-25T06:08:18Z" }
   { "index": { "_index": "test-index", "_id": "4" } }
   { "description": "new hightop", "name": "test4", "created": "2024-03-25T06:08:19Z" }
   ' 

4. Do a cross cluster search with following query

   curl -XGET -u $CREDS "https://$SOURCE/remote-alias:test-index/_search?pretty=true" \
   -H 'Content-Type: application/json' \
   -d '
   {
   "query": {
       "match_all": {}
   },
   "from": 1000,
   "size": 1,
   "sort": {
       "created": {
           "order": "asc"
       }
   }
   }'

5. Above query should pass with 200 and in response we should get skipped result as 1.
6. Now update the seed node setting of source cluster "cluster.remote.remote-alias.seeds"
7. Now again search with query mentioned in step 4. This time we get the response as 400

Expected behavior

CCS request should be return 200 as response with skipped as 1.

Additional Details

Plugins Please list all plugins currently enabled.

Screenshots If applicable, add screenshots to help explain your problem.

Host/Environment (please complete the following information):

• OS: [e.g. iOS]
• Version [e.g. 22]

Additional context Add any other context about the problem here.

[nisgoel-amazon]

@dblock This issue is in opensearch not in replication package. skip_unavailable cluster setting flag is part of core. This setting is not getting recognized when we update seed_nodes cluster setting.

Can you please move it back to opensearch. Also you can assign it to me. I will raise the PR with the fix.