[BUG] Inconsistent behavior on POST _aliases when two opposing actions are operating on the same index/alias pair

[theahura]

Describe the bug

I basically ran into https://github.com/elastic/elasticsearch/issues/27689 almost exactly.

# 1) Put an index
PUT some-concrete-index

# 2) no alias exists yet
GET some-concrete-index/_alias

# 3) add an alias, then remove it
POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "some-concrete-index",
        "alias": "oci-cmdb_service_members"
      }
    },
    {
      "remove": {
        "index": "some-concrete-index",
        "alias": "oci-cmdb_service_members"
      }
    }
  ]
}

# 4) Order shouldn't matter, and it does not seem to, as the alias now exists
GET some-concrete-index/_alias

#5) Execute the same operation again:
POST /_aliases
{
  "actions": [
    {
      "add": {
        "index": "some-concrete-index",
        "alias": "oci-cmdb_service_members"
      }
    },
    {
      "remove": {
        "index": "some-concrete-index",
        "alias": "oci-cmdb_service_members"
      }
    }
  ]
}

#6) The alias is removed now
GET some-concrete-index/_alias

Related component

Other

To Reproduce

See above.

Expected behavior

I expected the actions to resolve in order.

Additional Details

This is running on aws opensearch serverless.

[theahura]

To be more explicit, I got logs from my own system. Here are the actions:

Making the following alias modifications: [
  {
    remove: { index: 'indexedinstance', alias: 'current-published-instance' }
  },
  {
    add: {
      index: 'indexedinstance',
      alias: 'current-published-instance',
      filter: undefined,
      is_write_index: true
    }
  }
]

and then when I run

 30   // Get the mappings.                                                           
 31   const mappings = await client.cat.aliases();                                   
 32   console.log(JSON.stringify(mappings, null, 2));  

I get

 {                                                                                                                                                                                      
  "body": "",                                                                                                                                                                          
  "statusCode": 200,                         
  "headers": {                               
    "content-type": "text/plain; charset=UTF-8",                                           
    "content-length": "0",                   
    "x-envoy-upstream-service-time": "17",                                                 
    "date": "Sat, 22 Jun 2024 03:05:22 GMT",                                               
    "server": "aoss-amazon-m",                                                             
    "x-request-id": "865cd86d-77ce-9a30-a215-3fbbc77d49d2"                                 
  },                                         
  "meta": {                                  
    "context": null,                         
    "request": {                             
      "params": {                            
        "method": "GET",                     
        "path": "/_cat/aliases",             
        "body": null,                        
        "querystring": "",                   
        "headers": {                         
          "user-agent": "opensearch-js/2.3.1 (linux 5.15.0-1041-aws-x64; Node.js v17.9.1)"
        },                                                                                 
        "timeout": 30000                     
      },                                     
      "options": {},                         
      "id": 1                                
    },                                       
    "name": "opensearch-js",                 
...

[dblock]

Looks like a bug. @theahura Want to try and write a YAML REST test for it? https://github.com/opensearch-project/OpenSearch/blob/main/TESTING.md#testing-the-rest-layer

[theahura]

Maybe something like this?

---
setup:
  - do:
      indices.create:
        index: test_index
        body:
          settings:
            number_of_shards: 1
            number_of_replicas: 0

  - do:
      indices.update_aliases:
        body:
          actions:
            - add:
                index: test_index
                alias: alias1

---
- do:
    indices.update_aliases:
      body:
        actions:
          - remove:
              index: test_index
              alias: alias1
          - add:
              index: test_index
              alias: alias1

- do:
    indices.get_alias:
      name: alias1
  - match: { test_index.aliases.alias1: {} }

---
teardown:
  - do:
      indices.delete:
        index: test_index

Note that I didn't get around to actually testing this though, also not really sure if the syntax is correct

[peternied]

[Triage - attendees 1 2 3] @theahura Thanks for writing out that test case, could you make a pull request to add the test - maybe build off that to fix the issue?

[theahura]

How do I add the test? I'm not sure it's actually valid fwiw, and I don't have a ton of spare time so it may take me a bit before I am able to add the test in

[peternied]

I found a PR [1] that added a feature and tests - should give you some pointers on where the code goes, good luck

• [1] https://github.com/opensearch-project/OpenSearch/pull/13461