[BUG] Hot/warm architecture not working in 1.0 - indexes are not being moved

[sandervandegeijn]

Describe the bug Indexes are not moving from hot to warm nodes with OpenSearch 1.0

To Reproduce I've spun up a cluster with hot and warm nodes, terraform:

 env {
            name  = "node.attr.temp"
            value = "warm"
          }

Cluster is green:

 "cluster_name" : "opensearch",
  "status" : "green",

Environment variables are used by the nodes, temp is set: nodesinfo.zip

Setting required temp to warm:

PUT logstash-ikev2-2021.08.17/_settings
{
  "settings": {
    "index.routing.allocation.require.temp": "warm"
  }
}

GET logstash-ikev2-2021.08.17

 "settings" : {
      "index" : {
        "routing" : {
          "allocation" : {
            "require" : {
              "temp" : "warm",
              "box_type" : "warm"
            }
          }
        },
        "number_of_shards" : "1",
        "provided_name" : "logstash-ikev2-2021.08.17",
        "creation_date" : "1629184713520",
        "number_of_replicas" : "0",
        "uuid" : "6eQjhKeVSu-xUGGVI4dvDA",
        "version" : {
          "created" : "135217827"
        }
      }
    }

Does not move index to warm node (waited for hours):

Also ISM policy did not do it either:

{
    "policy_id": "logstash",
    "description": "Logstash",
    "last_updated_time": 1629466956497,
    "schema_version": 1,
    "error_notification": null,
    "default_state": "hot",
    "states": [
        {
            "name": "hot",
            "actions": [
                {
                    "replica_count": {
                        "number_of_replicas": 0
                    }
                },
                {
                    "allocation": {
                        "require": {
                            "box_type": "hot"
                        },
                        "wait_for": false
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "warm",
                    "conditions": {
                        "min_index_age": "3d"
                    }
                }
            ]
        },
        {
            "name": "warm",
            "actions": [
                {
                    "replica_count": {
                        "number_of_replicas": 0
                    }
                },
                {
                    "allocation": {
                        "require": {
                            "box_type": "warm",
                            "temp": "warm"
                        },
                        "wait_for": false
                    }
                }
            ],
            "transitions": [
                {
                    "state_name": "delete",
                    "conditions": {
                        "min_index_age": "14d"
                    }
                }
            ]
        },
        {
            "name": "delete",
            "actions": [
                {
                    "delete": {}
                }
            ],
            "transitions": []
        }
    ],
    "ism_template": {
        "index_patterns": [
            "logstash-*"
        ],
        "priority": 100,
        "last_updated_time": 1615931626029
    }
}

Expected behavior After setting the temp to warm cluster moves the shards to the warm nodes instead of hot.

Plugins Default plugins from 1.0.0 docker container

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information): Docker 1.0.0 image on kubernetes

Additional context

[sandervandegeijn]

Maybe I caused this myself by:

PUT _index_template/logstash-hot
{
  "index_patterns": [
    "logstash-*"
  ],
  "template": {
    "settings": {
      "number_of_shards": 1,
      "number_of_replicas": 0,
      "index.routing.allocation.require.temp" : "hot"
    }
  }
}

Expected the ISM to override, but maybe it does not. Testing.. edit:

Index template is necessary to prioritize hot nodes, without the temp setting it will distribute to the warm nodes as well. Manually setting the temp setting to hot/warm forces relocation so the mechanism itself is working. Will monitor the ISM actions the coming days, but it seems that there is something wrong there.

[sandervandegeijn]

I finally figured it out, came across a bug it seems ( https://github.com/opensearch-project/index-management/issues/130 ), but the hot/warm/cold setup does work. My error was that this page:

https://opensearch.org/docs/opensearch/cluster/

suggested using index.routing.allocation.require.temp and node.attr.temp while this page

https://opensearch.org/docs/im-plugin/ism/policies/

subsection allocation suggested box_type setting. If you have both present, OS does not move the indexes. Now that I understand the mechanism it is logical but for an inexperienced user this can be difficult. Will submit an issue for the docs instead.