How to reproduce this error

[TomonoriSoejima]

"message": "[.ds-itm2-it360a10027-2023.12.21-000480] lifecycle action [migrate] waiting for [1] shards to be moved to the [data_warm] tier (tier migration preference configuration is [data_warm, data_hot])",

PUT /_cluster/settings
{
    "persistent" : {
        "indices.lifecycle.poll_interval": "2s"
    }
}

PUT _index_template/tomo
{
  "index_patterns": ["tomo*"],
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "mk2_ilm"
        },
        "routing": {
          "allocation": {
            "require": {
              "data": "hot"
            }
          }
        },
        "number_of_shards": "1",
        "auto_expand_replicas": "0-1"
      }
    },
    "aliases": {},
    "mappings": {}
  }
}

PUT _ilm/policy/mk2_ilm
{
  "policy": {
    "phases": {
      "warm": {
        "min_age": "1s",
        "actions": {
          "forcemerge": {
            "max_num_segments": 1,
            "index_codec": "best_compression"
          },
          "readonly": {},
          "set_priority": {
            "priority": 50
          }
        }
      },
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "10s"
          },
          "set_priority": {
            "priority": 100
          }
        }
      }
    }
  }
}

DELETE tomo*

PUT /tomo-1
{
  "aliases": {
    "tomo": {
      "is_write_index": true
    }
  },
  "settings": {
    "index.lifecycle.name": "mk2_ilm",
    "index.lifecycle.rollover_alias": "tomo"
  }
}

POST tomo-1/_doc/1
{
  "field1": "newvalue"
}

GET tomo-1/_ilm/explain

GET _cat/shards/tomo-1

These commands are part of managing an Elasticsearch cluster and its indices using the Index Lifecycle Management (ILM) feature, which allows for the automation of various index management tasks based on policies that you define. Let's break down what each part of the script is intended to do:

1. Update Cluster Settings: The first PUT request updates the cluster settings to change the indices.lifecycle.poll_interval to 2s. This setting configures the interval at which the ILM (Index Lifecycle Management) policy is polled for updates and checks. By setting it to 2 seconds, you are making the ILM more responsive to lifecycle changes at the expense of potentially higher overhead on the cluster management tasks.

2. Create an Index Template: The PUT _index_template/tomo request creates an index template named tomo for indices matching the pattern tomo*. This template specifies that any new index matching the pattern should:

   • Be allocated to nodes with the data attribute set to hot, indicating these indices should reside on nodes optimized for high I/O operations typical of the "hot" phase, where data is frequently accessed and updated.
   • Have a single shard and auto-expand replicas from 0 to 1, allowing for dynamic adjustment of replicas based on the number of available nodes.
   • Use the mk2_ilm ILM policy for managing its lifecycle.

3. Create an ILM Policy: The PUT _ilm/policy/mk2_ilm request creates an ILM policy named mk2_ilm that defines two phases for managing the lifecycle of an index:

   • Hot Phase: Starts immediately (min_age: "0ms") and includes a rollover action that triggers when the index is older than 10s, along with setting the index priority to 100.
   • Warm Phase: Starts after 1s from index creation, and includes actions to force merge segments for better compression, set the index to read-only, and adjust its priority to 50.

4. Delete Existing Indices: The DELETE tomo* request deletes any indices that match the pattern tomo* to clean up the environment before creating a new index.

5. Create an Index: The PUT /tomo-1 request creates an index named tomo-1 with specific settings for ILM and an alias tomo that marks it as the write index, which means it's the current index to which new documents should be written.

6. Index a Document: The POST tomo-1/_doc/1 request indexes a new document with an ID of 1 into the tomo-1 index.

7. Check ILM Status: The GET tomo-1/_ilm/explain request retrieves information about the current state of the tomo-1 index in the context of its ILM policy.

8. Check Shard Allocation: The GET _cat/shards/tomo-1 request checks the allocation and status of the shards for the tomo-1 index.

The error message from GET tomo-1/_ilm/explain indicates that the index is in the process of a lifecycle action (migrate) and is waiting for 1 shard to be moved to the data_warm tier. This situation arises because the lifecycle policy includes a transition to a warmer tier (less frequently accessed data, but still needs to be available quickly when accessed) after certain conditions are met (in this case, 1s after creation). The error suggests that the cluster is either unable to move the shard to a node designated as data_warm due to the lack of such nodes or another configuration issue preventing the migration.

To resolve this issue, ensure that your Elasticsearch cluster has nodes with the data_warm role and that these nodes are capable of receiving shards. You may also need to review the cluster's allocation settings and the configuration of the nodes to ensure they align with your intended data tiering strategy.

[TomonoriSoejima]

PUT /tomo-1/_settings
{
  "index.routing.allocation.require.data": null
}

allowed the index to move to warm node.