[Feature Request] Shrink action must be work as Rollover action with data stream index

[disaster37]

Is your feature request related to a problem? Please describe

When we use Opensearch stack to ingest logs, it's normal to use data stream index. More over, It's normal on this use case to have hot / warm / delete architecture. We keep 24h of logs on hot node, then we keep them in long time (30d) on warm node.

Because of logs are not always searched on warm node, we shrink index to avoid ta have to many shard on warm nodes. So on ISM (index state management) policy we have shrink step on warm phase. But this step not working as expected with data stream index. When it create new index (ths shrunken index), it break the lineage with data stream index. The new index is no more part of data stream index. And it break all the next step of ISM because of the new index is outdoor of the current policy.

Describe the solution you'd like

We expect that this ISM policy work out of the box.

{
    "id": "policy-log",
    "policy": {
        "policy_id": "policy-log",
        "description": "Policy for logs index",
        "default_state": "hot",
        "states": [
            {
                "name": "hot",
                "actions": [
                    {
                        "rollover": {
                            "min_index_age": "1d",
                            "min_primary_shard_size": "50gb",
                            "copy_alias": false
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "warm"
                    }
                ]
            },
            {
                "name": "warm",
                "actions": [
                    {
                        "allocation": {
                            "require": {
                                "temp": "warm"
                            },
                            "include": {},
                            "exclude": {},
                            "wait_for": false
                        }
                    },
                    {
                        "index_priority": {
                            "priority": 50
                        }
                    },
                    {
                        "shrink": {
                            "percentage_of_source_shards": 0.5
                        }
                    },
                    {
                        "force_merge": {
                            "max_num_segments": 1
                        }
                    }
                ],
                "transitions": [
                    {
                        "state_name": "delete",
                        "conditions": {
                            "min_index_age": "30d"
                        }
                    }
                ]
            },
            {
                "name": "delete",
                "actions": [
                    {
                        "delete": {}
                    }
                ]
            }
        ],
        "ism_template": [
            {
                "index_patterns": [
                    "logs-*"
                ],
                "priority": 100
            }
        ]
    }
}

As human understanding, we should it work like this:

• Hot phase when new data stream index is created or new rollover index
  • When index exist during more than 1 day or one primary shard is biggest than 50Gb, we create new index and for previous index, we go on warm phase (this one work fine)
• Warm phase after rollover
  • It move all shard / replica of previous index on warm nodes (this one work fine)
  • It change de index priority of previous index to 50 (this one work fine)
  • It shrink the previous index to have 2 primary shard from 4. We expect the new index that reflect the shrunken action continue to be a part of datastream index. We also expect that all next step will be applied on shrunken index. And the source of shrunken index will be automatically deleted. (this one not work)
  • it force merge the shrunken index
  • After 30d, the shrunken index need to go on delete phase
• Delete phase
  • It delete the shrinken index

It can be usefull to orchestrate the steps from phase to the best order. Maybee the best order is not the order provided by user. Maybee it's better to start with force merge and finish by allocation ... Or maybee force merge need to be run after shrink index and allocate it on the right node ?

Related component

Other

Describe alternatives you've considered

No response

Additional context

I think shrink step must be work like rollover step with data stream index. After look a little bit the code, I think the problem is not on ISM plugin, but on Opensearch core. Especially on server/src/main/java/org/opensearch/action/admin/indices/shrink package.

[andrross]

[Triage - attendees 1 2 3 4 5]

Thanks for filing. This looks like a reasonable request. We will need the ability to add indexes to a data stream as a prerequisite (see opensearch-project/OpenSearch#8271).

[vikasvb90]

@opensearch-project/admin Please transfer this issue to https://github.com/opensearch-project/index-management/ repo.

[dblock]

[Catch All Triage - 1, 2]

[disaster37]

I have opened on another issue on main opensearch project because of the main part of issue seems to be on Opensearch core project.

https://github.com/opensearch-project/OpenSearch/issues/16063