[FEATURE] SM to support deletion of snapshots created manually

[aggarwalShivani]

Is your feature request related to a problem? Snapshot Managment should allow creating an SM policy that only deletes snapshots created manually (not through SM policy).

Problem statement -

• I create snapshots for opensearch to backup its data. These snapshots should be created on demand by a user when required (at random times) and not based on any cron schedule.

• The name of the snapshot too is decided by the user and not known beforehand. The name is appended with the timestamp of snapshot creation.

• As a housekeeping activity, in order to control the number of snapshots retained in the system, we want to automate deletion of old snapshots from the system. For ex. The user could trigger snapshots n number of times in a week, but we want to enable a cron that runs once everyday and deletes any snapshots older than 7days. I do not want to use the snapshot management plugin to "create" snapshots, but only "delete" the snapshots.

However, on creating a policy like below, it failed due to missing configuration for creation.

curl -XPOST "http://indexsearch:9200/_plugins/_sm/policies/daily-policy-delete" -H 'Content-Type: application/json' -d'
> {
>   "description": "Daily snapshot policy",
>   "deletion": {
>     "schedule": {
>       "cron": {
>         "expression": "0 1 * * *",
>         "timezone": "UTC"
>       }
>     },
>     "condition": {
>       "max_age": "7d"
>     },
>     "time_limit": "1h"
>   },
>   "snapshot_config": {
>     "date_format": "yyyy.MM.dd-HHu:mm:ss",
>     "timezone": "UTC",
>     "indices": "*",
>     "repository": "is_backup",
>     "ignore_unavailable": "true",
>     "include_global_state": "false",
>     "metadata": {
>       "any_key": "any_value"
>     }
>   }
> }'
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Must provide the creation configuration."}],"type":"illegal_argument_exception","reason":"Must provide the creation configuration."},"status":400}

What alternatives have you considered? Elasticsearch-curator provides this feature and we have been using the same successfully. We now are trying to migrate to ISM plugin (in replacement of curator) and want to achieve similar use-case. https://www.elastic.co/guide/en/elasticsearch/client/curator/current/delete_snapshots.html

What solution would you like?

1. SM should allow creating an SM policy with only "deletion" configuration, and no creation.
2. SM policy should accept some filter/regex-pattern - to match name of the snapshots to be deleted.

For ex. If user creates snapshots by the name, os-snapshot- (For ex. os-snapshot-2023.07.19-10:15:12, os-snapshot-2023.07.19-12:10:10 etc), there should be a provision to create a policy to only "delete" snapshots older than 7d that match the pattern os-snapshot-*

[aggarwalShivani]

Hi, Any thoughts on this request? Is this being evaluated as a potential feature to be added in the plan?

[ikibo]

Hi @bowenlan-amzn Could U assign me to this ticket plz

[aggarwalShivani]

Hi @ikibo

For my own learning, I have been trying to understand the code and where changes would be needed for this feature request. Just a few questions/suggestions from my side on the planned implementation around snapshot selection -

1. How are you planning to select snapshots to be deleted? Currently, during deletion, it searches for the policy name in the snapshot metadata.

fun List<SnapshotInfo>.filterBySMPolicyInSnapshotMetadata(policyName: String): List<SnapshotInfo> {
    return filter { it.userMetadata()?.get(SM_TYPE) == policyName }
}

Ref - https://github.com/opensearch-project/index-management/blob/main/src/main/kotlin/org/opensearch/indexmanagement/snapshotmanagement/SMUtils.kt#L205

But snapshots created externally (through REST APIs and not ISM), would not have this field. I believe this function could be modified for delete-only policies, to not look for metadata for policy-name.

2. Are you also planning to add a new field for some regex pattern to match snapshot-name to be deleted for policy? Or the matching will continue to be as done currently - i.e. snapshots starting with << policy-name >> would be marked for deletion?

3. Also, what is the plan for the 'creation' field in the SM policy spec? Would it made optional? Or some new end point would be created to have delete-only policies?

Would request maintainers too for their inputs.

[bowenlan-amzn]

@aggarwalShivani Thanks for spending time on this!

1,2 I think we can introduce new field under deletion in create policy API This new field contains the delete snapshot name pattern and its belonging repository.

3 we can make the creation field optional, no need to introduce new API.

[wilalalee]

In my case, we create snapshots with ISM, and expect to do housekeeping with SM.

[aggarwalShivani]

In my case, we create snapshots with ISM, and expect to do housekeeping with SM.

@wilalalee Yes, so this feature would ideally allow both the use-cases:

• ISM to create and delete snapshots
• ISM to have a provision to delete snapshots created outside of ISM

[aggarwalShivani]

Hi @ikibo, are you still working on this feature?