Support multiple aliases in rollover action

[Tedderouni]

Please add support for rolling over multiple aliases in a single rollover action. In my environment, there are many rollover aliases (with the number likely to increase over time), so it's impractical to define a separate action for each one - it would become a long list I'd have to maintain manually every time new ones are added, and it would still be a separate Curator run for each alias, which would take too much time due to the volume of data.

Ideally I'd like to be able to define a single action that can match all the rollover aliases - maybe using a pattern filter, or just supporting wildcards or regexes in the "name" setting for the rollover action. In my case the alias names all end with "_rollover", so any way to match those would be perfect, and then proceed to apply the rollover conditions to each match.

I'm using Curator 5.5.4 with Elasticsearch 6.2.4.

[cdenneen]

I have the same thing... a few different *-write aliases that get associated with the index and rollover seems to only move one of them and not the other 2 causing a manual of those 2 to be removed from previous and add to the now rolled over index

@Tedderouni have you had any luck finding a solution?

For example: myindex-000001

aliases are

app1-write app2-write app3-write

curator would only move the app1-write to myindex-000002 and app2-write and app3-write will be left on myindex-000001. But if you add all 3 to curator action then it could potentially end up in a worse situation with:

myindex-000001 - no alias myindex-000002 - app1-write myindex-000003 - app2-write myindex-000004 - app3-write

when in fact it just errors with the following:

2019-01-31 17:40:10,131 ERROR Failed to complete action: rollover. <class 'elasticsearch.exceptions.RequestError'>: TransportError(400, 'resource_already_exists_exception', 'index [myindex-000002/Bkb72QrZSLyCJf4HivBUZQ] already exists')

[Tedderouni]

@cdenneen I didn't find a way to do this in Curator, so I just wrote up a bash script as a temporary hack to emulate what Curator does, until it supports this.

Here's what I use, feel free to adapt it to your environment. In my environment, I define $ESUSER and $ESPASS in /etc/sysconfig/elasticsearch, and my alias names end with _rollover, so you'll need to update get_rollover_aliases() to match yours. I run this out of cron. And of course if you use this, make sure to run it as a dry run first (in the roll_over() function) :)

#!/bin/bash

. /etc/sysconfig/elasticsearch

ESHOST="localhost"
ESNODE="${ESHOST}:9299" # Must be a master node

MAX_AGE="31d"
MAX_DOCS="2000000000"
MAX_SIZE="1tb"

function esapi-get() {
    echo $(curl -s -XGET -u ${ESUSER}:${ESPASS} -H'Content-Type: application/json' http://${ESNODE}/$1)
}

function esapi-post() {
    echo $(curl -s -XPOST -u ${ESUSER}:${ESPASS} -d "$2" -H'Content-Type: application/json' http://${ESNODE}/$1)
}

function exit_if_not_master() {
    master_ip=$(get_master_ip)
    my_ip=$(hostname -i)

    if [ "$my_ip" != "$master_ip" ]; then
        log_write "This node is not the master. Exiting."
        exit 1
    else
        log_write "This node is the master. Continuing."
    fi
}

function get_rollover_aliases() {
    echo $(esapi-get "_cat/aliases/*_rollover?h=a")
}

function get_master_ip() {
    echo $(esapi-get "_cat/master?h=ip")
}

function log_write() {
    DATE=$(date +%Y-%m-%d\ %H:%M:%S)
    echo "$DATE $1"
}

function roll_over_all_aliases() {
    for alias in ${alias_list}; do
        log_write "Attempting to roll over $alias: "
        roll_over "$alias"
    done
}

function roll_over() {
    esapi-post "$1/_rollover?dry_run" '
    #esapi-post "$1/_rollover" '
        {
            "conditions": {
                "max_age":   "'${MAX_AGE}'",
                "max_docs":  '${MAX_DOCS}',
                "max_size":  "'${MAX_SIZE}'"
            }
        }'
}

exit_if_not_master
alias_list=$(get_rollover_aliases)
roll_over_all_aliases

[Tedderouni]

@cdenneen Also I should mention you might be able to do this with the new Index Lifecycle Management feature, which was just released with 6.6. I haven't had a chance to look into it yet, but if it works for this use case, it would be preferable over a temporary hack. https://www.elastic.co/guide/en/elasticsearch/reference/current/index-lifecycle-management.html

[frittentheke]

@Tedderouni I was thinking about the same thing you mention in your initial feature request:

A "scriptable" rollover action, allowing a rollover strategy to be applied all my indices (containing logs).

i.e. look at all indices (i.e. with a certain date postfix, or naming regex) and apply a rollover call checking for the conditions just like for a single index. But instead allow the rollover alias to "use" the initial index name to create some sort of "mapping" between each and every index and its roll over alias.

This would allow running indices with i.e. shard: 1 and have curator check each index every so often if a rollover would make sense as a certain size or document count has been reached. This is like "auto-matching" the number of resulting shards for each index.

But currently there seems to be no action that does allow i.e. the results of a filter to be used as option for the action that is just applied.