Support automatic rollover of indices

[andrejvanderzee]

Is your feature request related to a problem? Please describe. Opensearch supports automatic rollover of indices based on conditions such as data size. When Opensearch rolls over it creates a new index, but the output plugin keeps writing to the first index:

logs-000002 | -Wr_D7ckROiCwK-Z9pbjDA | Hot/Warm | Hot Warm Delete | hot | Rollover | Pending rollover of index [index=logs-000002] | Running
logs-000001 | Nu7aIKWdSQ-PzXSLQ_T5oQ | Hot/Warm | Hot Warm Delete | hot | Transition | Evaluating transition conditions [index=logs-000001] | Running

In the above example logs-000001 keeps growing and logs-000002 stays empty.

Describe the solution you'd like After Opensearch rolls the index, the output plugin should write to new index logs-000002.

Additional context This is the output plugin config:

opensearch {
    auth_type => {
      type => "aws_iam"
      region => "${AWS_REGION}"
      role_arn => "${AWS_ROLE_ARN}"
      web_identity_token_file => "${AWS_WEB_IDENTITY_TOKEN_FILE}"
    }
    hosts => ["https://some.domain.com:443"]
    index => "logs-000001"
    manage_template => false
    template_name => "logs"
    legacy_template => false
}

It looks like the elasticsearch output plugin does support this via enabling ILM: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html#plugins-outputs-elasticsearch-ilm

[andrejvanderzee]

Related to #151

[andrejvanderzee]

It looks like I have misunderstood the index aliases. The output plugin should be configured like this:

opensearch {
    auth_type => {
      type => "aws_iam"
      region => "${AWS_REGION}"
      role_arn => "${AWS_ROLE_ARN}"
      web_identity_token_file => "${AWS_WEB_IDENTITY_TOKEN_FILE}"
    }
    hosts => ["https://some.domain.com:443"]
    index => "logs"
    manage_template => false
}