support for indices with sequential numbering

[eye1]

I have been trying to set ILM policy for the logs generated via this plugin.

Currently, we use indexSuffixPattern [YYYY.MM]. In this format, it is not possible to do ILM policy well.

Is there a way to use sequence notation with -0000001?

[vanthome]

Hmm, you expect the index number then to be incremented automatically? If not, just use a static index name (w/o suffix).

[eye1]

I have found a way it seems so. Still, need to deploy it to production but the initial test has shown success.

First, it's needed to bootstrap the initial index manually

PUT logsIndexName-000001
{
  "aliases": {
    "logsIndexName": {
      "is_write_index": true
    }
  }
}

Then instead of using parameters: indexPrefix and indexSuffixPattern

I used

index [none | when dataStream is true, logs-app-default] The index to be used. This option is mutually exclusive with indexPrefix.

Now the logs are going to the sequenced index and ILM policy shall take care of managing the maximum size and moving old indices to cheaper instances

[vanthome]

yes that's what I had in mind,

[vanthome]

So we can close this?

[eye1]

I have deployed to production and all looks good. I am though getting a warning on regular basis

{"errorMessage":"illegal_argument_exception: [illegal_argument_exception] Reason: index template [template_logs] has index patterns [logs-app-default*] matching patterns from existing templates [template_v2-hc-cloud-logs-production] with patterns (template_v2-hc-cloud-logs-production => [logs-app-default*]) that have the same priority [200], multiple index templates may not match during index creation, please use a different priority"}

Any idea why?

[vanthome]

well you have obviously two competing index patterns. Maybe you had one already or created earlier? If you can, I suggest delete the whole index and start up the system again. The logger will create a pattern automatcially then.

[eye1]

I am not sure about this, but I believe both templates are created by the plugin itself.

The one that I marked on the screenshot is maybe used for the data streams approach. I was thinking to delete it as I do not use streams. Not sure of the consequences and if that is the right approach.