Reporting index managed by ILM policy got exception when rollover action is added in the default policy kibana-reporting

[hubbleview]

Kibana version: 7.16.3

Describe the bug:

Since https://github.com/elastic/kibana/pull/100130 and https://github.com/elastic/kibana/pull/103850, by default, reporting index will be applied ILM policy kibana-reporting, which has the default policy as below.

{
  "kibana-reporting" : {
    "version" : 1,
    "modified_date" : "2022-01-18T22:20:57.531Z",
    "policy" : {
      "phases" : {
        "hot" : {
          "min_age" : "0ms",
          "actions" : { }
        }
      }
    }
  }
}

And, this ILM policy allows to be customized.

When adding rollover action in the policy, which is common and reasonable request, these reporting indices start to see errors about ILM, like below.

{
  "indices" : {
    ".reporting-2022-01-16" : {
      "index" : ".reporting-2022-01-16",
      "managed" : true,
      "policy" : "kibana-reporting",
      "lifecycle_date_millis" : 1642551663536,
      "age" : "16.46h",
      "phase" : "hot",
      "phase_time_millis" : 1642610052954,
      "action" : "rollover",
      "action_time_millis" : 1642551663599,
      "step" : "ERROR",
      "step_time_millis" : 1642610652848,
      "failed_step" : "check-rollover-ready",
      "is_auto_retryable_error" : true,
      "failed_step_retry_count" : 49,
      "step_info" : {
        "type" : "illegal_argument_exception",
        "reason" : "setting [index.lifecycle.rollover_alias] for index [.reporting-2022-01-16] is empty or not defined",
        "stack_trace" : """java.lang.IllegalArgumentException: setting [index.lifecycle.rollover_alias] for index [.reporting-2022-01-16] is empty or not defined
    at org.elasticsearch.xpack.core.ilm.WaitForRolloverReadyStep.evaluateCondition(WaitForRolloverReadyStep.java:89)
    at org.elasticsearch.xpack.ilm.IndexLifecycleRunner.runPeriodicStep(IndexLifecycleRunner.java:226)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggerPolicies(IndexLifecycleService.java:408)
    at org.elasticsearch.xpack.ilm.IndexLifecycleService.triggered(IndexLifecycleService.java:339)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine.notifyListeners(SchedulerEngine.java:186)
    at org.elasticsearch.xpack.core.scheduler.SchedulerEngine$ActiveSchedule.run(SchedulerEngine.java:220)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
    at java.base/java.lang.Thread.run(Thread.java:833)
"""
      },
      "phase_execution" : {
        "policy" : "kibana-reporting",
        "phase_definition" : {
          "min_age" : "0ms",
          "actions" : {
            "rollover" : {
              "max_primary_shard_size" : "50gb",
              "max_age" : "1d"
            }
          }
        },
        "version" : 3,
        "modified_date_in_millis" : 1642551647174
      }
    }
  }
}

In above ILM explain output, we could see the updated policy kibana-reporting with rollover action.

The exception is because, the default reporting index name is .reporting-${date}, which doesn't match ILM rollover pattern. The initial index name should be like .reporting-${date}-000001 to allow rollover to happen.

Steps to reproduce:

1. New deployment of 7.16.3 ES/kibana.
2. Edit kibana-reporting policy to add rollover action.
3. Wait for rollover action taking place, then check GET .reporting-*/_ilm/explain output.

Expected behavior:

• Default reporting index name follows the ILM rollover pattern with number at the end.
• No above error happens when adding rollover action in ILM policy kibana-reporting.

[elasticmachine]

Pinging @elastic/kibana-reporting-services (Team:Reporting Services)

[hubbleview]

Or, migrating reporting index to data stream.

[elasticmachine]

Pinging @elastic/kibana-app-services (Team:AppServicesUx)

[tsullivan]

Or, migrating reporting index to data stream.

@hubbleview Reporting uses a data stream configuration as of #176022 but I'm not sure if it sufficiently takes care of this issue.