[Fleet] Trigger rollover when `time_series_dimension` mappings change between template versions

kpollich commented 3 weeks ago

When a new index template is created for an integration and the set of time_series_dimension mappings present in that template has changed from the previous template version, Fleet should trigger a rollover. This rollover will prevent data loss where Elasticsearch can potentially throw mapping errors on the new dimension field without a rollover.

elasticmachine commented 3 weeks ago

Pinging @elastic/fleet (Team:Fleet)

felixbarny commented 3 weeks ago

the set of time_series_dimension mappings present in that template has changed from the previous template version

What's particularly relevant are changes in dynamic templates. For example, changing the mapping parameter time_series_dimension to true on labels.*. If labels.* fields have already been mapped (and thus use the default value of false for time_series_dimension), a rollover is required.

When changing the time_series_dimension parameter on a specific field mapping, I don't expect there to be a gap. That's because ES would fail the mapping update, complaining that the immutable mapping parameter can't be changed, which Fleet takes as a signal to roll over the data stream.

juliaElastic commented 1 week ago

I reproduced this locally with installing prometheus 1.16.0 and upgrading to 1.17.0 which had labels.* set to dimension in the remote_write datastream (change here). I can see that the rollover is needed to update the mappings.

POST kbn:/api/fleet/epm/packages/prometheus/1.16.0
{
  "force": true
}

# index doc to create data stream
POST metrics-prometheus.remote_write-default/_doc
{
  "@timestamp": "2024-09-04T10:34:59.000Z",
  "prometheus" : {"labels": {"test": "label1"}},
  "agent": {
    "id": "agent1"
  },
  "cloud": {
    "account": {
      "id": "1234"
    },
    "availability_zone": "eu",
    "instance": {
      "id": "1234"
    },
    "provider": "aws",
    "region": "eu"
  }
}

POST kbn:/api/fleet/epm/packages/prometheus/1.17.0
{
  "force": true
}

# 1.16.0
GET metrics-prometheus.remote_write-default/_mapping

  ".ds-metrics-prometheus.remote_write-default-2024.09.04-000001": {
...
        "prometheus": {
          "dynamic": "true",
          "properties": {
            "labels": {
              "dynamic": "true",
              "properties": {
                "test": {
                  "type": "keyword"
                }
              }
            },

# 1.17.0

  ".ds-metrics-prometheus.remote_write-default-2024.09.04-000002": {
...
         "prometheus": {
          "dynamic": "true",
          "properties": {
            "labels": {
              "dynamic": "true",
              "properties": {
                "test": {
                  "type": "keyword",
                  "time_series_dimension": true
                }
              }
            },

# uninstall
DELETE kbn:/api/fleet/epm/packages/prometheus/1.17.0

elastic / kibana

[Fleet] Trigger rollover when `time_series_dimension` mappings change between template versions #190700