Remove min-age on deletion phase of apm_traces ILM policies

[kenoir]

What's changing and why?

This change allows indices rolled over hot to warm to be kept for 1 day, and then deleted straight away. Currently the hot phase rollover rules keep the apm_traces indices (these tend to be the largest) around for 30 days or until they reach 50GB (total disk size at present is 250GB).

The expected process is:

• After either 30 days, or the hot index reaches 30gb it is rolled over to the warm phase
• After another day the deletion can occur immediately, we've specified the min_age as 1 day (after rollover).

We reduce the max size of the hot index from 50gb to 30gb to allow expected space for 2 hot indices, the delete min_age providing us with a buffer before deletion.

For context see:

• https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-index-lifecycle.html#ilm-phase-transitions
• https://www.elastic.co/guide/en/elasticsearch/reference/current/ilm-rollover.html#_rollover_condition_blocks_phase_transition

Notes on max_age & min_age

• In the hot phase rollover configuration, max_age refers to the maximum age of the index before it is rolled over. I interpret this to mean: unless the max_size is met by the currently hot index, it will be kept for 30 days.
• In the delete phase, min_age refers to the minimum age of the index _before it is deleted. I interpret this to mean: indices that have been rolled over will be kept for at least one day before being deleted.

terraform plan

Terraform will perform the following actions:

  # elasticstack_elasticsearch_index_lifecycle.apm_traces will be updated in-place
  ~ resource "elasticstack_elasticsearch_index_lifecycle" "apm_traces" {
        id            = "J6V5gvONTve393_OJiiJnA/weco-traces-apm"
        name          = "weco-traces-apm"
        # (1 unchanged attribute hidden)

      ~ delete {
          ~ min_age = "10d" -> "1d"
        }

      ~ hot {
            # (1 unchanged attribute hidden)

          ~ rollover {
              ~ max_size = "50gb" -> "30gb"
                # (2 unchanged attributes hidden)
            }
        }
    }

  # elasticstack_elasticsearch_index_lifecycle.apm_traces_rum will be updated in-place
  ~ resource "elasticstack_elasticsearch_index_lifecycle" "apm_traces_rum" {
        id            = "J6V5gvONTve393_OJiiJnA/weco-traces-apm-rum"
        name          = "weco-traces-apm-rum"
        # (1 unchanged attribute hidden)

      ~ delete {
          ~ min_age = "10d" -> "1d"
        }

      ~ hot {
            # (1 unchanged attribute hidden)

          ~ rollover {
              ~ max_size = "50gb" -> "30gb"
                # (2 unchanged attributes hidden)
            }
        }
    }

Plan: 0 to add, 2 to change, 0 to destroy.

Related PRs:

Follows: https://github.com/wellcomecollection/platform-infrastructure/pull/408