Research Elastic Index Management

[aarz-snl]

Index management / automation will be an important configuration item with elasticsearch. There's 2 ways we can go about this.

The previous way we were doing it was using Index Lifecycle management. This can be done during the install process using a bash / python script and interacting with the api

For instance -- to set the wazuh alerts index to delete indexes after they are 3 days old you would interact with the api like so

curl -k -u elastic:password -k -X PUT "https://localhost:9200/_ilm/policy/wazuh_alerts_cleanup_policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms", // Actions start immediately upon index creation
        "actions": {}
      },
      "delete": {
        "min_age": "3d", // Set age at which the index should be deleted
        "actions": {
          "delete": {} // Action to delete the index
        }
      }
    }
  }
}'

This creates a policy -- then you APPLY said policy

curl -u elastic:password -k -X PUT "https://localhost:9200/_index_template/wazuh_alerts_template" -H 'Content-Type: application/json' -d'
{
  "index_patterns": ["wazuh-alerts-4.x-*"], // Target all Wazuh alert indices
  "template": {
    "settings": {
      "index.lifecycle.name": "wazuh_alerts_cleanup_policy", // Apply the defined ILM policy
      "index.lifecycle.rollover_alias": "wazuh-alerts" // Specify if using rollover; adjust if not needed
    }
  }
}'

Additionally, for elastic logs (non wazuh logs coming from elastic agent etc)

create a policy again... this one for 4 days with a rollover of 3 days (just to show different options we have making these)

curl -u elastic:password -k -X PUT "https://localhost:9200/_ilm/policy/logs" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",  // Start immediately after index creation
        "actions": {
          "rollover": {
            "max_age": "3d"  // Rollover the index after 3 days
          }
        }
      },
      "delete": {
        "min_age": "4d",  // Delete the index 1 day after it rolls over, totaling 4 days
        "actions": {
          "delete": {}  // Action to delete the index
        }
      }
    }
  }
}'

curl -u elastic:password -k -X PUT "https://localhost:9200/_ilm/policy/metrics" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",  // Start immediately after index creation
        "actions": {
          "rollover": {
            "max_age": "3d",  // Rollover the index after 3 days
            "max_primary_shard_size": "50gb"  // Assuming you want to keep the size condition as well
          }
        }
      },
      "delete": {
        "min_age": "4d",  // Delete the index 1 day after it rolls over, totaling 4 days
        "actions": {
          "delete": {}  // Action to delete the index
        }
      }
    }
  }
}'

This edits the built in policy that already exists for these... logs- and metrics- indices. As they already exist we didn't need to create a policy first.

But ultimate the flow goes create a policy if one doesn't already exist -> apply it to the indices.

The other way to manage indices would be to deploy something like curator into the compose stack

https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html

Curator allows you to setup automated rules to delete indices

For instance see code on lme-priv repo located here

https://github.com/cisagov/LME-PRIV/tree/lme-2.0/lme-2-arch/curator

With curator you create 'action files' that can perform different actions pending your selected triggers. In the example code above i have it deleting indices that are older than 2 days.

The benefit of using curator is it can be used to perform deletions based on size (or other triggers). So, if we wanted to deleted an index if the size of all of our indexes are larger than 500GB then you can do that.

The con here would be adding another service to the stack.

[aarz-snl]

Recently there has been discussion of implementing ansible into the deployment of lme...

I think you can still do this using the uri module build into ansible.

Example for adjusting logs-*

---
- name: Configure Elasticsearch ILM Policy
  hosts: localhost
  tasks:
    - name: Set ILM policy for logs
      uri:
        url: "https://localhost:9200/_ilm/policy/logs"
        method: PUT
        user: "elastic"
        password: "password"
        validate_certs: no
        headers:
          Content-Type: "application/json"
        body_format: json
        body: >
          {
            "policy": {
              "phases": {
                "hot": {
                  "min_age": "0ms",
                  "actions": {
                    "rollover": {
                      "max_age": "3d"
                    }
                  }
                },
                "delete": {
                  "min_age": "4d",
                  "actions": {
                    "delete": {}
                  }
                }
              }
            }
          }
      register: response

    - name: Print response from Elasticsearch
      debug:
        msg: "{{ response }}"

[aarz-snl]

https://docs.ansible.com/ansible/latest/collections/ansible/builtin/uri_module.html