search

elastic / ansible-elasticsearch

Ansible playbook for Elasticsearch
Other
1.59k stars 857 forks source link
ansible ansible-role elasticsearch role-elasticsearch

readme

ARCHIVED

This project is no longer maintained.

You are welcomed to keep using it and adapting it to work for your own needs, including with Elasticsearch 8.x.

For alternative getting started experiences, you may want to try one of these options:

ansible-elasticsearch

Ansible Galaxy

THIS ROLE IS FOR 7.x & 6.x, but should still work with 8.x (see note).

Ansible role for 7.x/6.x Elasticsearch - tests used to run and pass on the below platforms:

BREAKING CHANGES

Notice about multi-instance support

Removing the MAX_THREAD settings

Ansible-elasticsearch 7.5.2 is removing the option to customize the maximum number of threads the process can start in #637. We discovered that this option wasn't working anymore since multi-instance support removal in ansible-elasticsearch 7.1.1. This option will be added back in a following release if it's still relevant regarding latest Elasticsearch evolutions.

Changes about configuration files

Ansible-elasticsearch 7.5.2 is updating the configuration files provided by this role in #637 which contained some options deprecated in 6.x and 7.x:

Removing OSS distribution for versions >= 7.11.0

Starting from Elasticsearch 7.11.0, OSS distributions will no longer be provided following the recent Elasticsearch license change.

This Ansible role will fail if oss_version is set to true and es_version is greater than 7.11.0.

See Doubling down on open, Part II blog post for more details.

How to override configuration files provided by ansible-elasticsearch?

You can now override the configuration files with your own versions by using the following Ansible variables:

Dependency

This role uses the json_query filter which requires jmespath on the local machine.

Usage

Create your Ansible playbook with your own tasks, and include the role elasticsearch. You will have to have this repository accessible within the context of playbook.

ansible-galaxy install elastic.elasticsearch,v7.17.0

Then create your playbook yaml adding the role elasticsearch. The application of the elasticsearch role results in the installation of a node on a host.

The simplest configuration therefore consists of:

- name: Simple Example
  hosts: localhost
  roles:
    - role: elastic.elasticsearch
  vars:
    es_version: 7.17.0

The above installs Elasticsearch 7.17.0 in a single node 'node1' on the hosts 'localhost'.

Note: Elasticsearch default version is described in es_version. You can override this variable in your playbook to install another version. While we are testing this role only with one 7.x and one 6.x version (respectively 7.17.0 and 6.8.23 at the time of writing), this role should work with other versions also in most cases.

This role also uses Ansible tags. Run your playbook with the --list-tasks flag for more information.

Testing

This playbook uses Kitchen for CI and local testing.

Requirements

Running the tests

Install the ruby dependencies with bundler

make setup

If you want to test X-Pack features with a license you will first need to export the ES_XPACK_LICENSE_FILE variable.

export ES_XPACK_LICENSE_FILE="$(pwd)/license.json"

To converge an Ubuntu 16.04 host running X-Pack

$ make converge

To run the tests

$ make verify

To list all of the different test suits

$ make list

The default test suite is Ubuntu 16.04 with X-Pack. If you want to test another suite you can override this with the PATTERN variable

$ make converge PATTERN=security-centos-7

The PATTERN is a kitchen pattern which can match multiple suites. To run all tests for CentOS

$ make converge PATTERN=centos-7

The default version is 7.x. If you want to test 6.x you can override it with the VERSION variable, for example:

$ make converge VERSION=6.x PATTERN=security-centos-7

When you are finished testing you can clean up everything with

$ make destroy-all

Basic Elasticsearch Configuration

All Elasticsearch configuration parameters are supported. This is achieved using a configuration map parameter 'es_config' which is serialized into the elasticsearch.yml file. The use of a map ensures the Ansible playbook does not need to be updated to reflect new/deprecated/plugin configuration parameters.

In addition to the es_config map, several other parameters are supported for additional functions e.g. script installation. These can be found in the role's defaults/main.yml file.

The following illustrates applying configuration parameters to an Elasticsearch instance.

- name: Elasticsearch with custom configuration
  hosts: localhost
  roles:
    - role: elastic.elasticsearch
  vars:
    es_data_dirs:
      - "/opt/elasticsearch/data"
    es_log_dir: "/opt/elasticsearch/logs"
    es_config:
      node.name: "node1"
      cluster.name: "custom-cluster"
      discovery.seed_hosts: "localhost:9301"
      http.port: 9201
      transport.port: 9301
      node.data: false
      node.master: true
      bootstrap.memory_lock: true
    es_heap_size: 1g
    es_api_port: 9201

Whilst the role installs Elasticsearch with the default configuration parameters, the following should be configured to ensure a cluster successfully forms:

The network.publish_host setting allows to control the host the node will publish itself within the cluster so other nodes will be able to connect to it.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html for further details on default binding behavior and available options. The role makes no attempt to enforce the setting of these are requires users to specify them appropriately. It is recommended master nodes are listed and thus deployed first where possible.

A more complex example:

- name: Elasticsearch with custom configuration
  hosts: localhost
  roles:
    - role: elastic.elasticsearch
  vars:
    es_data_dirs:
      - "/opt/elasticsearch/data"
    es_log_dir: "/opt/elasticsearch/logs"
    es_config:
      node.name: "node1"
      cluster.name: "custom-cluster"
      discovery.seed_hosts: "localhost:9301"
      http.port: 9201
      transport.port: 9301
      node.data: false
      node.master: true
      bootstrap.memory_lock: true
    es_heap_size: 1g
    es_start_service: false
    es_api_port: 9201
    es_plugins:
      - plugin: ingest-attachment

Important Notes

The role uses es_api_host and es_api_port to communicate with the node for actions only achievable via http e.g. to install templates and to check the NODE IS ACTIVE. These default to "localhost" and 9200 respectively. If the node is deployed to bind on either a different host or port, these must be changed.

Only use es_data_dirs and es_log_dir for customizing the data and log dirs respectively. When using together with es_config['path.data'] and es_config['path.logs'] it would result in generating duplicate data- and logs-keys in elasticsearch.yml and thus let fail to start elasticsearch.

Multi Node Server Installations

The application of the elasticsearch role results in the installation of a node on a host. Specifying the role multiple times for a host therefore results in the installation of multiple nodes for the host.

An example of a three server deployment is shown below. The first server holds the master and is thus declared first. Whilst not mandatory, this is recommended in any multi node cluster configuration. The two others servers hosts data nodes.

Note that we do not support anymore installation of more than one node in the same host

- hosts: master_node
  roles:
    - role: elastic.elasticsearch
  vars:
    es_heap_size: "1g"
    es_config:
      cluster.name: "test-cluster"
      cluster.initial_master_nodes: "elastic02"
      discovery.seed_hosts: "elastic02:9300"
      http.host: 0.0.0.0
      http.port: 9200
      node.data: false
      node.master: true
      transport.host: 0.0.0.0
      transport.port: 9300
      bootstrap.memory_lock: false
    es_plugins:
     - plugin: ingest-attachment

- hosts: data_node_1
  roles:
    - role: elastic.elasticsearch
  vars:
    es_data_dirs:
      - "/opt/elasticsearch"
    es_config:
      cluster.name: "test-cluster"
      cluster.initial_master_nodes: "elastic02"
      discovery.seed_hosts: "elastic02:9300"
      http.host: 0.0.0.0
      http.port: 9200
      node.data: true
      node.master: false
      transport.host: 0.0.0.0
      transport.port: 9300
      bootstrap.memory_lock: false
    es_plugins:
      - plugin: ingest-attachment

- hosts: data_node_2
  roles:
    - role: elastic.elasticsearch
  vars:
    es_config:
      cluster.name: "test-cluster"
      discovery.seed_hosts: "elastic02:9300"
      http.host: 0.0.0.0
      http.port: 9200
      node.data: true
      node.master: false
      transport.host: 0.0.0.0
      transport.port: 9300
      bootstrap.memory_lock: false
    es_plugins:
      - plugin: ingest-attachment

Parameters can additionally be assigned to hosts using the inventory file if desired.

Make sure your hosts are defined in your inventory file with the appropriate ansible_ssh_host, ansible_ssh_user and ansible_ssh_private_key_file values.

Then run it:

ansible-playbook -i hosts ./your-playbook.yml

Installing X-Pack Features

es_role_mapping:
  power_user:
    - "cn=admins,dc=example,dc=com"
  user:
    - "cn=users,dc=example,dc=com"
    - "cn=admins,dc=example,dc=com"
es_users:
  native:
    kibana4_server:
      password: changeMe
      roles:
        - kibana4_server
  file:
    es_admin:
      password: changeMe
      roles:
        - admin
    testUser:
      password: changeMeAlso!
      roles:
        - power_user
        - user
es_roles:
  file:
    admin:
      cluster:
        - all
      indices:
        - names: '*'
          privileges:
            - all
    power_user:
      cluster:
        - monitor
      indices:
        - names: '*'
          privileges:
            - all
    user:
      indices:
        - names: '*'
          privileges:
            - read
    kibana4_server:
      cluster:
          - monitor
      indices:
        - names: '.kibana'
          privileges:
            - all
  native:
    logstash:
      cluster:
        - manage_index_templates
      indices:
        - names: 'logstash-*'
          privileges:
            - write
            - delete
            - create_index
es_xpack_license: "{{ lookup('file', playbook_dir + '/files/' + es_cluster_name + '/license.json') }}"

If you don't have a license you can enable the 30-day trial by setting es_xpack_trial to true.

X-Pack configuration parameters can be added to the elasticsearch.yml file using the normal es_config parameter.

For a full example see here

Important Note for Native Realm Configuration

In order for native users and roles to be configured, the role calls the Elasticsearch API. Given security is installed this requires definition of two parameters:

These can either be set to a user declared in the file based realm, with admin permissions, or the default "elastic" superuser (default password is changeme).

X-Pack Security SSL/TLS

Additional Configuration

In addition to es_config, the following parameters allow the customization of the Java and Elasticsearch versions as well as the role behavior. Options include:

Earlier examples illustrate the installation of plugins using es_plugins. For officially supported plugins no version or source delimiter is required. The plugin script will determine the appropriate plugin version based on the target Elasticsearch version. For community based plugins include the full url. This approach should NOT be used for the X-Pack plugin. See X-Pack below for details here.

If installing Monitoring or Alerting, ensure the license plugin is also specified. Security configuration currently has limited support, but more support is planned for later versions.

To configure X-pack to send mail, the following configuration can be added to the role. When require_auth is true, you will also need to provide the user and password. If not these can be removed:

    es_mail_config:
        account: <functional name>
        profile: standard
        from: <from address>
        require_auth: <true or false>
        host: <mail domain>
        port: <port number>
        user: <e-mail address> --optional
        pass: <password> --optional

Both es_user_id and es_group_id must be set for the user and group ids to be set.

To add, update or remove elasticsearch.keystore entries, use the following variable:

# state is optional and defaults to present
es_keystore_entries:
- key: someKeyToAdd
  value: someValue
  state: present

- key: someKeyToUpdate
  value: newValue
  # state: present
  force: Yes

- key: someKeyToDelete
  state: absent

This role ships with sample templates located in the test/integration/files/templates-7.x directory. es_templates_fileglob variable is used with the Ansible with_fileglob loop. When setting the globs, be sure to use an absolute path.

Proxy

To define proxy globally, set the following variables:

Notes

IMPORTANT NOTES RE PLUGIN MANAGEMENT

Questions on Usage

We welcome questions on how to use the role. However, in order to keep the GitHub issues list focused on "issues" we ask the community to raise questions at https://discuss.elastic.co/c/elasticsearch. This is monitored by the maintainers.