search

Xzeryn / Elastic-Stack-Docker

MIT License
3 stars 1 forks source link

readme

elastic-stack-docker

This project creates a full Elastic stack in docker using docker compose.

It is based heavily on the work done by elkninja and adds local copies of the Elastic Package Registry (EPR) and Elastic Artifact Registry (EAR) containers for air-gapped environments.

WARNING: The Elastic Package Registry image is ~15G and the Elastic Artifact Registry is ~4G in size.

The EPR and EAR are integrated into the project, but not required for the Elastic stack to function.

The project creates certs and stands up a 3-node Elasticsearch cluster, with Kibana and Fleet-Server already preconfigured. It also stands up Logstash, Metricbeat, Filebeat, and a webapp APM example container using docker profiles.

Elasticsearch and Kibana are preconfigured and insturmented with APM.

Stack Components

This project is broken into multiple docker compose files that build on each other, enabling multiple final configurations when the stack is brought up.

The docker-compose.yml is the base configuration of the stack. It generates the certs required and brings online the Elasticsearch nodes, Kibana, and Fleet/APM server. Therefore, it will always be used when issuing the docker compose up command.

The air-gapped.yml adds to the base configuration provided by the docker-compose.yml and provides the configuration changes and containers necessary to run the Elastic stack in an air-gapped environment.

The examples.yml adds different functionality to the base configuration by bringing online different containers using docker's profiles feature. (see below). This file is included at the top of the docker-compose.yml.

docker-compose.yml

air-gapped.yml

examples.yml

Prerequisites

Building Docker Images

Initially, internet access is required to build and pull the images. The images are built or pulled automatically when docker compose executes.

Initial Setup

Make a copy of the env.template file and name it .env. Use the .env file to change settings. You must set the DOCKER_HOST_IP variable to the correct host IP for the stack deployment to work

Deploying the stack

The stack can be deployed in many configurations including air-gapped. The various configurations can be enabled using the profiles feature of docker compose.

Usage:

To bring up the basic stack (Elasticsearch, Kibana and Fleet/APM Server):

docker compose up -d

To enable included examples reference the profiles section below. For example, to bring up the stack with Metricbeat enabled for cluster monitoring use the following command:

docker compose --profile monitoring up -d

Multiple profiles can also be chained together. The following command enables Metricbeat, Logstash and an APM example.

docker compose --profile monitoring --profile logstash --profile apm up -d

NOTE: You can view the configuration that docker compose will apply prior to starting the project by using the config parameter instead of up -d.

Examples:

docker compose config

or

docker compose --profile monitoring config

Running Air-Gapped

The air-gapped.yml configures the stack to utilize local Elastic Package Registry (EPR) and Elastic Artifact Registry (EAR) services. These services are required in an air-gapped environment to install integrations and binaries required by the stack.

Using the air-gapped configuration requires chaining multiple docker-compose files due to configuration changes that need to be made to the base configuration. This is done using the -f <filename> flag when executing the docker compose command.

Usage:

To bring up the basic air-gapped stack (Elasticsearch, Kibana, Fleet/APM Server, EAR, and EPR):

docker compose -f docker-compose.yml -f air-gapped.yml up -d

Profiles may also be used when using air-gapped. Using the same metricbeat example above, the command would be:

docker compose -f docker-compose.yml -f air-gapped.yml --profile monitoring up -d

Bring down the stack

To bring down the stack without purging the data volumes, execute the same command (including -f <filename> and --profile flags) but replace the up -d with down

docker compose down

or

docker compose --profile monitoring down

or

docker compose -f docker-compose.yml -f air-gapped.yml --profile monitoring down

To bring down the stack and remove the data volumes, add -v to your command

docker compose down -v

or

docker compose --profile monitoring down -v

or

docker compose -f docker-compose.yml -f air-gapped.yml --profile monitoring down -v

Profiles

Profiles are enabled to configure different services for demo/example purposes.

To use a profile add --profile <name> to the docker compose command. Each profile enabled but have its own --profile <name>, you cannot use a list of comma separated profile names.

Usage Examples:

Available Profiles

Machine Learning

Monitoring

Filebeat

Logstash

APM

Agent

  1. Using Custom Logs integraton:
    • Drop log files in the agent_ingest_data folder to ingest logs using the Custom Logs integration.
    • The data will be in the messages field of the logs-generic-* index.
    • Modify the processor field of the integration (in the settings) or the logs-generic-* pipeline to extract and format the data.
  2. Using the Custom UDP Logs integration:
    • Send logs over UDP to the docker host IP to the port designated in the .env file (default: 9003)
    • The integration has syslog parsing enabled by default
    • Changes can be made to the logs-udp.generic-* ingest pipeline for additional formatting or to the settings of the integration
  3. Using the Custom TCP Logs integration:
    • Send logs over TCP to the docker host IP to the port designated in the .env file (default: 9004)
    • The integration has syslog parsing enabled by default
    • Changes can be made to the logs-TCP.generic-* ingest pipeline for additional formatting or to the settings of the integration

References

Resources:

Fleet/Agent

APM:

Add more Elasticsearch nodes or configure for production

https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html#docker-compose-file