AckeeCZ / terraform-gcp-elasticsearch

Terraform module for deploying Elasticsearch cluster on GCP
MIT License
10 stars 7 forks source link
elasticsearch gcp terraform terraform-modules

Ackee GCE Elasticsearch Terraform module

This module is primary written for provisioning of GCE instance from our ES image (https://github.com/AckeeCZ/packer-elasticsearch)

It does a few things :

Usage

module "elasticsearch_prod" {
  source = "AckeeCZ/elasticsearch/gcp"

  project                   = "my-gcp-project"
  region                    = "europe-west3"
  zone                      = "europe-west3-c"
  instance_name             = "elasticsearch-prod"
  cluster_name              = "elasticsearch"
  cluster_ipv4_cidr         = "10.128.0.0/14"
  node_count                = "3"
  heap_size                 = "1500m"
  raw_image_source          = "https://storage.googleapis.com/ackee-images/ackee-elasticsearch-7-disk-79.tar.gz"
  data_disk_size            = "10"
  custom_pre_start_commands = "/usr/share/elasticsearch/bin/elasticsearch-plugin install -b analysis-stempel"
}

Running plugins needed for index startup

If you need some plugins that is required for index startup (e.g., analysis-stempel), you must install them utilizing custom_pre_start_commands variable, which is called before running systemctl start elasticsearch.

See Usage part above for example with analysis-stempel plugin installation

Upgrading to v12

To keep ingesting Elasticsearch logs with the new Google Cloud Ops agent, please upgrade your ES instances to image generated from v1.0.0 of the packer-elasticsearch AFTER upgrading this module to v12.0.0

Before you do anything in this module

Install pre-commit hooks by running following commands:

brew install pre-commit
pre-commit install

Requirements

Name Version
terraform >= 1.0

Providers

Name Version
google n/a
google-beta n/a
kubernetes n/a
random n/a
tls n/a

Modules

No modules.

Resources

Name Type
google-beta_google_compute_forwarding_rule.elasticsearch resource
google-beta_google_compute_health_check.elasticsearch resource
google-beta_google_compute_instance_group.elasticsearch resource
google-beta_google_compute_region_backend_service.elasticsearch resource
google-beta_google_compute_subnetwork.proxy resource
google_compute_disk.data resource
google_compute_firewall.elasticsearch_allow_external_subnets resource
google_compute_firewall.elasticsearch_allow_external_tags resource
google_compute_firewall.elasticsearch_allow_healthchecks resource
google_compute_firewall.elasticsearch_allow_ilb_traffic resource
google_compute_image.elasticsearch resource
google_compute_instance.elasticsearch resource
google_project_iam_member.elasticsearch_backup_role resource
google_service_account.elasticsearch_backup resource
google_service_account_key.elasticsearch_backup resource
google_storage_bucket.backup_repository resource
kubernetes_cron_job_v1.backup_cleanup resource
kubernetes_stateful_set.elasticsearch resource
random_string.es_name_suffix resource
tls_private_key.provision resource
google_compute_network.default data source
google_compute_zones.available data source

Inputs

Name Description Type Default Required
add_random_suffix Add random suffix to all resources with possible duplicates if the same module is deployed multiple times bool false no
allowed_ipv4_subnets IPv4 subnets allowed to communicate with ES instances. list(string) [] no
allowed_tags Network tags allowed to communicate with ES instances. list(string) [] no
backend_service_timeout_sec How many seconds to wait for the backend before considering it a failed request. Default is 30 seconds. Valid range is [1, 86400]. number 30 no
backup_failed_jobs_history_limit Set retention for failed jobs history number 5 no
backup_repository_create If GCS bucket should be created. Set to false, if you previously created bucket bool true no
backup_repository_name Custom name of Elasticsearch backup repository, same name is going to be used for backup bucket string "" no
backup_schedule Backup schedule in cron format string "0 3 * * *" no
backup_storage_class The storage class you set for an object affects the object's availability and pricing model string "STANDARD" no
backup_successful_jobs_history_limit Set retention for successful jobs history number 3 no
cluster_ca_certificate Public CA certificate that is the root of trust for the GKE K8s cluster string n/a yes
cluster_endpoint Cluster control plane endpoint string n/a yes
cluster_name ES cluster name. string n/a yes
cluster_token Cluster master token, keep always secret! string n/a yes
custom_init_commands Any custom commands which should be run after bootstrapping the Elasticsearch cluster after starting Elasticsearch service string "" no
custom_pre_start_commands Any custom commands which should be run after bootstrapping the Elasticsearch cluster before starting Elasticsearch service string "" no
data_disk_size Persistent disk size specified in GB. string n/a yes
data_disk_type Type of disk used as a persistent storage. string "pd-ssd" no
enable_health_check_logging Enable health check logging bool false no
es_image_creation_timeout Timeout for creating ES image string "10m" no
es_name_suffix_length Length of random generated suffix for ES name number 8 no
health_check_healthy_threshold How many consecutive successes must occur to mark a VM instance healthy. number 2 no
health_check_interval_sec How often (in seconds) to send a health check. The default value is 5 seconds. number 5 no
health_check_timeout_sec How long (in seconds) to wait before claiming failure. The default value is 5 seconds. It is invalid for timeoutSec to have greater value than checkIntervalSec. number 5 no
health_check_unhealthy_threshold How many consecutive failures must occur to mark a VM instance unhealthy. number 2 no
heap_size Heap size setting for ES. string "1800m" no
instance_name Base for GCE instances name. Must be unique within GCP project string n/a yes
load_balancer_subnetwork The subnetwork that the load balanced IP should belong to for this Forwarding Rule. If the network specified is in auto subnet mode, this field is optional. However, if the network is in custom subnet mode, a subnetwork must be specified. string "10.64.0.0/26" no
machine_type The machine type to create string "n1-standard-1" no
namespace K8s namespace used to deploy endpoints and services. string "production" no
network GCE VPC used for compute instances string "default" no
node_count Number of ES nodes to deploy. number 1 no
node_roles Map of node roles. Index is instance number, starting at zero. If not defined, node has all roles map(any) {} no
project Name of GCP project. string n/a yes
raw_image_source URL of tar archive containing RAW source for ES image (you can use Packer image template to generate image, as mentioned above). string "https://storage.googleapis.com/ackee-images/ackee-elasticsearch-7-disk-79.tar.gz" no
region Region of GCP project. string n/a yes
root_disk_size Persistent disk size specified in GB. string "10" no
zone Zone of GCP project - optional parameter, if not set, the instances will be spread across the available zones. string null no

Outputs

Name Description
ilb_dns DNS name follows GCP internal rule SERVICE_LABEL.FORWARDING_RULE_NAME.il4.REGION.lb.PROJECT_ID.internal
ip_address The internal IP assigned to the regional forwarding rule.