The ElasticSearch operator is designed to manage one or more elastic search clusters. Included in the project (initially) is the ability to create the Elastic cluster, deploy the data nodes
across zones in your Kubernetes cluster, and snapshot indexes to AWS S3.
The operator was built and tested on a 1.7.X Kubernetes cluster and is the minimum version required due to the operators use of Custom Resource Definitions.
NOTE: If using on an older cluster, please make sure to use version v0.0.7 which still utilize third party resources.
The operator was also currently designed to leverage Amazon AWS S3 for snapshot / restore to the elastic cluster. The goal of this project is to extend to support additional clouds and scenarios to make it fully featured.
By swapping out the storage types, this can be used in GKE, but snapshots won't work at the moment.
Watch a demo here:
https://www.youtube.com/watch?v=3HnV7NfgP6A
The operator is built using the controller + custom resource definition model. Once the controller is deployed to your cluster, it will automatically create the CustomResourceDefinition (CRD). Next create a Kubernetes object type elasticsearchCluster
to deploy the elastic cluster based upon the CRD.
Following parameters are available to customize the elastic cluster:
upmcenterprises/docker-elasticsearch-kubernetes:6.1.3_0
)key: value
gp2
)kubernetes.io/aws-ebs
)"true"
or "false"
Defaults to: "true"
parameters
are required by provisioner over and above type
and storage-class-provisioner
.zones
. You are free to choose any naming scheme providing they match. Master and Data pod names will include the zone name.Using a provisioner
section, specify type:
and provisioner:
values that match your custom Storage Classes.Retain
or Delete
)key: value
- effect:
eg: NoSchedule, NoExecute
key:
eg: somekey
operator:
eg: existsaffinity: affinity rules to put on the client node deployments
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: role
operator: In
values:
- client
topologyKey: kubernetes.io/hostname
The default image used adds TLS to the Elastic cluster. If not existing, secrets are automatically generated by the operator dynamically.
If supplying your own certs, first generate them and add to a secret. Secret should contain truststore.jks
and node-keystore.jks
. The name of the secret should follow the pattern: es-certs-[ClusterName]
. So for example if your cluster is named example-es-cluster
then the secret should be es-certs-example-es-cluster
.
The base image used is upmcenterprises/docker-elasticsearch-kubernetes:6.1.3_0
which can be overridden by adding to the custom cluster you create (See: CustomResourceDefinition above).
NOTE: If no image is specified, the default noted previously is used.
If you are using a private repository you can add a pull secret under spec in your ElasticsearchCluster manifest
spec:
client-node-replicas: 3
data-node-replicas: 3
data-volume-size: 10Gi
master-volume-size: 10Gi
java-options: -Xms256m -Xmx256m
master-node-replicas: 2
image-pull-secrets:
- name: pull-secret-name
snapshot:
bucket-name: elasticsnapshots99
cron-schedule: '@every 2m'
scheduler-enabled: false
storage:
storage-class-provisioner: kubernetes.io/aws-ebs
type: gp2
zones:
- us-east-1a
- us-east-1b
- us-east-1c
To deploy the operator simply deploy to your cluster:
$ kubectl create ns operator
$ kubectl create -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/controller.yaml -n operator
NOTE: In the example we're putting the operator into the namespace operator
. If you want to change this, then make sure to update the RBAC rules in the example/controller.yaml spec to match the namespace desired.
Run the following command to create a sample cluster on AWS and you most likely will have to update the zones to match your AWS Account, other examples are available as well if not running on AWS:
$ kubectl create -n operator -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/example-es-cluster.yaml
NOTE: Creating a custom cluster requires the creation of a CustomResourceDefinition. This happens automatically after the controller is created.
To run the operator on minikube, this sample file is setup to do that. It sets lower Java memory constraints as well as uses the default storage class in Minikube which writes to hostPath.
$ kubectl create -f https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/example/example-es-cluster-minikube.yaml
NOTE: Creating a custom cluster requires the creation of a CustomResourceDefinition. This happens automatically after the controller is created.
Both operator and cluster can be deployed using Helm charts:
$ helm repo add es-operator https://raw.githubusercontent.com/upmc-enterprises/elasticsearch-operator/master/charts/
$ helm install --name elasticsearch-operator es-operator/elasticsearch-operator --set rbac.enabled=True --namespace logging
$ helm install --name=elasticsearch es-operator/elasticsearch --set kibana.enabled=True --set cerebro.enabled=True --set zones="{eu-west-1a,eu-west-1b}" --namespace logging
⚡ $helm list
NAME REVISION UPDATED STATUS CHART NAMESPACE
elasticsearch 1 Thu Dec 7 11:53:45 2017 DEPLOYED elasticsearch-0.1.0 default
elasticsearch-operator 1 Thu Dec 7 11:49:13 2017 DEPLOYED elasticsearch-operator-0.1.0 default
Kibana and Cerebro can be automatically deployed by adding the cerebro piece to the manifest:
spec:
kibana:
image: docker.elastic.co/kibana/kibana-oss:6.1.3
cerebro:
image: upmcenterprises/cerebro:0.6.8
Once added the operator will create certs for Kibana or Cerebro and automatically secure with those certs trusting the same CA used to generate the certs for the Elastic nodes.
To access, just port-forward to the pod:
Kibana:
$ kubectl port-forward <podName> 5601:5601
$ curl https://localhost:5601
Cerebro:
$ kubectl port-forward <podName> 9000:9000
$ curl https://localhost:9000
(Note: Using custom image since upstream has x-pack installed and causes issues)
If changes are required to the cluster, say the replica count of the data nodes for example, just update the manifest and do a kubectl apply
on the resource.
Elasticsearch can snapshot it's indexes for easy backup / recovery of the cluster. Currently there's an integration to Amazon S3 or Google Cloud Storage as the backup repository for snapshots. The upmcenterprises
docker images include the S3 Plugin and the GCS Plugin which enables this feature in AWS and GCP.
Snapshots can be scheduled via a Cron syntax by defining the cron schedule in your elastic cluster. See: https://godoc.org/github.com/robfig/cron
NOTE: Be sure to enable the scheduler as well by setting scheduler-enabled=true
To enable the snapshots create a bucket in S3, then apply the following IAM permissions to your EC2 instances replacing {!YOUR_BUCKET!}
with the correct bucket name.
{
"Statement": [
{
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation",
"s3:ListBucketMultipartUploads",
"s3:ListBucketVersions"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::{!YOUR_BUCKET!}"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::{!YOUR_BUCKET!}/*"
]
}
],
"Version": "2012-10-17"
}
To enable snapshots with GCS on GKE, create a bucket in GCS and bind the storage.admin
role to the cluster service account replacing ${BUCKET}
with your bucket name:
gsutil mb gs://${BUCKET}
SA_EMAIL=$(kubectl run shell --rm --restart=Never -it --image google/cloud-sdk --command /usr/bin/curl -- -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/email)
PROJECT=$(gcloud config get-value project)
gcloud projects add-iam-policy-binding ${PROJECT} \
--role roles/storage.admin --member serviceAccount:${SA_EMAIL}
If you are using an elasticsearch image that requires authentication for the snapshot url, you can specify basic auth credentials.
spec:
client-node-replicas: 3
data-node-replicas: 3
data-volume-size: 10Gi
master-volume-size: 10Gi
java-options: -Xms256m -Xmx256m
master-node-replicas: 2
snapshot:
bucket-name: elasticsnapshots99
cron-schedule: '@every 2m'
scheduler-enabled: false
authentication:
password: betterpasswordneeded
username: basicauthadmin
storage:
storage-class-provisioner: kubernetes.io/aws-ebs
type: gp2
zones:
- us-east-1a
- us-east-1b
- us-east-1c
Once deployed and all pods are running, the cluster can be accessed internally via https://elasticsearch:9200/ or https://${ELASTICSEARCH_SERVICE_HOST}:9200/. Note: the service name for the ES client may also be "elasticsearch + ElasticsearchCluster
resource.
To run the Operator locally:
$ mkdir -p /tmp/certs/config && mkdir -p /tmp/certs/certs
$ go get -u github.com/cloudflare/cfssl/cmd/cfssl
$ go get -u github.com/cloudflare/cfssl/cmd/cfssljson
$ go run cmd/operator/main.go --kubecfg-file=${HOME}/.kube/config
Built by UPMC Enterprises in Pittsburgh, PA. http://enterprises.upmc.com/