How to deploy EFK stack on k8s

[oomichi]

Elasticsearch, Fluentd, Kibana間の設定をどうするのか一般的な方法を調べる。

[oomichi]

https://blog.ptrk.io/how-to-deploy-an-efk-stack-to-kubernetes/ がすばらしい。

• Fluentd: DaemonSet でデプロイ。各ノードの /var/lib/docker 配下のコンテナログを採取、Elasticsearch に送る。
• Elasticsearch: Fuluentd から送られてきたログを格納、Kibanaからの要求に応じて格納したログ情報を渡す。
• Kibana: ユーザがログ情報を見るためにアクセスする。

方法

• Elasticsearchのデプロイ 9300/tcp で discovery service (role:master) 9200/tcp で service (role:data) 9200/tcp で ingest service (role:ingest)

  
  git clone https://github.com/pires/kubernetes-elasticsearch-cluster
  cd kubernetes-elasticsearch-cluster/
  kubectl create namespace logging
  kubectl -n logging create -f es-discovery-svc.yaml
  kubectl -n logging create -f es-svc.yaml

kubectl -n logging create -f es-master.yaml kubectl -n logging rollout status -f es-master.yaml

kubectl -n logging create -f es-ingest-svc.yaml kubectl -n logging create -f es-ingest.yaml kubectl -n logging rollout status -f es-ingest.yaml

kubectl -n logging create -f es-data.yaml kubectl -n logging rollout status -f es-data.yaml

- Fluentdのデプロイ

git clone https://github.com/fluent/fluentd-kubernetes-daemonset cd fluentd-kubernetes-daemonset/

Elasticsearchに合わせて設定変更

--- a/fluentd-daemonset-elasticsearch-rbac.yaml +++ b/fluentd-daemonset-elasticsearch-rbac.yaml @@ -3,14 +3,14 @@ apiVersion: v1 kind: ServiceAccount metadata: name: fluentd

• namespace: kube-system

• namespace: logging

  ---

  apiVersion: rbac.authorization.k8s.io/v1beta1 kind: ClusterRole metadata: name: fluentd

• namespace: kube-system
• namespace: logging rules:
  • apiGroups:
  • "" @@ -34,13 +34,13 @@ roleRef: subjects:
  • kind: ServiceAccount name: fluentd
• namespace: kube-system

• namespace: logging

  apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: fluentd

• namespace: kube-system
• namespace: logging labels: k8s-app: fluentd-logging version: v1 @@ -63,7 +63,7 @@ spec: image: fluent/fluentd-kubernetes-daemonset:elasticsearch env:
  • name: FLUENT_ELASTICSEARCH_HOST
• value: "elasticsearch-logging"
• value: "elasticsearch"
  • name: FLUENT_ELASTICSEARCH_PORT value: "9200"
  • name: FLUENT_ELASTICSEARCH_SCHEME @@ -74,6 +74,8 @@ spec: value: "elastic"
  • name: FLUENT_ELASTICSEARCH_PASSWORD value: "changeme"
• • name: FLUENT_UID
• value: "0" resources: limits: memory: 200Mi

  デプロイ

  kubectl create -f fluentd-daemonset-elasticsearch-rbac.yaml

  Kibana のデプロイ

  $ cat kibana.yaml apiVersion: apps/v1 kind: Deployment metadata: namespace: logging name: kibana labels: component: kibana spec: replicas: 1 selector: matchLabels: component: kibana template: metadata: labels: component: kibana spec: containers:

  • name: kibana image: docker.elastic.co/kibana/kibana-oss:6.2.2 env:
    • name: CLUSTER_NAME value: elasticsearch resources: limits: cpu: 1000m requests: cpu: 100m ports:
    • containerPort: 5601 name: http $ kubectl create -f kibana.yaml $ kubectl get pods -n logging NAME READY STATUS RESTARTS AGE es-data-b479bcbd-8ngqk 1/1 Running 0 1d es-data-b479bcbd-zcnkf 1/1 Running 0 1d es-ingest-84fd6b464-55dls 1/1 Running 1 1d es-ingest-84fd6b464-hvmrj 1/1 Running 1 1d es-master-d4d46765-4pczj 1/1 Running 1 1d es-master-d4d46765-cjdtl 1/1 Running 0 1d es-master-d4d46765-vchdx 1/1 Running 1 1d fluentd-85kj5 1/1 Running 0 20h fluentd-tg87g 1/1 Running 0 20h kibana-7c77d9fdc8-qxrjv 1/1 Running 0 50s $

      サービス

      $ cat svc.yaml apiVersion: v1 kind: Service metadata: namespace: logging name: kibana labels: component: kibana spec: selector: component: kibana ports:

    • name: http port: 5601 $ kubectl create -f svc.yaml

[oomichi]

k8s v1.12 は https://github.com/oomichi/try-kubernetes/issues/55 の問題があるので v1.11.3 を使う。

[oomichi]

ingress を使うには ingress controller の設定が必要。ingress-nginx を使う。

必須手順と書かれた方法を実施。

$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml
$ kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/baremetal/service-nodeport.yaml

[oomichi]

結構な回数 ingress-nginx controller が再起動している。

NAMESPACE       NAME                                        READY     STATUS             RESTARTS   AGE
ingress-nginx   default-http-backend-7db7c45b69-z6bkg       1/1       Running            163        3d
ingress-nginx   nginx-ingress-controller-6bd7c597cb-ntbrm   1/1       Running            636        3d

また、es-data が CrashLoopBackOff している。

logging         es-data-b479bcbd-8ngqk                      0/1       CrashLoopBackOff   1339       4d
logging         es-data-b479bcbd-zcnkf                      0/1       CrashLoopBackOff   1331       4d