search

elastic / cloud-on-k8s

Elastic Cloud on Kubernetes
Other
2.53k stars 685 forks source link

[APM] Internal Server Error on new ELK Stack in Kibana #4835

Open SharpThunder opened 2 years ago

SharpThunder commented 2 years ago

Bug Report

What did you do? I tried to install APM Server with Elastic Search and Kibana. What did you expect to see? Working monitoring with internal monitoring in Stack Monitoring What did you see instead? Under which circumstances? image

in https://kibana/app/monitoring#/overview?_g=(cluster_uuid:********,inSetupMode:!t,refreshInterval:(pause:!f,value:10000),time:(from:now-15m,to:now))

Environment

$ kubectl version
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.1", GitCommit:"632ed300f2c34f6d6d15ca4cef3d3c7073412212", GitTreeState:"clean", BuildDate:"2021-08-19T15:38:26Z", GoVersion:"go1.16.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.13-eks-8df270", GitCommit:"8df2700a72a2598fa3a67c05126fa158fd839620", GitTreeState:"clean", BuildDate:"2021-07-31T01:36:57Z", GoVersion:"go1.15.14", Compiler:"gc", Platform:"linux/amd64"}
WARNING: version difference between client (1.22) and server (1.19) exceeds the supported minor version skew of +/-1

Also I have this but I don't know if it is related or not.

{"type":"log","@timestamp":"2021-09-14T16:13:56+00:00","tags":["error","plugins","monitoring","monitoring"],"pid":1208,"message":"TypeError: Cannot destructure property 'beats' of '(intermediate value)(intermediate value)(intermediate value)' as it is undefined.\n at handleResponse (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/apm/get_apms.js:46:5)\n at getApms (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/apm/get_apms.js:181:10)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async Promise.all (index 1)\n at Object.handler (/usr/share/kibana/x-pack/plugins/monitoring/server/routes/api/v1/apm/instances.js:51:31)\n at handler (/usr/share/kibana/x-pack/plugins/monitoring/server/plugin.js:406:28)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)"} 


I installed Elastic Cloud on K8S with Helm

helm repo add elastic https://helm.elastic.co helm repo update

helm install elastic-operator elastic/eck-operator -n elastic-system --create-namespace


Here is my yaml files for Elastic Search, Kibana, and APM Server

**Elastic Search**

This sample sets up an Elasticsearch cluster with 3 nodes.

apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: es-cluster namespace: elastic-system spec: version: 7.14.0 volumeClaimDeletePolicy: DeleteOnScaledownOnly nodeSets:

APM server is recognized by Kibana image

Indices also created image

Thanks.

thbkrkr commented 2 years ago

For APM Server, you need to add the monitoring settings in the configuration to use internal collection to send monitoring data (see the documentation):

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm
  namespace: elastic-system
spec:
  version: 7.14.0
  count: 1
  elasticsearchRef:
    name: "es-cluster"
  kibanaRef:
    name: "kibana"
  config:
    # enable internal collection to send monitoring data
    monitoring:
      enabled: true
  podTemplate:
    spec:
      containers:
      - name: apm-server
        resources:
          limits:
            memory: 1Gi
            cpu: 1
SharpThunder commented 2 years ago

Thank you, that solves my issue. However, Could you please add this to Elastic Cloud on Kubernetes documentation as a warning either this parameter to be applied or MetricBeats installed before installing APM? Also, It can be helpful if there is a meaningful error for this when debugging.