search

pjhampton / kibana-prometheus-exporter

Prometheus metrics for Kibana
Apache License 2.0
114 stars 37 forks source link

Plugin Returning 500 #226

Closed knechtionscoding closed 3 years ago

knechtionscoding commented 3 years ago

I don't think this is unique to 7.13.0 (I saw the same thing on 7.12.1). I'm sure this is related to kibana configuration somewhere, but I'm really not sure. I also recognize this is the 3rd Issue I've opened, so I appreciate your help. The previous ones were resolved, so... hoping this will also be resolved.

Installing in ECK kibana on k8s, 7.13.0 all the way through,

Here are the logs from kibana when making the call:

{"type":"log","@timestamp":"2021-05-29T02:29:19+00:00","tags":["error","http"],"pid":947,"message":"Error: connect ECONNREFUSED 10.24.1.8:5601\n    at TCPConnectWrap.afterConnect [as oncomplete] (net.js
:1146:16) {\n  errno: -111,\n  code: 'ECONNREFUSED',\n  syscall: 'connect',\n  address: '10.24.1.8',\n  port: 5601,\n  config: {\n    url: 'https://kibana-url.example.com:5601/api/status',\n
  method: 'get',\n    headers: {\n      Accept: 'application/json, text/plain, */*',\n      Authorization: 'Bearer 85ezAxYzc1NxQ2hZTVJ1dThRU3VSVThYQ3dnAAAAAAAAAAAA',\n      'User-Agent': 'axios/0.21.1'\
n    },\n    transformRequest: [ [Function: transformRequest] ],\n    transformResponse: [ [Function: transformResponse] ],\n    timeout: 0,\n    adapter: [Function: httpAdapter],\n    xsrfCookieName: '
XSRF-TOKEN',\n    xsrfHeaderName: 'X-XSRF-TOKEN',\n    maxContentLength: -1,\n    maxBodyLength: -1,\n    httpsAgent: Agent {\n      _events: [Object: null prototype],\n      _eventsCount: 2,\n      _ma
xListeners: undefined,\n      defaultPort: 443,\n      protocol: 'https:',\n      options: [Object],\n      requests: {},\n      sockets: [Object],\n      freeSockets: {},\n      keepAliveMsecs: 1000,\n
      keepAlive: false,\n      maxSockets: Infinity,\n      maxFreeSockets: 256,\n      scheduling: 'fifo',\n      maxTotalSockets: Infinity,\n      totalSocketCount: 1,\n      maxCachedSessions: 100,\n
      _sessionCache: [Object],\n      [Symbol(kCapture)]: false\n    },\n    validateStatus: [Function: validateStatus],\n    data: undefined\n  },\n  request: <ref *1> Writable {\n    _writableState: W
ritableState {\n      objectMode: false,\n      highWaterMark: 16384,\n      finalCalled: false,\n      needDrain: false,\n      ending: false,\n      ended: false,\n      finished: false,\n      destro
yed: false,\n      decodeStrings: true,\n      defaultEncoding: 'utf8',\n      length: 0,\n      writing: false,\n      corked: 0,\n      sync: true,\n      bufferProcessing: false,\n      onwrite: [Fun
ction: bound onwrite],\n      writecb: null,\n      writelen: 0,\n      afterWriteTickInfo: null,\n      buffered: [],\n      bufferedIndex: 0,\n      allBuffers: true,\n      allNoop: true,\n      pend
ingcb: 0,\n      prefinished: false,\n      errorEmitted: false,\n      emitClose: true,\n      autoDestroy: true,\n      errored: null,\n      closed: false\n    },\n    _events: [Object: null prototyp
e] {\n      response: [Function: handleResponse],\n      error: [Function: handleRequestError]\n    },\n    _eventsCount: 2,\n    _maxListeners: undefined,\n    _options: {\n      maxRedirects: 21,\n
   maxBodyLength: 10485760,\n      protocol: 'https:',\n      path: '/api/status',\n      method: 'GET',\n      headers: [Object],\n      agent: [Agent],\n      agents: [Object],\n      auth: undefined,
\n      hostname: 'kibana-url.example.com',\n      port: '5601',\n      nativeProtocols: [Object],\n      pathname: '/api/status'\n    },\n    _ended: true,\n    _ending: true,\n    _redirectC
ount: 0,\n    _redirects: [],\n    _requestBodyLength: 0,\n    _requestBodyBuffers: [],\n    _onNativeResponse: [Function (anonymous)],\n    _currentRequest: ClientRequest {\n      _events: [Object: nul
l prototype],\n      _eventsCount: 7,\n      _maxListeners: undefined,\n      outputData: [],\n      outputSize: 0,\n      writable: true,\n      destroyed: false,\n      _last: true,\n      chunkedEnco
ding: false,\n      shouldKeepAlive: false,\n      _defaultKeepAlive: true,\n      useChunkedEncodingByDefault: false,\n      sendDate: false,\n      _removedConnection: false,\n      _removedContLen: f
alse,\n      _removedTE: false,\n      _contentLength: 0,\n      _hasBody: true,\n      _trailer: '',\n      finished: true,\n      _headerSent: true,\n      socket: [TLSSocket],\n      _header: 'GET /a
pi/status HTTP/1.1\\r\\n' +\n        'Accept: application/json, text/plain, */*\\r\\n' +\n        'Authorization: Bearer 85ezAxYzc1NxQ2hZTVJ1dThRU3VSVThYQ3dnAAAAAAAAAAAA\\r\\n' +\n        'User-Agent: a
xios/0.21.1\\r\\n' +\n        'Host: kibana-url.example.com:5601\\r\\n' +\n        'Connection: close\\r\\n' +\n        '\\r\\n',\n      _keepAliveTimeout: 0,\n      _onPendingData: [Function:
 noopPendingOutput],\n      agent: [Agent],\n      socketPath: undefined,\n      method: 'GET',\n      maxHeaderSize: undefined,\n      insecureHTTPParser: undefined,\n      path: '/api/status',\n
_ended: false,\n      res: null,\n      aborted: false,\n      timeoutCb: null,\n      upgradeOrConnect: false,\n      parser: null,\n      maxHeadersCount: null,\n      reusedSocket: false,\n      host
: 'kibana-url.example.com',\n      protocol: 'https:',\n      _redirectable: [Circular *1],\n      [Symbol(kCapture)]: false,\n      [Symbol(kNeedDrain)]: false,\n      [Symbol(corked)]: 0,\n
     [Symbol(kOutHeaders)]: [Object: null prototype]\n    },\n    _currentUrl: 'https://kibana-url.example.com:5601/api/status',\n    [Symbol(kCapture)]: false\n  },\n  response: undefined,\n
 isAxiosError: true,\n  toJSON: [Function: toJSON]\n}"}
{"type":"error","@timestamp":"2021-05-29T02:29:19+00:00","tags":[],"pid":947,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n    at HapiR
esponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:61:19)\n    at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34)\n    at ru
nMicrotasks (<anonymous>)\n    at processTicksAndRejections (internal/process/task_queues.js:93:5)\n    at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n    at exports.Manage
r.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n    at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n    at exports.execute (/usr
/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n    at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n    at Request._execute (/usr/share/kibana/node_
modules/@hapi/hapi/lib/request.js:279:9)"},"url":"https://kibana-url.example.com/_prometheus/metrics","message":"Internal Server Error"}
{"type":"response","@timestamp":"2021-05-29T02:29:19+00:00","tags":[],"pid":947,"method":"get","statusCode":500,"req":{"url":"/_prometheus/metrics","method":"get","headers":{"host":"kibana.missionlane-i
nfra-dev.com","pragma":"no-cache","cache-control":"no-cache","sec-ch-ua":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","sec-ch-ua-mobile":"?0","upgrade-insecure-requests
":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q
=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-enco
ding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,sv-SE;q=0.8,sv;q=0.7","x-forwarded-for":"10.X.X.X","x-forwarded-proto":"https","x-envoy-internal":"true","x-request-id":"8f174380-745f-9131-
9e9a-18906001b53b","x-envoy-expected-rq-timeout-ms":"60000","content-length":"0"},"remoteAddress":"10.X.X.X","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, l
ike Gecko) Chrome/90.0.4430.212 Safari/537.36"},"res":{"statusCode":500,"responseTime":51,"contentLength":97},"message":"GET /_prometheus/metrics 500 51ms - 97.0B"}

Here is the configuration:

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  annotations:
    association.k8s.elastic.co/es-conf: '{"authSecretName":"kibana-kibana-user","authSecretKey":"kibana-kibana-user","caCertProvided":true,"caSecretName":"kibana-kb-es-ca","url":"https://es-http.big-brother.svc:9200","version":"7.13.0"}'
    common.k8s.elastic.co/controller-version: 1.5.0
  name: kibana
spec:
  config:
    elasticsearch.ssl.certificateAuthorities: /etc/certs/ca.crt
    elasticsearch.ssl.verificationMode: none
    telemetry.enabled: false
    xpack.encryptedSavedObjects.encryptionKey: ...
    xpack.security.authc.providers:
      basic.basic1:
        order: 1
      oidc.oidc1:
        description: Log in with SSO
        order: 0
        realm: oidc1
    xpack.security.session.idleTimeout: 72h
    xpack.security.session.lifespan: 72h
  count: 3
  elasticsearchRef:
    name: elasticsearch
  http:
    service:
      metadata:
        creationTimestamp: null
      spec: {}
    tls:
      certificate:
        secretName: kibana-internal-certs
  podTemplate:
    metadata:
      creationTimestamp: null
    spec:
      containers:
      - name: kibana
        resources: {}
        volumeMounts:
        - mountPath: /etc/certs
          name: elasticsearch-certs
          readOnly: true
        - mountPath: /usr/share/kibana/plugins
          name: plugins
      initContainers:
      - command:
        - sh
        - -c
        - |
          bin/kibana-plugin install https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/7.13.0/kibanaPrometheusExporter-7.13.0.zip && /usr/share/kibana/bin/kibana --optimize
        name: install-plugins
        resources: {}
        volumeMounts:
        - mountPath: /usr/share/kibana/plugins
          name: plugins
      volumes:
      - name: elasticsearch-certs
        secret:
          secretName: es-validation-certs
      - emptyDir: {}
        name: plugins
  secureSettings:
  - secretName: kibana-secret-settings
  version: 7.13.0
status:
  associationStatus: Established
  availableNodes: 3
  health: green
  version: 7.13.0
faust64 commented 3 years ago

Hi,

The first thing that stands out is that Error: connect ECONNREFUSED 10.24.1.8:5601, alongside mentions of 'Host: kibana-url.example.com:5601

Does kibana-url.example.com resolve to 10.24.1.8? Is this a Service within your SDN? Ingress controllers? A LB in front of your cluster? Either way: is port 5601 exposed, or is there some port translation? Can you query that address from your kibana container - or another Pod?

pjhampton commented 3 years ago

@KnechtionsCoding just checking in to see if @faust64 answered your question? He seems to know lots about k8s :-)