500 error

[davejhahn]

I've installed the plugin and when I try and call the endpoint (from the Kibana pod), I get a 500 error.

{"type":"error","@timestamp":"2020-09-04T02:29:32Z","tags":[],"pid":25,"level":"error","error":{"message":"404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}","name":"StatusCodeError","stack":"StatusCodeError: 404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}\n    at new StatusCodeError (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/errors.js:32:15)\n    at Request.plumbing.callback (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/plumbing.js:104:33)\n    at Request.RP$callback [as _callback] (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/plumbing.js:46:31)\n    at Request.self.callback (/opt/bitnami/kibana/node_modules/request/request.js:185:22)\n    at Request.emit (events.js:198:13)\n    at Request.<anonymous> (/opt/bitnami/kibana/node_modules/request/request.js:1161:10)\n    at Request.emit (events.js:198:13)\n    at IncomingMessage.<anonymous> (/opt/bitnami/kibana/node_modules/request/request.js:1083:12)\n    at Object.onceWrapper (events.js:286:20)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/_prometheus/metrics","path":"/_prometheus/metrics","href":"/_prometheus/metrics"},"message":"404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}"}

From the Kibana pod: curl http://127.0.0.1:5601/_prometheus/metrics

Plugin is installed on top of the Docker image from bitnami/kibana:

FROM bitnami/kibana:7.9.0-debian-10-r0
RUN /opt/bitnami/kibana/bin/kibana-plugin install https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/7.9.0/kibana-prometheus-exporter-7.9.0.zip

Any ideas on why I might be seeing this?

Thanks.

[pjhampton]

Hey @davejhahn

That's a 404, not a 500. That suggests it isn't installed which sounds right. A 5xx would suggest there is an internal error with the plugin or Kibana. A 4xx error suggests it's just not there. Can you run and confirm it's not there?

bin/kibana-plugin list

[davejhahn]

It shows that is installed:

./kibana-plugin list kibana-prometheus-exporter@7.9.0

Yes, the 400 error happens immediately before the 500 error. This is the result after running the curl command: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

In the log, there are two entries, a 400 which indicates it is doing something with the plugin, then a 500 error:

{"type":"error","@timestamp":"2020-09-04T20:13:17Z","tags":[],"pid":25,"level":"error","error":{"message":"404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}","name":"StatusCodeError","stack":"StatusCodeError: 404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}\n    at new StatusCodeError (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/errors.js:32:15)\n    at Request.plumbing.callback (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/plumbing.js:104:33)\n    at Request.RP$callback [as _callback] (/opt/bitnami/kibana/plugins/kibana-prometheus-exporter/node_modules/request-promise-core/lib/plumbing.js:46:31)\n    at Request.self.callback (/opt/bitnami/kibana/node_modules/request/request.js:185:22)\n    at Request.emit (events.js:198:13)\n    at Request.<anonymous> (/opt/bitnami/kibana/node_modules/request/request.js:1161:10)\n    at Request.emit (events.js:198:13)\n    at IncomingMessage.<anonymous> (/opt/bitnami/kibana/node_modules/request/request.js:1083:12)\n    at Object.onceWrapper (events.js:286:20)\n    at IncomingMessage.emit (events.js:203:15)\n    at endReadableNT (_stream_readable.js:1145:12)\n    at process._tickCallback (internal/process/next_tick.js:63:19)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/_prometheus/metrics","path":"/_prometheus/metrics","href":"/_prometheus/metrics"},"message":"404 - {\"statusCode\":404,\"error\":\"Not Found\",\"message\":\"Not Found\"}"}

{"type":"response","@timestamp":"2020-09-04T20:13:17Z","tags":[],"pid":25,"method":"get","statusCode":500,"req":{"url":"/_prometheus/metrics","method":"get","headers":{"host":"10.240.0.119:5601","user-agent":"Prometheus/2.18.2","accept":"application/openmetrics-text; version=0.0.1,text/plain;version=0.0.4;q=0.5,*/*;q=0.1","accept-encoding":"gzip","x-prometheus-scrape-timeout-seconds":"10.000000"},"remoteAddress":"::ffff:10.240.0.8","userAgent":"::ffff:10.240.0.8"},"res":{"statusCode":500,"responseTime":4,"contentLength":9},"message":"GET /_prometheus/metrics 500 4ms - 9.0B"}

[pjhampton]

Does Bitnami have their own version of Kibana? I know some compaies such as AWS maintain their own fork. Works fine for me, so I'm assuming some misconfiguration

[davejhahn]

It looks like there is an issue that has a workaround in the Bitnami version--@andresbono provided the workaround and I was able to get it working. If anyone else has this particular issue with the Bitnami chart, please see the thread on this above.

Thanks @andresbono and @pjhampton !

[pjhampton]

Thanks for the update @davejhahn! Glad you got it sorted

[andresbono]

We narrowed the issue a little bit more. If you set the server.host to :: to support IPv6, the metrics endpoint fails. The error doesn't seem specific to the Bitnami container though. It is reproducible with the Elastic one:

Dockerfile

FROM docker.elastic.co/kibana/kibana-oss:7.9.1
RUN bin/kibana-plugin install https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/7.9.1/kibana-prometheus-exporter-7.9.1.zip

docker-compose.yaml

version: '2.0'
services:
  elasticsearch:
    image: bitnami/elasticsearch:7.9.1
  kibana:
    build: .
    environment:
      - 'SERVER_HOST=::'
      - 'ELASTICSEARCH_HOSTS=http://elasticsearch:9200'

$ docker-compose up -d --build
$ docker-compose exec kibana curl http://localhost:5601/_prometheus/metrics
{"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred"}

It works fine if we set SERVER_HOST=0.0.0.0.

Is IPv6 supported for this plugin?

[pjhampton]

Thanks for the details @andresbono!

I've opened https://github.com/pjhampton/kibana-prometheus-exporter/issues/187 to investigate With ES 8.0.0 around the corner and Kibana re-architecture (https://www.elastic.co/blog/introducing-a-new-architecture-for-kibana) I might completely re-write this plugin. There are hangovers from the original hacked together product that need to be moved away from.