search

pjhampton / kibana-prometheus-exporter

Prometheus metrics for Kibana
Apache License 2.0
114 stars 37 forks source link

Internal Server Error when Installing 7.10.0 in ECK Kibana #217

Closed knechtionscoding closed 3 years ago

knechtionscoding commented 3 years ago

Installing the plugin and running in K8s I'm getting an internal 500 when trying to hit /_prometheus/metrics I'm using SSO, but I've validated that the behavior exists whether I use direct authentication or sso, and I've validated that I have valid SSO credentials at the time of hitting the endpoint.

Dockerfile

FROM docker.elastic.co/kibana/kibana:7.10.0

ENV KIBANA_VERSION 7.10.0
RUN /usr/share/kibana/bin/kibana-plugin install https://github.com/pjhampton/kibana-prometheus-exporter/releases/download/${KIBANA_VERSION}/kibanaPrometheusExporter-${KIBANA_VERSION}.zip
RUN /usr/share/kibana/bin/kibana --optimize

Kibana config

    telemetry.enabled: false
    elasticsearch.ssl.certificateAuthorities: /etc/certs/ca.crt
    elasticsearch.ssl.verificationMode: none
    xpack.encryptedSavedObjects.encryptionKey: <key>
    xpack.security.session.lifespan: 72h
    xpack.security.session.idleTimeout: 72h
    xpack.security.authc.providers:
      oidc.oidc1:
        order: 0
        realm: oidc1
        description: "Log in with SSO"
      basic.basic1:
        order: 1

Error message: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}

Kibana Error message:

{"type":"response","@timestamp":"2021-03-26T13:37:29Z","tags":[],"pid":6,"method":"get","statusCode":500,"req":{"url":"/_prometheus/metrics","method":"get","headers":{"host":"<url>","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","sec-ch-ua":"\"Google Chrome\";v=\"89\", \"Chromium\";v=\"89\", \";Not A Brand\";v=\"99\"","sec-ch-ua-mobile":"?0","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,sv-SE;q=0.8,sv;q=0.7","x-forwarded-for":"10.24.1.56","x-forwarded-proto":"https","x-envoy-internal":"true","x-request-id":"28856723-e40c-418d-8afc-10be4f7b4077","x-envoy-expected-rq-timeout-ms":"60000","content-length":"0"},"remoteAddress":"10.24.132.182","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36"},"res":{"statusCode":500,"responseTime":57,"contentLength":9},"message":"GET /_prometheus/metrics 500 57ms - 9.0B"}

Start up log showing the kibanaPrometheusExporter installed

{"type":"log","@timestamp":"2021-03-26T13:22:22Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [96] plugins: [taskManager,licensing,globalSearch,globalSearchProviders,code,usageCollect
ion,xpackLegacy,telemetryCollectionManager,telemetryCollectionXpack,kibanaUsageCollection,securityOss,newsfeed,mapsLegacy,kibanaLegacy,translations,share,legacyExport,embeddable,uiActionsEnhanced,esUiSh
ared,expressions,data,home,observability,cloud,console,consoleExtensions,apmOss,searchprofiler,painlessLab,grokdebugger,management,indexPatternManagement,advancedSettings,fileUpload,savedObjects,dashboa
rd,visualizations,visTypeVega,visTypeTimelion,timelion,features,upgradeAssistant,security,snapshotRestore,enterpriseSearch,encryptedSavedObjects,ingestManager,indexManagement,remoteClusters,crossCluster
Replication,indexLifecycleManagement,dashboardMode,beatsManagement,transform,ingestPipelines,maps,licenseManagement,graph,dataEnhanced,visTypeTable,visTypeMarkdown,tileMap,regionMap,inputControlVis,visu
alize,kibanaPrometheusExporter,bfetch,canvas,charts,lens,visTypeVislib,visTypeTimeseries,rollup,visTypeTagcloud,visTypeMetric,watcher,discover,discoverEnhanced,savedObjectsManagement,spaces,reporting,li
sts,eventLog,actions,case,alerts,stackAlerts,triggersActionsUi,ml,securitySolution,infra,monitoring,logstash,apm,uptime]"}

Curious what I might be doing wrong

pjhampton commented 3 years ago

Howdy @KnechtionsCoding

Thanks for your issue + all the details. This is super helpful.

I do know what the problem is :-( It's the same as #215 - This plugin had to do a complete rewrite @ 7.10.0 because the Kibana team broken backward compatibility without warning - which is fine I suppose. I can imagine this is the only plugin that actually uses the stats APIs outside of Kibana itself.

I will try to find the time to rewrite a fix for it in the next couple of days.

knechtionscoding commented 3 years ago

Good to know that I wasn't just doing something wrong!

No super rush, I'll keep an eye on this ticket and wait for you to finish it up.

pjhampton commented 3 years ago

This should be fixed now @KnechtionsCoding! Apologies for the delay - life has been busy https://github.com/pjhampton/kibana-prometheus-exporter/commit/6a9a066bb390df01757c49f4c731b709a65f64d3#diff-8885c91b939c521c41b1df64d6048a6f5ef31e7776088c7fc45f09dc616c721bR3

If you try to install 7.10.0 again it should be fixed. Let me know if you are having any other issues