search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.56k stars 8.08k forks source link

CSS randomly blocked in Kibana when using self-signed certificates with error "net::ERR_CONNECTION_RESET" #32754

Closed xucito closed 5 years ago

xucito commented 5 years ago

Kibana version: docker.elastic.co/kibana/kibana:6.6.1 Elasticsearch version: docker.elastic.co/elasticsearch/elasticsearch:6.6.1 Server OS version: Centos 7.4 Browser version: Google Chrome Version 72.0.3626.121 (Official Build) (64-bit) Browser OS version: Windows 10 Original install method (e.g. download page, yum, from source, etc.): Docker Describe the bug: When accessing Kibana, css files are reporting ERR_CONNECTION_RESET Steps to reproduce:

  1. Run Elasticsearch and Kibana with SSL/TLS enabled using self-signed certificate
  2. Use a self-signed certificate
  3. Access Kibana using chrome

Expected behavior:

  1. No issue loading Kibana

Screenshots (if relevant): Error I receive image Errors in browser console (if relevant): Example:

GET https://10.10.10.32:5601/plugins/tagcloud/index.css net::ERR_CONNECTION_RESET

Provide logs and/or server output (if relevant): Nothing suspicious in logs.

{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/rollup/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /plugins/rollup/index.css 200 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/index_lifecycle_management/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":2,"contentLength":9},"message":"GET /plugins/index_lifecycle_management/index.css 200 2ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/index_management/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /plugins/index_management/index.css 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/license_management/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":10,"contentLength":9},"message":"GET /plugins/license_management/index.css 200 10ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/canvas/style/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":11,"contentLength":9},"message":"GET /plugins/canvas/style/index.css 200 11ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/dlls/vendors.style.dll.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":13,"contentLength":9},"message":"GET /dlls/vendors.style.dll.css 200 13ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/watcher/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /plugins/watcher/index.css 200 5ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/ml/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":18,"contentLength":9},"message":"GET /plugins/ml/index.css 200 18ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/searchprofiler/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":6,"contentLength":9},"message":"GET /plugins/searchprofiler/index.css 200 6ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/spaces/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":6,"contentLength":9},"message":"GET /plugins/spaces/index.css 200 6ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/monitoring/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":9,"contentLength":9},"message":"GET /plugins/monitoring/index.css 200 9ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/graph/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":11,"contentLength":9},"message":"GET /plugins/graph/index.css 200 11ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/commons.style.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":28,"contentLength":9},"message":"GET /bundles/commons.style.css 200 28ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/login.style.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /bundles/login.style.css 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/plugins/vega/index.css","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"text/css,*/*;q=0.1","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":6,"contentLength":9},"message":"GET /plugins/vega/index.css 200 6ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:52Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/commons.bundle.js","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"*/*","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":234,"contentLength":9},"message":"GET /bundles/commons.bundle.js 200 234ms - 9.0B"}
{"type":"response","@timestamp":"2019-03-08T11:52:53Z","tags":[],"pid":1,"method":"get","statusCode":200,"req":{"url":"/bundles/login.bundle.js","method":"get","headers":{"host":"10.10.10.32:5601","connection":"keep-alive","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.121 Safari/537.36","accept":"*/*","referer":"https://10.10.10.32:5601/login?next=%2F","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9,en-AU;q=0.8"},"remoteAddress":"10.10.10.1","userAgent":"10.10.10.1","referer":"https://10.10.10.32:5601/login?next=%2F"},"res":{"statusCode":200,"responseTime":5,"contentLength":9},"message":"GET /bundles/login.bundle.js 200 5ms - 9.0B"}

Any additional context:

  1. I have disabled TLS 1.3 for testing
  2. I have disabled all my extensions
  3. I have reset all my browser caches
  4. I can replicate this issue in two different elasticsearch environments (one using docker and one hosted on-prem)
  5. My computer can open Kibana using Internet Explorer and Edge
  6. I can access my Kibana instances that are not using self-signed certificates using Chrome
elasticmachine commented 5 years ago

Pinging @elastic/kibana-operations

xucito commented 5 years ago

To add to this issue, it seems accessing the identical cluster via a public IP does not cause this issue, it only happens if I use the local ip address from another machine within the same network (i.e. VPN'd device).

aeijdenberg commented 5 years ago

I'm seeing similar since updating from a 6.4.x to 6.5.4 yesterday on an AWS hosted Elasticsearch.

Page loads OK in browser, basic searches work, but then (maybe 30 seconds after initial page load?). Page resets with Kibana did not load properly. Check the server output for more information. big red error message. Chrome then shows ERR_CONNECTION_RESET messages in the console for static assets such as kibana.style.css, manifest.json,...

(Note we have our own proxy in front that signs the AWS requests with AWS credentials, and right now I'm kubectl port-forwarding to get there, so a few extra potential culprits in the serving path - but this previously worked fine prior to the update)

spalger commented 5 years ago

@xucito @aeijdenberg could either of you provide copies of the assets that are regularly getting blocked, and any proxy configuration you can share? I'm 99% sure this happening at the proxy layer and not in Kibana, but maybe we can help figure out what change between 6.4 and 6.5/6.6 could have triggered the new behavior.

aeijdenberg commented 5 years ago

@spalger - sadly for this bug, I'm no longer able to reproduce. This occurred for a few hours yesterday quite consistently, and then stopped. I wonder if maybe something was cached somewhere that has now cleared?

Thank you for the follow-up - if we see this re-occur I'll update this issue.

spalger commented 5 years ago

In that case I'll close the issue for now, if either of you can reproduce later feel free to reopen here.