search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.69k stars 8.12k forks source link

Canvas not playing nice with our SA Demo environment #28181

Closed jamiesmith closed 4 years ago

jamiesmith commented 5 years ago

Kibana version: 6.5.0

Elasticsearch version: 6.5.0

Server OS version: ECE

There isn't much data on these clusters - We are set up to have 32gb RAM for Elasticsearch, and 4gb RAM for the Kibana instance.

Browser version: Chrome Version 71.0.3578.98 (Official Build) (64-bit) and Safari Version 12.0.1 (14606.2.104.1.1)

Browser OS version: MacOS 10.14.1

Original install method (e.g. download page, yum, from source, etc.): ECE/6.5.0 stack pack Note that the ECE demo env runs behind an oauth proxy

Describe the bug:

This is really an "across the board" set of things that keep happening with two of our canvas demos in the SA demo environment. General performance issues, errors, 404's, websocket retries, slow/not loading workpads-- even just getting the list of workpads. The issues of things not loading at all is intermittent, but the slowness persists - in the middle of writing this ticket it started working again, then started failing again (failing, meaning it never finished loading the workpad or list of workpads). I have tried several different workpads. Often, just the the "canvas is loading" thing shows up, or it gets stuck on a workpad with the clocks and never finishes loading.

As I was typing this, it started working, then got back into another stall: image

Steps to reproduce:

Visit the IoT or canvas demos in the SA Demo Env (ping jamie.smith on slack for the address of them if you don't know them)

Expected behavior:

Screenshots (if relevant): This page got stuck in this state (3 minutes, never finished):

image

with nothing other than the websocket errors and one of these in the console: canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWewhO4&sid=0i_CNJSnL9yUHxCZAIBo net::ERR_CONNECTION_CLOSED

Errors in browser console (if relevant): We continually get the WebSocket error, which is expected because of the proxy, but I don't know why it keeps trying a websocket when it's not supported.

Various permutations of these errors in chrome:

canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=DTxb1ZRfwRVTOts8AH8W' failed: 
canvas.bundle.js:15 GET https://canvas.<redacted>/api/canvas/plugins?type=datasourceUIs net::ERR_CONNECTION_RESET 200
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=9QmazQ8Ccgo0lAkaAH8Z' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWegtnU&sid=DTxb1ZRfwRVTOts8AH8W 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=9qdIJEoRK1kI_cTnAH8b' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=NqjtI6OmdNxJyW64AH8d' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=L36ku1AG39S9ZUGUAH8e' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWehElM&sid=NqjtI6OmdNxJyW64AH8d 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=ZG4UxtJ3j80Xu3-GAH8g' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=l2DEYM1sNgaaq_CRAH8i' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=W_dILKIzJHZj9OOkAH8l' failed: 
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=O-_bVem4tVy2NL7lAH8o' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=ZNMLw0Dh8191hah-AH8r' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWehxc4&sid=O-_bVem4tVy2NL7lAH8o 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=V3gxfrKpTCMI0OgiAH8u' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=G2Qaw70UtBQyLBLMAH8x' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWeiC0h&sid=V3gxfrKpTCMI0OgiAH8u 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=PiM0ZiHi3YM_z7PtAH80' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=DYX7poRNw9O2klSXAH81' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=Xf_bVS2HiLqMHJS8AH84' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWeialI&sid=DYX7poRNw9O2klSXAH81 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=DmXMppp8H4uPk6PMAH86' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWeiiQo&sid=Xf_bVS2HiLqMHJS8AH84 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=CJxbrtmdUvKe04olAH89' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=5kx3JRtS9GhBQRq-AH8_' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWeixnf&sid=CJxbrtmdUvKe04olAH89 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=MoMtbhuZiCZwXF6nAH9C' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWejB8_&sid=MoMtbhuZiCZwXF6nAH9C net::ERR_CONNECTION_CLOSED
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=Zd5Q0LsdAJFj-ghXAH9D' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWej3VQ&sid=5kx3JRtS9GhBQRq-AH8_ 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=5bcl4CdouRF4kLO1AH9G' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=2DW3XbCcQTG4sGxHAH9J' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=wnESc7wrDqeclS0XAH9L' failed: 
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWejfHs&sid=wnESc7wrDqeclS0XAH9L 500
canvas.bundle.js:4 POST https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWejhnJ&sid=wnESc7wrDqeclS0XAH9L 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=dZN9hk6DvDXxI5vmAH9O' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=JvRFiHb2KSPhpWGiAH9R' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=Csb7G7Pf2xOv9ybdAH9S' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWejq1C&sid=JvRFiHb2KSPhpWGiAH9R 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=qqkFDihXQkgYavIqAH9V' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=0AwvUfyESnnWEEosAH9Y' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWek3bJ&sid=qqkFDihXQkgYavIqAH9V 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=C5lAe5veZ4gtt0DgAH9b' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWekBCJ&sid=0AwvUfyESnnWEEosAH9Y 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=Fu_zZXoF1hiNFag7AH9d' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWekJ16&sid=C5lAe5veZ4gtt0DgAH9b 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWekRAF&sid=Fu_zZXoF1hiNFag7AH9d 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=Q_UkXf64XhXV8OUIAH9g' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=8594NBDYUITHU4B4AH9i' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=3_lINYaGEa9VjXLgAH9l' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=fAv2nzBuKcicZ_QsAH9o' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=eGpuQfVr73xQbRY9AH9r' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=CCgqVvjmT_LWzco_AH9s' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=WSOb3CoKIqO4tYhcAH9v' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=gw00GSlkRhxTPqPPAH9z' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=H8s8hN8bqRGve1ulAH93' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWelblA&sid=H8s8hN8bqRGve1ulAH93 net::ERR_CONNECTION_CLOSED
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=V9Pul5ekGK892K9IAH97' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=94EPQosiQkMBNgIyAH99' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=O29ADMdlpTMpf2YTAH-A' failed: 
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWelq0L&sid=94EPQosiQkMBNgIyAH99 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=EI33H-b8_jjFX6jZAH-E' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=lR78S34n87dx6Po_AH-H' failed: 
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWemAWs&sid=EI33H-b8_jjFX6jZAH-E 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWemP0t&sid=lR78S34n87dx6Po_AH-H net::ERR_CONNECTION_CLOSED
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=3WTHcyHXHn36FALqAH-K' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=IohuFfJ99dv_clTwAH-N' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=gpA5FNWwDg9znSBKAH-Q' failed: Error during WebSocket handshake: Unexpected response code: 400
canvas.bundle.js:4 GET https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWemm1h&sid=gpA5FNWwDg9znSBKAH-Q 500
canvas.bundle.js:4 POST https://canvas.<redacted>/socket.io/?EIO=3&transport=polling&t=MWemmlX&sid=gpA5FNWwDg9znSBKAH-Q 400
canvas.bundle.js:4 WebSocket connection to 'wss://canvas.<redacted>/socket.io/?EIO=3&transport=websocket&sid=l5EQWOZ-NDCXuQTNAH-R' failed: Error during WebSocket handshake: Unexpected response code: 400

And some different ones in safari, on a different load:

[Error] Failed to load resource: the server responded with a status of 404 () (tinymath.min.js.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (performance-now.js.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (performance-now.js.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (tinymath.min.js.map, line 0)
[Error] Failed to load resource: the server responded with a status of 404 () (tinymath.min.js.map, line 0)

Provide logs and/or server output (if relevant): If I get access to the ES/Kibana logs for the timeframe (07-jan-2019 @ ~9:10 am EST) I will add them

Any additional context:

Note that this is intermittent. Canvas loads are usually pretty slow, even for small ones (much, much slower than running locally). Some more visibility as to what is going on would be helpful. In this case, the "canvas is loading" was on the screen for a couple of minutes before I finally opened up devtools to see that it was looping

Also pinging @mindbat who does a lot of work with the SA demo env.

elasticmachine commented 5 years ago

Pinging @elastic/kibana-canvas

alexfrancoeur commented 5 years ago

Thanks for filing @jamiesmith. @w33ble @clintandrewhall this environment might be a good way to reproduce intermittent performance issues on our cloud environment.

clintandrewhall commented 5 years ago

Thanks @alexfrancoeur ... this is really detailed and helpful... you're awesome, @jamiesmith !!

timductive commented 4 years ago

Closing this as all of these issues should be solved in the 7.x series, please re-open if not.