Closed tsullivan closed 3 years ago
tylersmalley
commented
4 years ago It looks like this was caused by receiving HTML from the URL configured for Elasticsearch. Since ES only returns HTML, I know the current/legacy ES client does throw an exception in this case.
<center><h1>502 Bad Gateway</h1></center>\\r\\n<hr><center>nginx/1.17.5</center>
tsullivan
commented
4 years ago I restarted Kibana and the proxy, and Kibana crashed again with the message:
FATAL [security_exception] failed to authenticate user [kibana]
``` Attaching to esd_kibana-7.5.0 kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: server"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: plugins"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: dev"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: server"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: elasticsearch"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: uiSettings"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: elasticsearch"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: logging"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: server"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: plugins"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: dev"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: kibana"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: migrations"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: uiSettings"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:29Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: logging"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","root"],"pid":6,"message":"setting up root"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","server"],"pid":6,"message":"setting up server"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-service"],"pid":6,"message":"Discovering plugins"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Discovering plugins..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Scanning \"/usr/share/kibana/src/plugins\" for plugin sub-directories..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Scanning \"/usr/share/kibana/x-pack/plugins\" for plugin sub-directories..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Scanning \"/usr/share/kibana/plugins\" for plugin sub-directories..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Scanning \"/usr/share/kibana-extra\" for plugin sub-directories..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"data\" at \"/usr/share/kibana/src/plugins/data\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"embeddable\" at \"/usr/share/kibana/src/plugins/embeddable\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"eui_utils\" at \"/usr/share/kibana/src/plugins/eui_utils\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"expressions\" at \"/usr/share/kibana/src/plugins/expressions\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"inspector\" at \"/usr/share/kibana/src/plugins/inspector\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"newsfeed\" at \"/usr/share/kibana/src/plugins/newsfeed\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"timelion\" at \"/usr/share/kibana/src/plugins/timelion\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"uiActions\" at \"/usr/share/kibana/src/plugins/ui_actions\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"advancedUiActions\" at \"/usr/share/kibana/x-pack/plugins/advanced_ui_actions\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"code\" at \"/usr/share/kibana/x-pack/plugins/code\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"features\" at \"/usr/share/kibana/x-pack/plugins/features\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"licensing\" at \"/usr/share/kibana/x-pack/plugins/licensing\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"security\" at \"/usr/share/kibana/x-pack/plugins/security\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"spaces\" at \"/usr/share/kibana/x-pack/plugins/spaces\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins-discovery"],"pid":6,"message":"Successfully discovered plugin \"translations\" at \"/usr/share/kibana/x-pack/plugins/translations\""} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","plugins","data"],"pid":6,"message":"\"/usr/share/kibana/src/plugins/data/server\" does not export \"config\"."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: timelion"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:30Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,code"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:31Z","tags":["debug","plugins","features"],"pid":6,"message":"\"/usr/share/kibana/x-pack/plugins/features/server\" does not export \"config\"."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:31Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: x-pack,licensing"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:33Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,security"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,spaces"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","plugins","translations"],"pid":6,"message":"\"/usr/share/kibana/x-pack/plugins/translations/server\" does not export \"config\"."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","plugins-service"],"pid":6,"message":"Discovered 15 plugins."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","http"],"pid":6,"message":"starting NotReady server"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","elasticsearch-service"],"pid":6,"message":"Setting up elasticsearch service"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","elasticsearch-service"],"pid":6,"message":"Creating elasticsearch clients"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","ui-settings-service"],"pid":6,"message":"Setting up ui settings service"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","plugins-service"],"pid":6,"message":"Setting up plugins service"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [15] plugins: [security,licensing,code,timelion,features,spaces,translations,uiActions,newsfeed,inspector,embeddable,advancedUiActions,expressions,eui_utils,data]"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:34Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"security\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","security"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,security"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["warning","plugins","security","config"],"pid":6,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","security","authentication"],"pid":6,"message":"Successfully initialized authenticator."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","security","authentication"],"pid":6,"message":"Successfully registered core authentication handler."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"licensing\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","licensing"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: x-pack,licensing"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"code\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","code"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,code"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"timelion\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","timelion"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: timelion"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"features\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","features"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"spaces\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","spaces"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: xpack,spaces"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"translations\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","translations"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins-system"],"pid":6,"message":"Setting up plugin \"data\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","data"],"pid":6,"message":"Initializing plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:54:35Z","tags":["debug","plugins","licensing"],"pid":6,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/x-pack","message":"Found plugin at /usr/share/kibana/x-pack"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","plugins","licensing"],"pid":6,"message":"Calling [data] Elasticsearch _xpack API. Polling frequency: 30001"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/apm_oss","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/apm_oss"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/dashboard_embeddable_container","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/dashboard_embeddable_container"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/data","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/data"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/elasticsearch","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/elasticsearch"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/embeddable_api","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/embeddable_api"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/expressions","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/expressions"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/input_control_vis","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/input_control_vis"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/inspector_views","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/inspector_views"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/interpreter","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/interpreter"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/kbn_doc_views","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/kbn_doc_views"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/console","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/console"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/kbn_vislib_vis_types","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/kbn_vislib_vis_types"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/kibana","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/kibana"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/kibana_react","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/kibana_react"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/navigation","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/navigation"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/newsfeed","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/newsfeed"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/region_map","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/region_map"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/state_session_storage_redirect","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/state_session_storage_redirect"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/status_page","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/status_page"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/telemetry","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/telemetry"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/tile_map","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/tile_map"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/timelion","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/timelion"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/ui_metric","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/ui_metric"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/vis_type_markdown","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/vis_type_markdown"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/vis_type_metric","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/vis_type_metric"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/vis_type_table","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/vis_type_table"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/vis_type_tagcloud","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/vis_type_tagcloud"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/vis_type_timeseries","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/vis_type_timeseries"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["debug","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/vis_type_vega","message":"Found plugin at /usr/share/kibana/src/legacy/core_plugins/vis_type_vega"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:09Z","tags":["warning","legacy-plugins"],"pid":6,"path":"/usr/share/kibana/src/legacy/core_plugins/visualizations","message":"Skipping non-plugin directory at /usr/share/kibana/src/legacy/core_plugins/visualizations"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","savedobjects-service"],"pid":6,"message":"Setting up SavedObjects service"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: kibana"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: migrations"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","root"],"pid":6,"message":"starting root"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","server"],"pid":6,"message":"starting server"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-service"],"pid":6,"message":"Plugins service starts plugins"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["info","plugins-system"],"pid":6,"message":"Starting [8] plugins: [security,licensing,code,timelion,features,spaces,translations,data]"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"security\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins","security"],"pid":6,"message":"Starting plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"licensing\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"code\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"timelion\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins","timelion"],"pid":6,"message":"Starting plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"features\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins","features"],"pid":6,"message":"Starting plugin"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"spaces\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"translations\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","plugins-system"],"pid":6,"message":"Starting plugin \"data\"..."} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","savedobjects-service"],"pid":6,"message":"Starting SavedObjects service"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["debug","config"],"pid":6,"message":"Marking config path as handled: migrations"} kibana_1 | {"type":"log","@timestamp":"2019-11-14T16:55:10Z","tags":["warning","plugins","licensing"],"pid":6,"message":"License information could not be obtained from Elasticsearch for the [data] cluster. Bad Gateway :: {\"path\":\"/_xpack\",\"statusCode\":502,\"response\":\"\\r\\n
tylersmalley
commented
4 years ago @tsullivan it's still getting a bad gateway and returning HTML. It seems like an issue with your proxy.
kibana_1 | FATAL Bad Gateway :: {"path":"/.kibana","query":{},"statusCode":502,"response":"<html>\r\n<head><title>502 Bad Gateway</title></head>\r\n<body>\r\n<center><h1>502 Bad Gateway</h1></center>\r\n<hr><center>nginx/1.17.5</center>\r\n</body>\r\n</html>\r\n"}
kibana_1 | FATAL [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate={ 0="Bearer realm=\"security\"" & 1="ApiKey" & 2="Basic realm=\"security\" charset=\"UTF-8\"" } } } :: {"path":"/.kibana","query":{},"statusCode":401,"response":"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}}],\"type\":\"security_exception\",\"reason\":\"failed to authenticate user [kibana]\",\"header\":{\"WWW-Authenticate\":[\"Bearer realm=\\\"security\\\"\",\"ApiKey\",\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"]}},\"status\":401}","wwwAuthenticateDirective":"Bearer realm=\"security\", ApiKey, Basic realm=\"security\" charset=\"UTF-8\""}
tylersmalley
commented
4 years ago Have you confirmed you have the correct credentials for the kibana user?
elasticmachine
commented
4 years ago Pinging @elastic/kibana-operations (Team:Operations)
tsullivan
commented
4 years ago It looks like this was caused by receiving HTML from the URL configured for Elasticsearch. Since ES only returns HTML, I know the current/legacy ES client does throw an exception in this case.
Wouldn't this be a non-issue if the ES client looked at the HTTP response code (200, 503, etc)?
tsullivan
commented
4 years ago I think since it is logged FATAL Bad Gateway in the message, it does understand the HTTP response code. To me, it looks like there is code using the client that should handle the failure response and is not, which crashes Kibana server's main event loop.
tsullivan
commented
3 years ago Marking this as stale since I don't think this issue persists in the latest code
jbudz
commented
3 years ago I'm going to close this out, lets reopen if it comes back.
Related to https://github.com/elastic/kibana/issues/25027
Kibana version: 7.5.0 BC
Server OS version: Docker Linux
Original install method (e.g. download page, yum, from source, etc.): Docker
Describe the bug: Start up multiple containers for the Elastic stack simultaneously, and the Kibana process will crash when it can't get a response from Elasticsearch on startup
Provide logs and/or server output (if relevant):