search

elastic / kibana

Your window into the Elastic Stack
https://www.elastic.co/products/kibana
Other
19.62k stars 8.22k forks source link

[Monitoring] Logstash/Beats/APM Server Standalone Cluster monitoring displays a blank page if accessed through CCS #96189

Open hendry-lim opened 3 years ago

hendry-lim commented 3 years ago

Kibana version: 7.12.0 7.12.1 7.13.x 7.14.0

Elasticsearch version: 7.12.0 7.12.1 7.13.x 7.14.0

Browser version: Firefox 87.0 88.0

Describe the bug: No Logstash nodes are displayed on Stack Monitoring on Overview, Nodes, and Pipelines if Logstash is under a Standalone Cluster and accessed through CCS.

Steps to reproduce:

  1. Set up and run Logstash to output to non-Elasticsearch output
  2. Set up Logstash monitoring using Metricbeat
  3. Set up CCS from a different cluster to access Logstash monitoring data
  4. Go to Stack Monitoring --> Standalone Cluster image
  5. Click on either Overview, Nodes, or Pipelines
  6. Observe a blank list/charts are displayed image image

It's working fine Logstash that sends data directly to an Elasticsearch cluster.

Expected behavior: Charts/list should be populated with relevant Logstash instances and metrics.

Relates to #93617

Tried with 7.13.0-SNAPSHOT Direct access to the cluster that stores the monitoring data image

Access to the monitoring data via CCS image

No error for Beats, but observed the following errors for APM Server

Stack Monitoring standalone cluster error ``` POST http://localhost:5601/api/monitoring/v1/clusters/__standalone_cluster__/apm/instances TypeError: data is undefined ApmServerInstances http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 ds http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 xa http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 fl http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 ml http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 ol http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 Gr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 unstable_runWithPriority http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:398 qr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 Gr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 Vr http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 el http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 Wl http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 Gl http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 render http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:390 renderReact http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 renderComponent http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 controller http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 $digest http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 $apply http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 updateData http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 promise callback*base_controller_MonitoringViewBaseController/this.updateData http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 base_controller_MonitoringViewBaseController http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 $eval http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 applyAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 C http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 $eval http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 $apply http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 completeTask http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 setTimeout handler*cn/o.defer http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 $applyAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 titleProvider http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 base_controller_MonitoringViewBaseController http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 base_eui_table_controller_MonitoringViewBaseEuiTableController http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 controller http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 instantiate http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 link http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 he http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 Pe http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 se http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 le http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 oe http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 ge http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 _ http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 update http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 $broadcast http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 commitRoute http://localhost:5601/40058/bundles/plugin/monitoring/8.0.0/monitoring.chunk.1.js:30 u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 $digest http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 evalAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 completeTask http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 i http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 setTimeout handler*cn/o.defer http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 $evalAsync http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 e http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 h http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 resolve http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 C http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 a http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 y http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 C http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 onload http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 y http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 y http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 u http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 $digest http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 get http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 x http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 T http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 x http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 S http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:375 dispatch http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:353 handle http://localhost:5601/40058/bundles/kbn-ui-shared-deps/kbn-ui-shared-deps.js:353 kbn-ui-shared-deps.js:390:77072 ```
elasticmachine commented 3 years ago

Pinging @elastic/stack-monitoring (Team:Monitoring)

hendry-lim commented 3 years ago

Issue persists in 7.15.0-SNAPSHOT.

Kibana error log when viewing APM server instances under Standalone Cluster ``` { "type": "response", "@timestamp": "2021-08-21T14:22:18+00:00", "tags": [], "pid": 1215, "method": "post", "statusCode": 200, "req": { "url": "/api/monitoring/v1/clusters/__standalone_cluster__", "method": "post", "headers": { "host": "localhost:5601", "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0", "accept": "application/json, text/plain, */*", "accept-language": "en-US,en;q=0.5", "accept-encoding": "gzip, deflate", "referer": "http://localhost:5601/app/monitoring", "kbn-system-request": "true", "content-type": "application/json;charset=utf-8", "kbn-version": "7.15.0-SNAPSHOT", "content-length": "101", "origin": "http://localhost:5601", "dnt": "1", "connection": "keep-alive", "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin" }, "remoteAddress": "172.20.0.1", "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0", "referer": "http://localhost:5601/app/monitoring" }, "res": { "statusCode": 200, "responseTime": 88, "contentLength": 406 }, "message": "POST /api/monitoring/v1/clusters/__standalone_cluster__ 200 88ms - 406.0B" }{ "type": "log", "@timestamp": "2021-08-21T14:22:18+00:00", "tags": ["error", "plugins", "monitoring", "monitoring"], "pid": 1215, "message": "TypeError: Cannot destructure property 'beats' of '(intermediate value)(intermediate value)(intermediate value)' as it is undefined.\n at handleResponse (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/apm/get_apms.js:46:5)\n at getApms (/usr/share/kibana/x-pack/plugins/monitoring/server/lib/apm/get_apms.js:181:10)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async Promise.all (index 1)\n at Object.handler (/usr/share/kibana/x-pack/plugins/monitoring/server/routes/api/v1/apm/instances.js:51:31)\n at handler (/usr/share/kibana/x-pack/plugins/monitoring/server/plugin.js:411:28)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)\n at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)" }{ "type": "log", "@timestamp": "2021-08-21T14:22:18+00:00", "tags": ["error", "http"], "pid": 1215, "message": "Error: Cannot wrap an error\n at new module.exports (/usr/share/kibana/node_modules/@hapi/hoek/lib/error.js:23:19)\n at Object.module.exports [as assert] (/usr/share/kibana/node_modules/@hapi/hoek/lib/assert.js:20:11)\n at Toolkit.response (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:191:18)\n at HapiResponseAdapter.toSuccess (/usr/share/kibana/src/core/server/http/router/response_adapter.js:97:43)\n at HapiResponseAdapter.toHapiResponse (/usr/share/kibana/src/core/server/http/router/response_adapter.js:86:19)\n at HapiResponseAdapter.handle (/usr/share/kibana/src/core/server/http/router/response_adapter.js:73:17)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:164:34)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)" }{ "type": "error", "@timestamp": "2021-08-21T14:22:18+00:00", "tags": [], "pid": 1215, "level": "error", "error": { "message": "Internal Server Error", "name": "Error", "stack": "Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:61:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:177:34)\n at runMicrotasks ()\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)\n at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)\n at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)\n at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)\n at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:370:32)\n at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:279:9)" }, "url": "http://localhost:5601/api/monitoring/v1/clusters/__standalone_cluster__/apm/instances", "message": "Internal Server Error" }{ "type": "response", "@timestamp": "2021-08-21T14:22:18+00:00", "tags": [], "pid": 1215, "method": "post", "statusCode": 500, "req": { "url": "/api/monitoring/v1/clusters/__standalone_cluster__/apm/instances", "method": "post", "headers": { "host": "localhost:5601", "user-agent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0", "accept": "application/json, text/plain, */*", "accept-language": "en-US,en;q=0.5", "accept-encoding": "gzip, deflate", "referer": "http://localhost:5601/app/monitoring", "content-type": "application/json;charset=utf-8", "kbn-version": "7.15.0-SNAPSHOT", "content-length": "81", "origin": "http://localhost:5601", "dnt": "1", "connection": "keep-alive", "sec-fetch-dest": "empty", "sec-fetch-mode": "cors", "sec-fetch-site": "same-origin" }, "remoteAddress": "172.20.0.1", "userAgent": "Mozilla/5.0 (X11; Linux x86_64; rv:90.0) Gecko/20100101 Firefox/90.0", "referer": "http://localhost:5601/app/monitoring" }, "res": { "statusCode": 500, "responseTime": 57, "contentLength": 97 }, "message": "POST /api/monitoring/v1/clusters/__standalone_cluster__/apm/instances 500 57ms - 97.0B" } ```