APM Monitoring in Kibana throws error 500 in logs

[whyyouwannaknow]

Kibana version: 8.10.4

Elasticsearch version: 8.10.4

Server OS version: Debian 11

Browser version: Latest Chrome version

Browser OS version: Windows 10

Original install method (e.g. download page, yum, from source, etc.): apt

Describe the bug:

When APM Server is configured (via Elastic Agent / Kibana integration on an on-prem cluster), we get this error when trying to access the APM Monitoring status in Kibana :

`[2023-11-21T09:25:27.658+01:00][ERROR][plugins.monitoring.monitoring] TypeError: Cannot destructure property 'beats' of '(intermediate value)(intermediate value)(intermediate value)' as it is undefined. at handleResponse (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/lib/apm/get_apms.js:29:5) at getApms (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/lib/apm/get_apms.js:160:10) at processTicksAndRejections (node:internal/process/task_queues:95:5) at async Promise.all (index 1) at Object.handler (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/routes/api/v1/apm/instances.js:40:31) at handler (/usr/share/kibana/node_modules/@kbn/monitoring-plugin/server/plugin.js:315:28) at Router.handle (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:154:30) at handler (/usr/share/kibana/node_modules/@kbn/core-http-router-server-internal/src/router.js:113:50) at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28) at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20) at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20) at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32) at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

[2023-11-21T09:25:27.660+01:00][ERROR][http] 500 Server Error`

Steps to reproduce:

1. Install Elastic Agent via APT on a different server than Kibana or ElasticSearch nodes
2. Enroll this Elastic Agent has a Fleet Server
3. Add the APM Integration on this Fleet Server to give it the APM Server "role"
4. Go to the Monitoring tab of Kibana and see the HTTP 500 error thrown when clicking on APM Server

Expected behavior: Showing the correct usage of APM in the Monitoring Kibana setup

[elasticmachine]

Pinging @elastic/apm-ui (Team:APM)

[elasticmachine]

Pinging @elastic/infra-monitoring-ui (Team:Monitoring)

[miltonhultgren]

What confuses me is that APM Server would show up at all in the Stack Monitoring UI.

When Elastic Agent runs APM Server as a collector that is treated as a hidden detail of the agent, the same way we don't expose a way to monitor the underlying Beats processes that are used to collect metrics and logs.

@cmacknz Can you verify my understanding here that such an instance of APM Server should not appear in the Stack Monitoring UI?

[cmacknz]

When Elastic Agent runs APM Server as a collector that is treated as a hidden detail of the agent, the same way we don't expose a way to monitor the underlying Beats processes that are used to collect metrics and logs.

Yes this is the correct view of things. There is an Elastic Agent not an APM Server. Monitoring APM Server would be done with Fleet and the APM integration.

However there is a way to get at the underlying process metrics through agent if that helps, see https://www.elastic.co/guide/en/fleet/current/elastic-agent-container.html#_debugging. Specifically the http://localhost:6791/processes/{process-name} route documented there.