carsonip commented 5 months ago

Discovered by #13244

EA managed apm-server is only shipping a subset of apm-server monitoring metrics. This limits observability of EA managed apm-server. The problem is spread over the following use cases:

APM server self-monitoring
- metricsets state, stats
- indices (not data stream) .monitoring-beats-7-*
- does NOT use TSDS, but fields are unmapped due to dynamic: false
- code in ES repo monitoring-beats.json
Stack monitoring: metricbeat beat-xpack module uses index template embedded in ES
- metricsets state, stats
- data stream .monitoring-beats-8-mb
- use case: Elastic Cloud self-monitoring, metricbeat standalone beat-xpack
- does NOT use TSDS, but fields are unmapped due to dynamic: false
- code in ES repo monitoring-beats-mb.json
Metricbeat without xpack: metricbeat beat module
- metricsets state, stats
- data stream metricbeat-*
- use case: metricbeat standalone
- does NOT use TSDS, fields are mapped dynamically
- code in beats repo
EA integration
- metricsets stats only
- data stream metrics-elastic_agent.apm_server.* (see code)
- use case: elastic-agent standalone with monitoring
- to see EA integration package content: POST kbn:/api/fleet/epm/packages/elastic_agent
- uses TSDS, see https://github.com/elastic/integrations/pull/7214 . The issue with using TSDS is that unmapped fields are dropped, as it uses synthetic source.
- code in integrations repo
- it doesn't contain any apm-server fields as of today

Unknowns:

As the outdated mappings above contain fields that are no longer used, mapping generated from apm-server:5066/stats will not be pure addition. Not sure if dropping a field mapping will be problematic.
- Should it be a superset of all historically available monitoring fields? That doesn't sound realistic either

carsonip commented 4 months ago

Testing

Completed local testing on all https://github.com/elastic/beats/pull/40127, https://github.com/elastic/elasticsearch/pull/110568, and https://github.com/elastic/integrations/pull/10414. Note that both https://github.com/elastic/elasticsearch/pull/110568 and https://github.com/elastic/integrations/pull/10414 require https://github.com/elastic/beats/pull/40127 to have output.elasticsearch.* metrics parsed.

apm-server self monitoring (indices `.monitoring-beats-7-*`)

before

after

metricbeat beat-xpack (DS `.monitoring-beats-8-mb`)

before

after

metricbeat beat standalone (DS `metricbeat-*`)

before

after

EA agent monitoring (DS `metrics-elastic_agent-apm_server-*`)

before

after

Long term solution

While the Python script in https://github.com/elastic/apm-server/pull/13638 generates the correct mapping for all the above use cases, this approach is not very maintainable. Ideally, having dynamic: true like metricbeat standalone enables apm-server monitoring fields to be mapped dynamically and will require minimal maintenance. However, since EA agent mapping already has TSDB enabled, disabling TSDB now and enabling dynamic mapping sounds like a step back.

lahsivjar commented 4 months ago

Tested on 8.15 BC2, (UPDATE) followup testing with 8.15 BC3:

[x] APM server self-monitoring: tested with ESS elasticsearch, ~❌ the backport PR was not merged. It is merged now, and will require testing again in BC3.~ Tested with updated BC and is working as expected.

Snapshots from testing
[x] APM server stack monitoring: tested on ESS, ~❌ the backport PR was not merged. It is merged now, and will require testing again in BC3.~ Tested with updated BC and is working as expected.

Snapshots from testing
[x] Metricbeat without x-pack, tested locally with apm-server 008b12a7e9929ce00e01040e67e83cf19c0b7de7 and beats 417ac43a6193bdf26da490d41532b2f2d5bc3f70

Snapshots from testing
[x] EA integration tested with locally running EA using elastic-agent-package 475d247d16a94af14b75d991a8da3d63d68e8042. ~❌ While the test succeeded, I had to manually upgrade EA integration as the version bundled in Kibana is still older (v1.20.0) whereas the latest version is v2.0.1 - since this is not related to the current issue, marking this point as done.~ Confirmed that with BC3 the latest version v2.0.1 of elastic-agent integration is bundled.

Snapshots from testing