search

elastic / apm-server

https://www.elastic.co/guide/en/apm/guide/current/index.html
Other
1.17k stars 515 forks source link

Validate go-docappender metrics are produced when monitoring is enabled #13244

Open kruskall opened 1 month ago

kruskall commented 1 month ago

We should make sure that go-docappender metrics are produced when monitoring is enabled.

flush latency and similar metrics are important to debug issues

carsonip commented 1 month ago

go-docappender itself exposes metrics as OTel metrics, as opposed to elastic agent monitoring where EA polls metrics from APM server. The go-docappender metrics should be available if instrumentation.enabled is true in apm-server.yml because apm-server configures otel bridge.

carsonip commented 1 month ago

I looked into this again. apm-server does get stats from the appender and publish libbeat monitoring metrics, other than the fact that go-docappender publishes OTel metrics itself (what I mentioned in the previous comment).

The observation is that there are no output.elasticsearch.bulk_requests metrics published by this code, but I see output.events.* metrics. This may reveal a bug somewhere. I wonder if using the monitoring.Default registry is causing a problem. This issue has to be investigated.

carsonip commented 3 weeks ago

Might be related to https://github.com/elastic/apm-server/issues/8383

carsonip commented 2 weeks ago

Observation: when monitoring.enabled: true and monitoring.elasticsearch is configured, the metrics are sent to ES directly under .monitoring-beats-7-*. The output.elasticsearch.* fields exist in source but are unmapped.

carsonip commented 2 weeks ago

In that case there may be 2 problems:

  1. the .monitoring-beats-7-* does not have the mappings for some of our fields
  2. metricbeat beat module may be ignoring the fields because some of our fields do not fall into "exported fields" defined by metricbeat. Pending confirmation.
carsonip commented 1 week ago

metricbeat beat module may be ignoring the fields because some of our fields do not fall into "exported fields" defined by metricbeat. Pending confirmation.

(2) is not right. The mapping is actually defined in EA integration. Created https://github.com/elastic/apm-server/issues/13475 to take care of missing metrics due to EA integration apm-server metrics mapping.