Fix the telemetry collection of Logstash with metricbeat monitoring.

mashhurs commented 2 weeks ago

Summary

Telemetry data collection is broken for Logstash, monitoring with metricbeat. This PR change covers following issues faced:

1) Resolve cluster UUID

With self monitoring, KB creates .monitoring-es* index with mapping type field and defaults to type:cluster_state key-value. It uses type:cluster_state condition when fetching cluster UUID.
However, with metricbeat, this type field doesn't exist under mapping which metricbeat creates, so cluster UUID will not be resolved as query is wrong (results empty output).

2) type field mismatch in (especially in state) queries, also collapse field

metricbeat sends the state event with metricset.name:node and state fetch query doesn't care about this condition, instead uses legacy type:logstash_state condition which is incorrect.
in the queries, collapse field is not correct: it is due to data shape change from legacy to metricbeat monitoring and queries are tightly coupled with legacy one (1, 2, 3)
in the queries, filter_path is also not correct: in both state query and stats query

3) Logstash state data frequency

metricbeat sends state event (_node/stats/pipeline?graph=true) data once but legacy frequently sends. KB telemetry fetcher queries against ES with latest update period where state data will not be available. It might be a reasonable design which results network efficiency. If that's the case, we should decide the expectation
- to still collect: we have to tune the query to collect the state data.
- leave it as an empty assuming state didn't change (personally, I would not align with this option since collecting once it a data loss risky)

Checklist

Delete any items that are not applicable to this PR.

~~[ ] Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support~~
~~[ ] Documentation was added for features that require explanation or tutorials~~
[x] Unit or functional tests were updated or added to match the most common scenarios
~~[ ] Flaky Test Runner was used on any tests changed~~
~~[ ] Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)~~
~~[ ] Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)~~
~~[ ] If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list~~
~~[ ] This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)~~
~~[ ] This was checked for cross-browser compatibility~~

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

[ ] This was checked for breaking API changes and was labeled appropriately

apmmachine commented 2 weeks ago

:robot: GitHub comments

Expand to view the GitHub comments

Just comment with: - `/oblt-deploy` : Deploy a Kibana instance using the Observability test environments. - `run` `docs-build` : Re-trigger the docs validation. (use unformatted text in the comment!)

mashhurs commented 2 weeks ago

FYI: I have updated the unit test cases which align with current changes, wil try to add for metricbeat.

afharo commented 1 week ago

Hmm, the failing tests indicate that we're somehow not returning the monitoring data... Maybe the new query is failing?

UPDATE: Found it in the logs:

[00:00:06]           â”‚ proc [kibana] [2024-05-08T00:55:24.693+00:00][WARN ][plugins.usageCollection.usage-collection.collector-set] ResponseError: search_phase_execution_exception
[00:00:06]           â”‚ proc [kibana]  Caused by:
[00:00:06]           â”‚ proc [kibana]      illegal_argument_exception: no mapping found for `logstash.node.stats.logstash.uuid` in order to collapse on
[00:00:06]           â”‚ proc [kibana]  Root causes:
[00:00:06]           â”‚ proc [kibana]      illegal_argument_exception: no mapping found for `logstash.node.stats.logstash.uuid` in order to collapse on
[00:00:06]           â”‚ proc [kibana]     at KibanaTransport.request (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/transport/lib/Transport.js:492:27)
[00:00:06]           â”‚ proc [kibana]     at processTicksAndRejections (node:internal/process/task_queues:95:5)
[00:00:06]           â”‚ proc [kibana]     at KibanaTransport.request (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/core-elasticsearch-client-server-internal/src/create_transport.js:51:16)
[00:00:06]           â”‚ proc [kibana]     at ClientTraced.SearchApi [as search] (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@elastic/elasticsearch/lib/api/api/search.js:66:12)
[00:00:06]           â”‚ proc [kibana]     at fetchLogstashStats (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/get_logstash_stats.js:225:19)
[00:00:06]           â”‚ proc [kibana]     at getLogstashStats (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/get_logstash_stats.js:312:5)
[00:00:06]           â”‚ proc [kibana]     at async Promise.all (index 2)
[00:00:06]           â”‚ proc [kibana]     at getAllStats (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/get_all_stats.js:34:49)
[00:00:06]           â”‚ proc [kibana]     at async Promise.all (index 1)
[00:00:06]           â”‚ proc [kibana]     at Collector.fetch (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/monitoring-plugin/server/telemetry_collection/register_monitoring_telemetry_collection.js:227:33)
[00:00:06]           â”‚ proc [kibana]     at CollectorSet.fetchCollector (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/usage-collection-plugin/server/collector/collector_set.js:141:24)
[00:00:06]           â”‚ proc [kibana]     at fetch_monitoringTelemetry (/var/lib/buildkite-agent/builds/kb-n2-4-spot-a9d9a28162911021/elastic/kibana-pull-request/kibana-build-xpack/node_modules/@kbn/usage-collection-plugin/server/collector/collector_set.js:175:103) {"service":{"node":{"roles":["background_tasks","ui"]}}}

mashhurs commented 1 week ago

@elasticmachine merge upstream

mashhurs commented 1 week ago

LGTM! This is great! Thanks for such an effort!

Thank you so much @afharo. This happened because of your huge help, appreciate!

kibana-ci commented 1 week ago

:green_heart: Build Succeeded

Buildkite Build
Commit: 37f39121157efaad3c6d2d25e3a18e033d4ca99c

Metrics [docs]

Unknown metric groups

#### ESLint disabled line counts | id | [before](https://github.com/elastic/kibana/commit/bc103c7016245901a04fc4921c1b213a4fbe2695) | [after](https://github.com/elastic/kibana/commit/37f39121157efaad3c6d2d25e3a18e033d4ca99c) | diff | | --- | --- | --- | --- | | `monitoring` | 18 | 20 | +2 | #### Total ESLint disabled count | id | [before](https://github.com/elastic/kibana/commit/bc103c7016245901a04fc4921c1b213a4fbe2695) | [after](https://github.com/elastic/kibana/commit/37f39121157efaad3c6d2d25e3a18e033d4ca99c) | diff | | --- | --- | --- | --- | | `monitoring` | 25 | 27 | +2 |

History

:green_heart: Build #208859 succeeded 22b51e847c83e5ece6b91ffa22deb90800d35a99
:broken_heart: Build #208800 failed 1701038d508c443a73b7f2c3b16a9cc2ce7bd5a6
:broken_heart: Build #208539 failed f86444cccb3b628a5bbd8ea7c8d79d3ff08d733b
:broken_heart: Build #208309 failed af3c267a7c532e071ce1832c5f7674d51f0edc6d
:broken_heart: Build #208301 failed 73160e1c38e3dc9f92c8d89962eceec34db93a74
:broken_heart: Build #207998 failed 540b7cebd243efb5463fb73b9db2b2fffb2504ca

To update your PR or re-run it, just comment with: @elasticmachine merge upstream

cc @mashhurs

mashhurs commented 6 days ago

@afharo, @neptunian can we please backport this change to upcoming 8.14.x releases?

afharo commented 5 days ago

I've added the appropriate label to back port this PR to the previous minor.

Did the same with https://github.com/elastic/kibana/pull/182857

Hopefully, our kibanamachine bot backports them for us.

kibanamachine commented 5 days ago

💚 All backports created successfully

Status	Branch	Result
✅	8.14

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

miltonhultgren commented 5 days ago

@afharo Thank you so much for sharing all your knowledge here and getting this to done!

elastic / kibana