search

elastic / beats

:tropical_fish: Beats - Lightweight shippers for Elasticsearch & Logstash
https://www.elastic.co/products/beats
Other
109 stars 4.93k forks source link

Agentbeat breaks configuration only Metricbeat modules #39223

Closed cmacknz closed 6 months ago

cmacknz commented 7 months ago

There are several test failures that appear to be introduced by agentbeat. On initial investigation it appears that agentbeat has broken lightweight Metricbeat modules that only use configuration.

Sample failures:

1 error: metricset ‘mysql/performance’ not found
1 error: metricset ‘activemq/topic’ not found
1 error: metricset ‘activemq/queue’ not found

See https://github.com/elastic/beats/tree/main/metricbeat/module/mysql/performance for example there are no .go files.

The performance module is defined in https://github.com/elastic/beats/blob/5d1b442d4bb3698a0ab6f3283b5514ed353966b0/metricbeat/module/mysql/module.yml#L1-L6

With the actual implementation defined in https://github.com/elastic/beats/blob/5d1b442d4bb3698a0ab6f3283b5514ed353966b0/metricbeat/module/mysql/performance/manifest.yml#L1-L23

It seems like the module.yml file is compiled into metricbeat and the manifest.yml is no longer present at runtime.

elasticmachine commented 7 months ago

Pinging @elastic/elastic-agent (Team:Elastic-Agent)

cmacknz commented 7 months ago

https://github.com/elastic/beats/tree/main/x-pack/metricbeat/module/activemq seems to be the same, I'm sure there are more of these.

nimarezainia commented 7 months ago

@cmacknz should we ask the other teams (integration owners) to also run full tests? i imagine they are as part of their 8.14 efforts.

strawgate commented 7 months ago

Are modules used with agentbeat?

cmacknz commented 7 months ago

@cmacknz should we ask the other teams (integration owners) to also run full tests? i imagine they are as part of their 8.14 efforts.

Automated tests found this and we should be doing full test for every release regardless of this change. I will send out a wider notice about this change to minimize the chances of things slipping through the cracks.

Are modules used with agentbeat?

It appears that parts of them are. The assets from modules (dashboards, ingest pipelines, mappings, etc) are not as those are obsoleted by integrations in agent.

However this issue has discovered that modules can define metricsets purely with configuration, and metricsets are used in agent as they are part of the metricbeat inputs in the agent policy. We need to figure out the best way to include these in agentbeat.

elasticmachine commented 7 months ago

Pinging @elastic/elastic-agent-control-plane (Team:Elastic-Agent-Control-Plane)