Telemetry missing when Logstash is monitored exclusively by Metricbeat

[yaauie]

Kibana version: Any 7.17-8.x (including 7.17.19 and 8.13.0)

Elasticsearch version: Any 7.17-8.x (including 7.17.19 and 8.13.0)

Server OS version: ANY

Browser version: ANY

Browser OS version: ANY

Original install method (e.g. download page, yum, from source, etc.): ANY

Describe the bug:

When a Logstash has monitoring data that has exclusively been sent via Metricbeat, Monitoring Telemetry fails to send telemetry data.

Steps to reproduce:

1. Set up Elasticsearch and Kibana with security and stack monitoring as usual, with telemetry enabled, making note of its cluster UUID
2. Set up and run Logstash without monitoring itself
   • ${LOGSTASH_HOME}/config/logstash.yml should have:

     monitoring.enabled: false
     monitoring.cluster_uuid: "MONITORING_CLUSTER_UUID_HERE"

   • Execute a simple heartbeat pipeline and leave it running: ${LOGSTASH_HOME}/bin/logstash --config.string="input { heartbeat {} } output { stdout { codec => dots } }"
3. Set up Metricbeat to observe only Logstash
   • Create a role beats_system_expanded that has additional permissions necessary to push logstash monitoring data:
   • cluster: monitor, cluster:admin/xpack/monitoring/bulk, manage_ilm, manage_index_templates (same as beats_system)
   • index .monitoring-beats-*: create_index, create (same as beats_system)
   • index .monitoring-logstash-*: create_index, create
   • Create a user beats_system_expanded with role beats_system_expanded and record its password
   • Configure metricbeat:
   • configure the output to send monitoring events to Elasticsearch using our beats_system_expanded user:

     output.elasticsearch:
      # Array of hosts to connect to.
      hosts: ["localhost:9200"]
     
      # Performance preset - one of "balanced", "throughput", "scale",
      # "latency", or "custom".
      preset: balanced
     
      # Protocol - either `http` (default) or `https`.
      protocol: "https"
     
      # Authentication credentials - either API key or username/password.
      #api_key: "id:api_key"
      username: "beats_system_expanded"
      password: "${BEATS_SYSTEM_EXPANDED_PASSWORD}"
     
      ssl.ca_trusted_fingerprint: "${ELASTIC_CA_FINGERPRINT}"

   • configure it to not monitor itself (but include the cluster UUID for good measure):

     # Set to true to enable the monitoring reporter.
     monitoring.enabled: false
     
     # Sets the UUID of the Elasticsearch cluster under which monitoring data for this
     # Metricbeat instance will appear in the Stack Monitoring UI. If output.elasticsearch
     # is enabled, the UUID is derived from the Elasticsearch cluster referenced by output.elasticsearch.
     monitoring.cluster_uuid: "MONITORING_CLUSTER_UUID_HERE"

   • disable the system module by renaming ${METRICBEAT_HOME}/modules.d/system.yml to ${METRICBEAT_HOME}/modules.d/system.yml.disabled
   • enable logstash monitoring by renaming ${METRICBEAT_HOME}/modules.d/logstash-xpack.yml.disabled to ${METRICBEAT_HOME}/modules.d/logstash-xpack.yml, ensuring that the module's config points to localhost:9600 (where Logstash's API is) and has xpack.enabled: true

     # Module: logstash
     # Docs: https://www.elastic.co/guide/en/beats/metricbeat/main/metricbeat-module-logstash.html
     
     - module: logstash
       xpack.enabled: true
       metricsets: ["node", "node_stats"]
       period: 10s
       hosts: ["127.0.0.1:9600"]
       #username: "user"
       #password: "secret"

   • start metricbeat with console logging enabled: (cd "${METRICBEAT_HOME}"; ./metricbeat -e)
4. In Kibana, observe the monitoring data for Logstash is making its way to the correct indices, ensuring at least one result for each of:

   POST /.monitoring-logstash*/_search
   { "query": { "term": {"metricset.name":"node"} } }

   POST /.monitoring-logstash*/_search
   { "query": { "term": {"metricset.name":"node_stats"} } }

5. Leave it running for at least 24h to give the Telemetry collector a chance to run

Expected behavior:

The data pushed to telemetry for this cluster ID should have Logstash-related telemetry

Any additional context:

In #122345 Stack Monitoring was extended to allow monitoring from standalone metricbeat, but Telemetry Collection works by reaching into the same monitoring indices and made no matching changes.

• Logstash stats are stored in a logstash.node.stats namespace via metricbeat, and in legacy collection a similar structure was available in a logstash_stats namespace
  • in Telemetry collector, logstash stats only accommodates the logstash_stats namespace.
  • the query also matches docs that have metricset.name of stats (not node_stats), but still includes a collapse on a field that exists only in the logstash_stats namespace.
• Similarly, Logstash state is stored in a logstash.node namespace via metricbeat, while legacy collection had a similar structure in logstash_state.
  • in Telemetry collector, logstash state only accommodates the logstash_state namespace.
  • the query includes a collapse on a field that only exists in the logstash_state namespace.

I have attempted to remedy this by adding the fields to the filter_path directives (and normalizing the results to the old shape), but the existing use of collapse on a legacy-only field greatly expands the scope of the fix. I attempted to add a runtime field to collapse on, but discovered that is not supported (elastic/elasticsearch#66459).

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[afharo]

It might be addressed by https://github.com/elastic/kibana/pull/182304