For this metricset the name of the called destination is missing.

[holiday-sunrise]

Among other things, the Mule OpenTelemetry Agent sends aggregated metrics relating to remote service calls (- metricset.name: service_destination, metricset.interval: 1m) For this metricset the name of the called destination is missing. In other record types (span.type: external) we find service.target.name, destination.address or span.destination.resource. In this metricset we find no information which service was called.

Example

`{ "_index": ".ds-metrics-apm.service_destination.1m-default-2024.07.03-000060", "_id": "KkRQfZABieRRS1uBvCgx", "_version": 1, "_score": 0, "_source": { "@timestamp": "2024-07-04T10:35:00.000Z", "_doc_count": 47, "agent":

{ "name": "opentelemetry/java" }

, "data_stream":

{ "dataset": "apm.service_destination.1m", "namespace": "default", "type": "metrics" }

, "ecs":

{ "version": "8.6.0-dev" }

, "event":

{ "agent_id_status": "missing", "ingested": "2024-07-04T10:36:01Z", "outcome": "unknown" }

, "labels":

{ "mule_api_id": "2239871", "mule_env": "test", "mule_home": "/opt/mule", "process_runtime_description": "Temurin OpenJDK 64-Bit Server VM 25.412-b08" }

, "metricset":

{ "interval": "1m", "name": "service_destination" }

, "observer":

{ "hostname": "40695aa79e54", "type": "apm-server", "version": "8.10.4" }

, "processor":

{ "event": "metric", "name": "metric" }

, "service": { "environment": "test", "language":

{ "name": "java" }

, "name": "echo-eapi", "target":

{ "type": "http" }

},
"span": {
  "destination": {
    "service": {
      "response_time": {

"count": 47, "sum":

{ "us": 498000 }

      }
    }
  },
  "name": "/health"
}

}, "fields":

{ "span.name": [ "/health" ], "labels.process_runtime_description": [ "Temurin OpenJDK 64-Bit Server VM 25.412-b08" ], "span.destination.service.response_time.count": [ 47 ], "service.language.name": [ "java" ], "labels.mule_home": [ "/opt/mule" ], "processor.event": [ "metric" ], "service.target.type": [ "http" ], "agent.name": [ "opentelemetry/java" ], "event.agent_id_status": [ "missing" ], "event.outcome": [ "unknown" ], "labels.mule_env": [ "test" ], "service.environment": [ "test" ], "_doc_count": [ 47 ], "service.name": [ "echo-eapi" ], "data_stream.namespace": [ "default" ], "processor.name": [ "metric" ], "metricset.interval": [ "1m" ], "data_stream.type": [ "metrics" ], "observer.hostname": [ "40695aa79e54" ], "labels.mule_api_id": [ "2239871" ], "metricset.name": [ "service_destination" ], "event.ingested": [ "2024-07-04T10:36:01.000Z" ], "@timestamp": [ "2024-07-04T10:35:00.000Z" ], "span.destination.service.response_time.sum.us": [ 498000 ], "ecs.version": [ "8.6.0-dev" ], "observer.type": [ "apm-server" ], "observer.version": [ "8.10.4" ], "data_stream.dataset": [ "apm.service_destination.1m" ] }

}`

[manikmagar]

Hi @holiday-sunrise, AFAIK, this module does not send these aggregated metrics. If we refer to https://www.elastic.co/guide/en/observability/current/apm-data-model-metrics.html#_example_service_destination_document, these look like the ones calculated by the APM backend server based on the span data ingested from the OTEL.

[manikmagar]

@holiday-sunrise As indicated above, these are APM server-generated metrics and not from this module. I am closing this for now. Please feel free to reopen if any information is available for the module. Thanks!