[APM] Service Map not rendering the complete path

[crespocarlos]

Summary

We need to investigate why the service map fails to render the full path in the situation below.

Scenario

The scenario tested below is: ServiceX -> HTTP request -> Service Y -> HTTP request -> Service Z.

High level service map

Service map for service X

Service map for service Y

Service map for service Z

The trace waterfall of a transaction from X shows the complete trace correctly

The service map query returns the same thing for all the cases above, the difference is the the high level and Z maps manage to include the ExampleHttpServerZ from discoveredServices, whereas the others don't.

 {
    "service_map": {
      "value": {
        "paths": [
          [
            {
              "service.environment": null,
              "service.name": "ExampleServiceX",
              "agent.name": "java"
            },
            {
              "span.subtype": "http",
              "span.destination.service.resource": "localhost:12926",
              "span.type": "external"
            }
          ],
          [
            {
              "service.environment": null,
              "service.name": "ExampleServiceX",
              "agent.name": "java"
            },
            {
              "service.environment": null,
              "service.name": "ExampleHttpServerY",
              "agent.name": "java"
            },
            {
              "span.subtype": "http",
              "span.destination.service.resource": "localhost:12928",
              "span.type": "external"
            }
          ]
        ],
        "discoveredServices": [
          {
            "from": {
              "span.subtype": "http",
              "span.destination.service.resource": "localhost:12926",
              "span.type": "external"
            },
            "to": {
              "service.environment": null,
              "service.name": "ExampleHttpServerY",
              "agent.name": "java"
            }
          },
          {
            "from": {
              "span.subtype": "http",
              "span.destination.service.resource": "localhost:12928",
              "span.type": "external"
            },
            "to": {
              "service.environment": null,
              "service.name": "ExampleHttpServerZ",
              "agent.name": "java"
            }
          }
        ]
      }
   }
}

Steps to reproduce

• Start a local Kibana and ES instances
• Start Fleet locally
• Start APM server locally
• Install the APM integration
• Download uitest.jar (internal slack thread)

To run the scenario

curl -o opentelemetry-context-1.42.1.jar https://repo1.maven.org/maven2/io/opentelemetry/opentelemetry-context/1.42.1/opentelemetry-context-1.42.1.jar
curl -o opentelemetry-api-1.42.1.jar https://repo1.maven.org/maven2/io/opentelemetry/opentelemetry-api/1.42.1/opentelemetry-api-1.42.1.jar
curl -o elastic-apm-agent-1.51.0.jar https://repo1.maven.org/maven2/co/elastic/apm/elastic-apm-agent/1.51.0/elastic-apm-agent-1.51.0.jar
export ELASTIC_APM_SERVER_URL=http://0.0.0.0:8200
export ELASTIC_APM_SECRET_TOKEN=<apm integration token>
export AGENT_JAR_PATH=./elastic-apm-agent-1.51.0.jar

java -javaagent:$AGENT_JAR_PATH -cp uitest.jar:opentelemetry-context-1.42.1.jar:opentelemetry-api-1.42.1.jar co.elastic.apm.uiservicemap2.ExampleServiceX

[elasticmachine]

Pinging @elastic/obs-ux-infra_services-team (Team:obs-ux-infra_services)

[crespocarlos]

Here is a PR with a possible fix: https://github.com/elastic/kibana/pull/192859. I've only tested it against the scenario from this ticket.