Using X-Ray Agent in Spring Boot & Zuul Proxy API Gateway

[yukselcodingwithyou]

We are trying to instrument our microservices using x-ray java agent. Our microservices are implemented using Spring Boot framework. For other microservices, we accomplished the auto instrumentation using the disco plugins by adding environment variables to our applications as below.

-Dcom.amazonaws.xray.strategy.tracingName=serviceName -Dcom.amazonaws.xray.strategy.contextMissingStrategy=IGNORE_ERROR -Dcom.amazonaws.xray.emitter.daemonAddress=127.0.0.1:2000 -Dcom.amazonaws.xray.tracingEnabled=True -javaagent:src/main/resources/disco/disco-java-agent.jar=pluginPath=src/main/resources/disco/disco-plugins

But in our API Gateway, we failed to do so. While we try to trace any request from our API Gateway, we came across an error like this below:

{ "error": "Exception while serializing entity.", "stack_trace":"com.fasterxml.jackson.databind.JsonMappingException: Failed to load class 'com.amazonaws.xray.entities.SegmentImpl$Access4JacksonSerializerdb8acfce': com.fasterxml.jackson.module.afterburner.ser.BeanPropertyAccessor com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:295) } full-stack-trace.json.zip

Also in our x-ray daemon running locally, we encountered with this warning below:

[Warn] Missing header or segment: {"format": "json", "version": 1}

To resolve this error, we tried to exclude plugin path from command like below:

-Dcom.amazonaws.xray.strategy.tracingName=bid-service-local -Dcom.amazonaws.xray.strategy.contextMissingStrategy=IGNORE_ERROR -Dcom.amazonaws.xray.emitter.daemonAddress=127.0.0.1:2000 -Dcom.amazonaws.xray.tracingEnabled=True -javaagent:src/main/resources/disco/disco-java-agent.jar

But there was no segment sent to the daemon that is running locally. Then we tried to add the plugins to its path one by one,

aws-xray-agent-plugin.jar disco-java-agent-aws-plugin.jar

First, we added the two jars above to disco-plugins path above, there was no error, but segment also not sent to our daemon .

Then, we added the jar below and, we saw that this error is related to this plugin, with some dependencies that we use in our API Gateway application.

disco-java-agent-web-plugin.jar

Dependencies & Technologies

Spring Boot version is 2.3.8.

 - implementation "org.springframework.boot:spring-boot-configuration-processor"
 - implementation "org.springframework.boot:spring-boot-starter-actuator"
 - implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-zuul'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix-dashboard'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-hystrix'
 - implementation 'org.springframework.cloud:spring-cloud-starter-netflix-ribbon'
 - implementation "org.springframework.cloud:spring-cloud-starter-sleuth:2.2.8.RELEASE"

Also we are using ZuulFilter to add custom headers to incoming requests' header. Could this also be an issue for our error that we encounter, how can we move forward from here? 🙏

[willarmiros]

Hi @yukselcodingwithyou,

Apologies for the delayed response. Just to confirm, when you mention your Zuul proxy API Gateway, you're referring to a service that you own and not AWS's API Gateway service right?

Regardless, I'm not too sure about this issue as I haven't seen it before but perhaps it is related to https://github.com/FasterXML/jackson-modules-base/issues/131 based on a quick google search. Currently the agent pulls in v2.11.0, so it could be that upgrading the agent's version would fix this. Is there any way you could provide some reproduction code for this, so we can verify the fix?

Also, just as an FYI, if you're still having the X-Ray java agent feel free to try out the AWS Distro for OpenTelemetry Java Agent. It natively supports instrumenting many more libraries/frameworks, and has 1st-class support for X-Ray.