XRAY Agent Application Logging

[davekant]

We use the java auto-instrumentation agent to capture traces for our applications in fargate. However, in our application logs we are seeing a lot of verbosity coming from the agent (below).

Is there any way we can suppress/turn-off the logging, or send it to another CloudWatch log so that the application logs are not contaminated. The XRAY auto-instrumentation agent is supposed to be a non-invasive/non-intrusive tool to monitoring applications so we are looking for either a system property similar to logging.level.com.amazonaws.xray or a switch in the xray-agent.json config file.

The auto-instrumentation java agent is a great solution because its not possible for us to instrument the underlying application code using the SDK. XRAY is critical to our overall monitoring strategy because we are attempting to remove DataDog in favour of AWS native monitoring solutions and to reduce our overall costs running the estate.

26-Mar-2021 06:08:04.669 SEVERE [kyc-quartz-scheduler_Worker-5] com.amazonaws.xray.agent.runtime.listeners.XRayListener.listen The X-Ray Agent had encountered an unexpected exception for the following event: software.amazon.disco.agent.event.ServiceDownstreamResponseEvent@2c0dc96f java.lang.NullPointerException at com.amazonaws.xray.agent.runtime.handlers.downstream.SqlHandler.handleResponse(SqlHandler.java:80) at com.amazonaws.xray.agent.runtime.dispatcher.EventDispatcher.dispatchResponseEvent(EventDispatcher.java:75) at com.amazonaws.xray.agent.runtime.listeners.XRayListener.listen(XRayListener.java:38) at software.amazon.disco.agent.event.EventBus.publish(EventBus.java:66)

xray_application_log.txt

[willarmiros]

Hi @davekant,

The agent's log level can be configured by the same logging.level.com.amazonaws.xray system property as the SDK for Java. You can set it to FATAL to suppress all log messages. Our documentation is definitely not clear on this, so I've opened up a PR to explain it. Hope this resolves your issue.

[davekant]

@willarmiros,

We applied the system property but still see the SEVERE messages reported in application logs

JAVA_OPTS="-javaagent:/usr/local/disco/disco-java-agent.jar=pluginPath=/usr/local/disco/disco-plugins -Dcom.amazonaws.xray.configFile=/usr/local/disco/xray-agent.json -Dlogging.level.com.amazonaws.xray=FATAL"

31-Mar-2021 12:50:17.777 SEVERE [kyc-quartz-scheduler_Worker-5] com.amazonaws.xray.agent.runtime.listeners.XRayListener.listen The X-Ray Agent had encountered an unexpected exception for the following event: software.amazon.disco.agent.event.ServiceDownstreamResponseEvent@55924043 java.lang.NullPointerException at com.amazonaws.xray.agent.runtime.handlers.downstream.SqlHandler.handleResponse(SqlHandler.java:80) at com.amazonaws.xray.agent.runtime.dispatcher.EventDispatcher.dispatchResponseEvent(EventDispatcher.java:75)