search

aws / aws-xray-daemon

The AWS X-Ray daemon listens for traffic on UDP port 2000, gathers raw segment data, and relays it to the AWS X-Ray API.
Apache License 2.0
189 stars 69 forks source link

Spring Boot app cannot connect to Daemon in Docker container #165

Open haiko opened 2 years ago

haiko commented 2 years ago

I have instrumented a Spring Boot application with XRay. I run a Docker container with a proxy to XRay using the instructions on https://docs.aws.amazon.com/xray/latest/devguide/xray-daemon-local.html. But my application cannot connect to the XRay daemon in the docker container. The daemon in the docker container start up logs the following:

2022-02-01T10:51:23Z [Info] Initializing AWS X-Ray daemon 3.3.3
2022-02-01T10:51:23Z [Info] Using buffer memory limit of 19 MB
2022-02-01T10:51:23Z [Info] 304 segment buffers allocated
2022-02-01T10:51:23Z [Info] Using region: eu-central-1
2022-02-01T10:51:23Z [Info] HTTP Proxy server using X-Ray Endpoint : https://xray.eu-central-1.amazonaws.com
2022-02-01T10:51:23Z [Info] Starting proxy http server on 127.0.0.1:2000

My application tries to connect but fails with the following exception:

2022-02-01 11:52:59.150 ERROR 48013 --- [pool-2-thread-1] c.a.x.s.sampling.pollers.RulePoller      : Encountered error polling GetSamplingRules: 
com.amazonaws.xray.internal.XrayClientException: Could not serialize and send request.
    at com.amazonaws.xray.internal.UnsignedXrayClient.sendRequest(UnsignedXrayClient.java:142)
    at com.amazonaws.xray.internal.UnsignedXrayClient.getSamplingRules(UnsignedXrayClient.java:112)
    at com.amazonaws.xray.strategy.sampling.pollers.RulePoller.pollRule(RulePoller.java:100)
    at com.amazonaws.xray.strategy.sampling.pollers.RulePoller.lambda$start$0(RulePoller.java:72)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
    at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.net.ConnectException: Connection refused (Connection refused)

When I run the xray daemon locally on my system everything works and traces are uploaded to AWS. Anyone an idea?

wangzlei commented 2 years ago

By default XRay Daemon turns on UDP port 2000 for sending trace to XRay and TCP port 2000 for remote sampling rules. I guess you are running Daemon in docker for Mac, so, get error log about Sampling rule poller but still see trace in xray console. There is a known issue that docker for Mac does not map to local TCP port, please try workaround by searching google, such as https://github.com/docker/for-mac/issues/770

If not using remote sampling rule, you can ignore that log.

Gaurav-Vishvakarman commented 2 months ago

I am also facing same kind of issue and these stack traces are getting injected under cloudwatch which in turn increasing cost down the line. Is there any way using which we can disable polling GetSamplingRules at service or x-ray daemon level?