search

aws / aws-xray-sdk-dotnet

The official AWS X-Ray SDK for .NET.
Apache License 2.0
112 stars 64 forks source link

Problems with AWS_XRAY_DAEMON_ADDRESS containing host:port #158

Open Kralizek opened 4 years ago

Kralizek commented 4 years ago

I am unable to reach the daemon hosted in a contain from another container.

If I set AWS_XRAY_DAEMON_ADDRESS to xray-daemon:2000, I get this:

[web_a2edb13e-b]: AWSXRayMiddleware 2|2020-11-04T01:00:10.731Z|DEBUG|Trace header doesn't exist or not valid : (). Injecting a new one.
[web_a2edb13e-b]: IPEndPointExtension 3|2020-11-04T01:00:10.747Z|DEBUG|Determined that xray-daemon:2000 is not an IP address, will try to parse it as a hostname.
[web_a2edb13e-b]: IPEndPointExtension 4|2020-11-04T01:00:10.747Z|INFO|Using custom daemon address: xray-daemon:2000
[web_a2edb13e-b]: HostEndPoint 5|2020-11-04T01:00:13.037Z|INFO|IP cache invalid: failed to resolve DNS. (xray-daemon) Cached IP address not updated!
[web_a2edb13e-b]: IPEndPointExtension 6|2020-11-04T01:00:13.037Z|ERROR|Invalid daemon address. (xray-daemon:2000) --> System.NullReferenceException: Object reference not set to an instance of an object.
[web_a2edb13e-b]: at Amazon.XRay.Recorder.Core.Internal.Utils.IPEndPointExtension.ParseSingleForm(String[] daemonAddress, DaemonConfig& endPoint)
[web_a2edb13e-b]: at Amazon.XRay.Recorder.Core.Internal.Utils.IPEndPointExtension.TryParseDaemonAddress(String[] daemonAddress, DaemonConfig& endPoint)
[web_a2edb13e-b]: at Amazon.XRay.Recorder.Core.Internal.Utils.IPEndPointExtension.TryParse(String daemonAddress, DaemonConfig& daemonEndPoint)
[web_a2edb13e-b]: DaemonConfig 7|2020-11-04T01:00:13.048Z|INFO|The given daemonAddress (xray-daemon:2000) is invalid, using default daemon UDP and TCP address 127.0.0.1:2000.
[web_a2edb13e-b]: RuntimePipelineCustomizerRegistry 8|2020-11-04T01:00:13.058Z|INFO|Applying runtime pipeline customization X-Ray Registration Customization
[web_a2edb13e-b]: DefaultSamplingStrategy 9|2020-11-04T01:00:13.070Z|INFO|No effective centralized sampling rule match. Fallback to local rules.
[web_a2edb13e-b]: LocalizedSamplingStrategy 10|2020-11-04T01:00:13.070Z|DEBUG|Can't match a rule for host = localhost, path = /, method = GET

I thought of a problem of networking, so I got into one of the application containers and pinged the xray-daemon hostname, but it works as expected:

C:\Users\Renato> docker ps
CONTAINER ID   IMAGE                                   COMMAND                  CREATED          STATUS          PORTS                              NAMES
99468894af97   mcr.microsoft.com/dotnet/core/sdk:3.1   "dotnet Microsoft.Ty…"   47 seconds ago   Up 46 seconds                                      webapi-proxy_22679936-b
d83fb805928f   mcr.microsoft.com/dotnet/core/sdk:3.1   "dotnet Microsoft.Ty…"   47 seconds ago   Up 46 seconds                                      service-proxy_b5dccf6f-a
a839f7ba29c9   mcr.microsoft.com/dotnet/core/sdk:3.1   "dotnet Microsoft.Ty…"   47 seconds ago   Up 46 seconds                                      web-proxy_adce8a0a-b
c204c52c9f51   amazon/aws-xray-daemon                  "/usr/bin/xray -t 0.…"   47 seconds ago   Up 47 seconds   2000/udp, 0.0.0.0:2000->2000/tcp   xray-daemon_baf994f0-1
C:\Users\Renato> docker exec -it 9946 ping xray-daemon
PING xray-daemon (192.168.112.2) 56(84) bytes of data.
64 bytes from xray-daemon_baf994f0-1.tye_network_dd6ecb25-9 (192.168.112.2): icmp_seq=1 ttl=64 time=0.110 ms
64 bytes from xray-daemon_baf994f0-1.tye_network_dd6ecb25-9 (192.168.112.2): icmp_seq=2 ttl=64 time=0.051 ms
64 bytes from xray-daemon_baf994f0-1.tye_network_dd6ecb25-9 (192.168.112.2): icmp_seq=3 ttl=64 time=0.059 ms
64 bytes from xray-daemon_baf994f0-1.tye_network_dd6ecb25-9 (192.168.112.2): icmp_seq=4 ttl=64 time=0.050 ms
64 bytes from xray-daemon_baf994f0-1.tye_network_dd6ecb25-9 (192.168.112.2): icmp_seq=5 ttl=64 time=0.077 ms
^C
--- xray-daemon ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 141ms
rtt min/avg/max/mdev = 0.050/0.069/0.110/0.023 ms

You can see the whole application here: https://github.com/Kralizek/ToDoList

Steps to reproduce:

srprash commented 4 years ago

Hi @Kralizek Thanks for providing details of the issue and the steps to reproduce it. We will prioritize to investigate this shortly.

srprash commented 4 years ago

Hey @Kralizek What I see from the tye.yaml and the attached log, is that you're setting the env var AWS_XRAY_DAEMON_ADDRESS to the string xray-daemon:2000 whereas you should set it to the actual address of the daemon like 0.0.0.0:2000 as you have it in your dockerfile. Could you try this and let me know if it works?

Thanks!

Kralizek commented 4 years ago

I'll try it straight away.

srprash commented 4 years ago

Wow! That was fast! :D

Kralizek commented 4 years ago

@srprash It didn't work :(

[webapi_d466b2cb-f]:C:\Users\Renato\Development\Tests\ToDoList\src\entrypoints\WebAPI\bin\Debug\net5.0\WebAPI.exe
[webapi_d466b2cb-f]: RuntimePipelineCustomizerRegistry 1|2020-11-12T22:33:25.108Z|INFO|Registering runtime pipeline customizer X-Ray Registration Customization
[webapi_d466b2cb-f]: info: Microsoft.Hosting.Lifetime[0]
[webapi_d466b2cb-f]: Now listening on: https://localhost:54654
[webapi_d466b2cb-f]: info: Microsoft.Hosting.Lifetime[0]
[webapi_d466b2cb-f]: Application started. Press Ctrl+C to shut down.
[webapi_d466b2cb-f]: info: Microsoft.Hosting.Lifetime[0]
[webapi_d466b2cb-f]: Hosting environment: Development
[webapi_d466b2cb-f]: info: Microsoft.Hosting.Lifetime[0]
[webapi_d466b2cb-f]: Content root path: C:\Users\Renato\Development\Tests\ToDoList\src\entrypoints\WebAPI
[webapi_d466b2cb-f]: info: WebAPI.Controllers.ToDoController[0]
[webapi_d466b2cb-f]: Fetching all items
[webapi_d466b2cb-f]: info: System.Net.Http.HttpClient.ToDoClient.LogicalHandler[100]
[webapi_d466b2cb-f]: Start processing HTTP request POST https://localhost:23123/toDo.ToDo/List
[webapi_d466b2cb-f]: info: System.Net.Http.HttpClient.ToDoClient.ClientHandler[100]
[webapi_d466b2cb-f]: Sending HTTP request POST https://localhost:23123/toDo.ToDo/List
[webapi_d466b2cb-f]: info: System.Net.Http.HttpClient.ToDoClient.ClientHandler[101]
[webapi_d466b2cb-f]: Received HTTP response headers after 249.5895ms - 200
[webapi_d466b2cb-f]: info: System.Net.Http.HttpClient.ToDoClient.LogicalHandler[101]
[webapi_d466b2cb-f]: End processing HTTP request after 262.3015ms - 200
[webapi_d466b2cb-f]: UdpSegmentEmitter 2|2020-11-12T22:33:51.122Z|DEBUG|UDP Segment emitter endpoint: 0.0.0.0:2000.
[webapi_d466b2cb-f]: UdpSegmentEmitter 3|2020-11-12T22:33:51.123Z|ERROR|Failed to send package through socket. --> System.Net.Sockets.SocketException (10049): The requested address is not valid in its context.
[webapi_d466b2cb-f]: at System.Net.Sockets.Socket.UpdateStatusAfterSocketErrorAndThrowException(SocketError error, String callerName)
[webapi_d466b2cb-f]: at System.Net.Sockets.Socket.SendTo(Byte[] buffer, Int32 offset, Int32 size, SocketFlags socketFlags, EndPoint remoteEP)
[webapi_d466b2cb-f]: at System.Net.Sockets.UdpClient.Send(Byte[] dgram, Int32 bytes, IPEndPoint endPoint)
[webapi_d466b2cb-f]: at Amazon.XRay.Recorder.Core.Internal.Emitters.UdpSegmentEmitter.Send(Entity segment)
srprash commented 4 years ago

Sorry. I think I misunderstood it since I don't really know how exactly dotnet tye deploys application to containers. But what you really need to do is, set the AWS_XRAY_DAEMON_ADDRESS to the container host address of the xray daemon container and port to 2000 since that's what daemon listens on.

I'll read more about dotnet tye and give this a try.

Kralizek commented 4 years ago

@srprash Allow me this shameless plug to the blog post I just posted about it!

https://renatogolia.com/2020/11/12/developing-distributed-applications-with-tye/

srprash commented 4 years ago

@Kralizek Thanks for sharing the blog. It's a good learning for me. FWIU from this section, in order to let your services discover the xray-daemon, you'd need to do some bindings to expose the daemon address. Then within your application you can programmatically build the AWSXRayRecorder with this address by doing something like this:

var recorder = new AWSXRayRecorderBuilder().WithDaemonAddress(xrayDaemonAddressString).Build();
AWSXRayRecorder.InitializeInstance(configuration, recorder);
Kralizek commented 4 years ago

Hi @srprash My intention was to use the default port 2000 for the daemon, but I'll try using a custom port to see if it solves the issue.