Entity doesn't exist in AsyncLocal exception when X-Ray tracing disabled

[djluck]

Hi there,

When calling GetAsyncResponseTraced, I get the following exception:

Entity doesn't exist in AsyncLocal\n at Amazon.XRay.Recorder.Core.Internal.Context.AsyncLocalContextContainer.GetEntity() at Amazon.XRay.Recorder.Handlers.System.Net.Utils.RequestUtil.ProcessRequest(Uri uri, String method, Action`1 addHeaderAction) at Amazon.XRay.Recorder.Handlers.System.Net.Utils.RequestUtil.ProcessRequest(WebRequest request) at Amazon.XRay.Recorder.Handlers.System.Net.HttpWebRequestTracingExtension.GetAsyncResponseTraced(WebRequest request)

It seems that RequestUtil.ProcessRequest is attempting to access AWSXRayRecorder.Instance.TraceContext.GetEntity even though I have disabled x-ray tracing via my configuration (debugging my application confirms x-ray tracing is disabled). This seems like a bug to me- I wouldn't expect these instrumented methods to do anything if tracing was disabled.

[yogiraj07]

Hi @djluck , Thanks for reporting the issue. I believe this code is causing the issue. It should be part of above if block. I would fix it soon.

A PR is also welcomed.

[djluck]

I believe you are right @yogiraj07, I can put together a simple PR as I believe the fix will be just moving the referenced line inside the block that checks if X-ray is enabled.

[jrlang1]

I am having the same bug with TraceableSqlCommand. Should I open a new issue

[yogiraj07]

@jrlang1 , please feel free to open a separate issue. I will verify and do necessary changes. A PR is also welcomed. Thanks for posting the issue.

[mbp]

Would be nice to get this fixed (new version on nuget). Impossible for us to use the library when we cannot ignore runtime errors.

[yogiraj07]

@mbp, we are planning for a release soon. I will prioritize this release. Please stay tuned. On the side note, can you help me explain your use case ? What difficulties are you facing?

Thanks for patience, Yogi

[mbp]

@yogiraj07 as part of ASP.NET Core application, we have added xray tracing. This ASP.NET Core application has some background processes (some code coming from other libraries), which we cannot change easily, so we need to set AWS_XRAY_CONTEXT_MISSING = LOG_ERROR, which is not respected. We get exception from the same method which is mentioned in this issue (RequestUtil.ProcessRequest)

[yogiraj07]

@jrlang1 , I am unable to reproduce this issue for TraceSqlCommand. Please open a separate issue and let me know more details about the issue you are facing and how to reproduce it locally.

[tobymiller]

@jrlang1 Is there likely to to be a release (even prerelease) soon? We're also struggling, as we can't debug locally with calls to live aws.

[yogiraj07]

@tobymiller1 , we have planned release coming soon. I have this issue on priority. Please stay tuned and thanks for the patience.

[yogiraj07]

Hello, This has been fixed in release 2.6.0. Please refer Changelog.

[Ellosaur]

Update: Below solved by reducing the context missing strategy to a log event from an exception: AWSXRayRecorder.Instance.ContextMissingStrategy = ContextMissingStrategy.LOG_ERROR;

I'm seeing this exception when calling AWSXRayRecorder.Instance.BeginSubsegment with our code deployed as a local Web Api instead of deployed as a Lambda. Must we prevent calls like this with preprocessor directives for AWS vs Local deployments or can this call be configured to do nothing when not in AWS land? AWSXRayRecorder Version="2.6.2"

[yogiraj07]

@Ellosaur , AWSXRayRecorder.Instance.BeginSubsegment in local deployment should automatically attach the subsegment to the middleware Segment.

1. Would you mind posting the exception stack trace?
2. Do you get this exception while running tests or running app? Since I wonder if you run app and have correctly instrumented the web app using instrumentor, ideally there should have been no error for local deployment.
3. .NET or .NET Core platform?
4. Is the call to AWSXRayRecorder.Instance.BeginSubsegment in asynchronous background thread or job or a part of some regular path of an incoming request?

[thinkOfaNumber]

Hi @yogiraj07, I'm using AWSXrayRecorder 2.10.1 in a dotnet core 3.1 web api app and have come across this in my local environment.

Some notes:

1. in my startup configure function I'm using:

   if (!env.IsDevelopment())
   {
   // AWS XRay tracing
   app.UseXRay("...");
   }

2. an error was thrown within an async http request (while locally testing the web app)
3. I have configured error handling in startup.cs: app.UseExceptionHandler("/error");
4. in my error handler route, I get the XRay error in the last line shown here:

var error = HttpContext
    .Features
    .Get<IExceptionHandlerPathFeature>();

recorder.AddMetadata("ExceptionHandler", error);

The stack trace is:

Amazon.XRay.Recorder.Core.Exceptions.EntityNotAvailableException
  HResult=0x80131500
  Message=Entity doesn't exist in AsyncLocal
  Source=AWSXRayRecorder.Core
  StackTrace:
   at Amazon.XRay.Recorder.Core.Internal.Context.AsyncLocalContextContainer.GetEntity()
   at Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl.AddMetadata(String key, Object value)
--- End of stack trace from previous location ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Amazon.XRay.Recorder.Core.Internal.Context.TraceContextImpl.HandleEntityMissing(IAWSXRayRecorder recorder, Exception e, String message)
   at Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl.HandleEntityNotAvailableException(EntityNotAvailableException e, String message)
   at Amazon.XRay.Recorder.Core.AWSXRayRecorderImpl.AddMetadata(String key, Object value)
   at CI.LoggerSuite.Api.Controllers.ErrorController.Error() in C:\Users\Iain\projects\LoggerSuite\src\CI.LoggerSuite.Api\Controllers\ErrorController.cs:line 36
   at Microsoft.Extensions.Internal.ObjectMethodExecutor.Execute(Object target, Object[] parameters)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ActionMethodExecutor.SyncActionResultExecutor.Execute(IActionResultTypeMapper mapper, ObjectMethodExecutor executor, Object controller, Object[] arguments)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeActionMethodAsync()
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker.InvokeNextActionFilterAsync()

  This exception was originally thrown at this call stack:
    [External Code]
    CI.LoggerSuite.Api.Controllers.ErrorController.Error() in ErrorController.cs
    [External Code]

hope this helps.