aspnet / AspNetKatana

Microsoft's OWIN implementation, the Katana project
Apache License 2.0
963 stars 332 forks source link

OWIN Stack overflow exception #424

Closed kpatil-hi closed 9 months ago

kpatil-hi commented 3 years ago

We've our application on ASP.NET using following nuget packages:

 <package id="Microsoft.Owin" version="4.1.1" targetFramework="net472" />
  <package id="Microsoft.Owin.Host.SystemWeb" version="4.0.0" targetFramework="net472" />
  <package id="Microsoft.Owin.Security" version="4.1.1" targetFramework="net472" />
  <package id="Microsoft.Owin.Security.Cookies" version="4.0.0" targetFramework="net472" />
  <package id="Microsoft.Owin.Security.Jwt" version="4.0.0" targetFramework="net472" />
  <package id="Microsoft.Owin.Security.OAuth" version="4.0.0" targetFramework="net472" />
  <package id="Microsoft.Owin.Security.OpenIdConnect" version="4.1.1" targetFramework="net472" />
  <package id="Microsoft.Web.Infrastructure" version="1.0.0.0" targetFramework="net472" />
  <package id="Newtonsoft.Json" version="11.0.2" targetFramework="net472" />

Our application has been crashing since sometime due to Stack overflow exception (~70 times per day)

App is hosted on Azure PaaS (P2V2*4). Since sometime we've been trying to capture crash dump using Crash Monitoring tool. But whenever we used to attach debugger. Application won't crash at all. But as soon as we stop Crash monitoring, app will start crashing.

Recently we've captured following stack trace using Proactive Crash monitoring :

========================================================
 Dump Analysis for w3wp__app-ss-sc-pd-eastus-cd__6f6d__PID__8612__Date__07_20_2021__Time_02_28_56PM__20__ntdll!ZwTerminateProcess_pd0mdwk00000N.dmp             
========================================================
Thread 5272
ExitCode 800703E9
ExitCodeString COR_E_STACKOVERFLOW
Managed Exception = System.StackOverflowException:
CallStack - Managed Exception
========================================================
CallStack - Crashing Thread
========================================================
     FaultingExceptionFrame
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1+<Invoke>d__5[[System.__Canon, mscorlib]].MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
     System.Threading.Tasks.Task.FinishContinuations()
     System.Threading.Tasks.Task.Finish(Boolean)
     System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].TrySetException(System.Object)
     System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].SetException(System.Exception)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
     System.Threading.Tasks.Task.FinishContinuations()
     System.Threading.Tasks.Task.Finish(Boolean)
     System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].TrySetException(System.Object)
     System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].SetException(System.Exception)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
     System.Threading.Tasks.Task.FinishContinuations()
     System.Threading.Tasks.Task.Finish(Boolean)
     System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].TrySetException(System.Object)
     System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].SetException(System.Exception)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
     System.Threading.Tasks.Task.FinishContinuations()
     System.Threading.Tasks.Task.Finish(Boolean)
     System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].TrySetException(System.Object)
     System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].SetException(System.Exception)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
     System.Threading.Tasks.Task.FinishContinuations()
     System.Threading.Tasks.Task.Finish(Boolean)
     System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].TrySetException(System.Object)
     System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].SetException(System.Exception)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
     System.Threading.Tasks.Task.FinishContinuations()
     System.Threading.Tasks.Task.Finish(Boolean)
     System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].TrySetException(System.Object)
     System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].SetException(System.Exception)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)
     System.Threading.Tasks.Task.FinishContinuations()
     System.Threading.Tasks.Task.Finish(Boolean)
     System.Threading.Tasks.Task`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].TrySetException(System.Object)
     System.Runtime.CompilerServices.AsyncTaskMethodBuilder`1[[System.Threading.Tasks.VoidTaskResult, mscorlib]].SetException(System.Exception)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     HelperMethodFrame
     System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
     System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
     Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
     System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
     System.Runtime.CompilerServices.AsyncMethodBuilderCore+MoveNextRunner.Run()
     System.Threading.Tasks.AwaitTaskContinuation.RunCallback(System.Threading.ContextCallback, System.Object, System.Threading.Tasks.Task ByRef)

Is there any known issue? If not, Can you please recommend how to capture crash dump or other diagnostics information for such issues?

Thank you!

Tratcher commented 3 years ago

Side note: Use consistent versions for your dependencies. Right now you're mixing 4.0 and 4.1.1 dependencies.

Can you share your application Startup code? The stack traces imply an issue with you how you're calling Map.

kpatil-hi commented 3 years ago

Side note: Use consistent versions for your dependencies. Right now you're mixing 4.0 and 4.1.1 dependencies.

Sure @Tratcher - Noted. Thank you!

Can you share your application Startup code? The stack traces imply an issue with you how you're calling Map.

This application is based on Sitecore 9.3 Federated Authentication Implementation : https://doc.sitecore.com/en/developers/93/sitecore-experience-manager/using-federated-authentication-with-sitecore.html

namespace SCBasics.Foundation.Security.Pipelines.IdentityProviders
{
    public abstract class Wso2IdentityProvider : IdentityProvidersProcessor
    {
        /// <summary>
        /// Unique Name for IdentityProvider.
        /// </summary>
        protected abstract string Name { get; } // This is basically IdentityProviderName but to force derived classes to give unique name we have this property

        protected override string IdentityProviderName => Name;

        protected IdentityProvider IdentityProvider { get; set; }

        /// <summary>
        /// SiteName to resolve other openid settings like ClientId, ClientSecret, etc. for a IdP
        /// </summary>
        public abstract string SiteName { get; }

        private SecurityConfiguration _configuration;
        public SecurityConfiguration Configuration => _configuration ?? (_configuration = SecurityConfiguration.GetConfiguration(SiteName));

        protected Wso2IdentityProvider(FederatedAuthenticationConfiguration federatedAuthenticationConfiguration,
            ICookieManager cookieManager, BaseSettings settings)
            : base(federatedAuthenticationConfiguration, cookieManager, settings)
        {
        }

        protected override void ProcessCore(IdentityProvidersArgs args)
        {
            try
            {
                Assert.ArgumentNotNull(args, "args");
                IdentityProvider = GetIdentityProvider();
                IdentityModelEventSource.ShowPII = true;
                if (IsValidOpenIdSettings())
                {
                    args.App.UseWhen(context => CanExecute(context, new Uri(Configuration.RedirectUri).Host), site =>
                    {
                        var openIdConnectOptions = GetOpenIdConnectOptions();
                        site.UseOpenIdConnectAuthentication(openIdConnectOptions);
                    });
                }
                else
                {
                    Log.Error($"Invalid OpenIdConnect Settings found for '{SiteName}' site.", this);
                }
            }
            catch (Exception ex)
            {
                Log.Error(ex.Message, ex, this);
            }
        }

        /// <summary>
        /// Validate OpenIdConnect Settings
        /// </summary>
        protected virtual bool IsValidOpenIdSettings()
        {
            return !string.IsNullOrWhiteSpace(Configuration.ClientId) && Configuration.WellKnownMetadata.Contains(Uri.UriSchemeHttps);
        }

        /// <summary>
        /// Checks if current request can be executed by this middleware
        /// </summary>
        protected virtual bool CanExecute(IOwinContext context, string hostname)
        {
            return context.Request.Headers.Get("Host").Equals(hostname);
        }

        /// <summary>
        /// Gets OpenIdConnect Setting object.
        /// </summary>
        /// <returns>Returns object of OpenIdConnectAuthenticationOptions if all good, if exception occurs then returns null.</returns>
        protected virtual OpenIdConnectAuthenticationOptions GetOpenIdConnectOptions()
        {
            try
            {
                return new OpenIdConnectAuthenticationOptions
                {
                    MetadataAddress = Configuration.WellKnownMetadata,
                    AuthenticationType = this.GetAuthenticationType(),
                    Authority = Configuration.AuthorityUri,
                    ClientId = Configuration.ClientId,
                    ClientSecret = Configuration.ClientSecret,
                    RedirectUri = Configuration.RedirectUri,
                    PostLogoutRedirectUri = Configuration.PostLogoutRedirectUri,
                    ProtocolValidator = new OpenIdConnectProtocolValidator()
                    {
                        RequireNonce = false,
                        RequireState = false,
                        RequireStateValidation = false
                    },
                    ResponseType = OpenIdConnectResponseType.Code,
                    ResponseMode = "fragment",
                    Scope = OpenIdConnectScope.OpenId,
                    TokenValidationParameters = new TokenValidationParameters
                    {
                        NameClaimType = ClaimTypes.NameIdentifier
                    },
                    CookieManager = CookieManager,
                    Notifications = BindNotifications()
                };
            }
            catch (Exception ex)
            {
                Log.Error(ex.Message, ex, this);
                return null;
            }
        }

        /// <summary>
        /// Bind OpenIdConnect Notifications Handlers. Can be overridden by derived class to override behavior or add to it.
        /// </summary>
        /// <returns></returns>
        protected virtual OpenIdConnectAuthenticationNotifications BindNotifications()
        {
            return new OpenIdConnectAuthenticationNotifications
            {
                AuthenticationFailed = OnAuthenticationFailed,
                MessageReceived = OnMessageReceived,
                RedirectToIdentityProvider = OnRedirectToIdentityProvider,
                SecurityTokenValidated = OnSecurityTokenValidated
            };
        }

        /// <summary>
        /// Invoked if exceptions are thrown during request processing. The exceptions will be re-thrown after this event unless suppressed.
        /// </summary>
        protected virtual Task OnAuthenticationFailed(AuthenticationFailedNotification<OpenIdConnectMessage, OpenIdConnectAuthenticationOptions> notification)
        {
            Log.Error($"FederatedLogin: AuthenticationFailed {notification.Exception.Message}", notification.Exception, this);
            return Task.CompletedTask;
        }

        /// <summary>
        /// Invoked to manipulate redirects to the identity provider for SignIn, SignOut, or Challenge.
        /// </summary>
        protected virtual Task OnRedirectToIdentityProvider(RedirectToIdentityProviderNotification<OpenIdConnectMessage,
            OpenIdConnectAuthenticationOptions> notification)
        {
            // If signing out, add the id_token_hint
            if (notification.ProtocolMessage.RequestType == OpenIdConnectRequestType.Logout)
            {
                var signOutProperties = notification.OwinContext.Get<IDictionary<string, string>>
                    ("security.SignOutProperties");
                if (signOutProperties != null &&
                    signOutProperties.TryGetValue(IdentityModel.OidcConstants.TokenTypes.IdentityToken, out var idTokenClaim))
                {
                    notification.ProtocolMessage.IdTokenHint = idTokenClaim;
                }
            }

            if (notification.ProtocolMessage.RequestType == OpenIdConnectRequestType.Authentication)
            {
                try
                {
                    var themeIdValue = notification.OwinContext.Get<string>(IdentityHelper.ThemeIdQueryStringKey);
                    notification.ProtocolMessage.Parameters.Add(IdentityHelper.ThemeIdQueryStringKey, themeIdValue);
                }
                catch (Exception ex)
                {
                    Log.Error($"Error generating auth parameters for Theme Id for CIAM", ex, this);
                }

                try
                {
                    var layoutIdValue = notification.OwinContext.Get<string>(IdentityHelper.LayoutIdQueryStringKey);
                    notification.ProtocolMessage.Parameters.Add(IdentityHelper.LayoutIdQueryStringKey, layoutIdValue);
                }
                catch (Exception ex)
                {
                    Log.Error($"Error generating auth parameters for Layout Id for CIAM", ex, this);
                }

                try
                {
                    var componentValue = notification.OwinContext.Get<string>(IdentityHelper.ComponentQueryStringKey);
                    notification.ProtocolMessage.Parameters.Add(IdentityHelper.ComponentQueryStringKey, componentValue);
                }
                catch (Exception ex)
                {
                    Log.Error($"Error generating auth parameters for component value for CIAM", ex, this);
                }

                try
                {
                    var cancelUrlValue = notification.OwinContext.Get<string>(IdentityHelper.CancelUrlStringKey);
                    notification.ProtocolMessage.Parameters.Add(IdentityHelper.CancelUrlStringKey, cancelUrlValue);
                }
                catch (Exception ex)
                {
                    Log.Error($"Error generating auth parameters for cancel URL for CIAM", ex, this);
                }

                try
                {
                    var ibeUrlValue = notification.OwinContext.Get<string>(IdentityHelper.IbeUrlStringKey);
                    notification.ProtocolMessage.Parameters.Add(IdentityHelper.IbeUrlStringKey, ibeUrlValue);
                }
                catch (Exception ex)
                {
                    Log.Error($"Error generating auth parameters for cancel URL for CIAM", ex, this);
                }
            }

            return Task.CompletedTask;
        }

        /// <summary>Invoked when a protocol message is first received.</summary>
        protected virtual async Task OnMessageReceived(MessageReceivedNotification<OpenIdConnectMessage, OpenIdConnectAuthenticationOptions> notification)
        {
            var claims = new List<Claim>();
            var openIdConfiguration = notification.Options.ConfigurationManager.GetConfigurationAsync(new CancellationToken()).Result;

            // Exchange code for access and ID tokens
            var tokenClient = new TokenClient(openIdConfiguration.TokenEndpoint, notification.Options.ClientId, notification.Options.ClientSecret);
            var tokenResponse = await tokenClient.RequestAuthorizationCodeAsync(notification.ProtocolMessage.Code, notification.Options.RedirectUri);

            if (tokenResponse.IsError)
            {
                throw new Exception(tokenResponse.Error);
            }

            notification.ProtocolMessage.IdToken = tokenResponse.IdentityToken;
            claims.Add(new Claim(IdentityModel.OidcConstants.TokenTypes.IdentityToken, tokenResponse.IdentityToken));
            claims.Add(new Claim(IdentityModel.OidcConstants.TokenTypes.AccessToken, tokenResponse.AccessToken));
            if (!string.IsNullOrEmpty(tokenResponse.RefreshToken))
            {
                claims.Add(new Claim(IdentityModel.OidcConstants.TokenTypes.RefreshToken, tokenResponse.RefreshToken));
            }

            notification.OwinContext.Request.Environment.Add(nameof(claims), claims);
        }

        /// <summary>
        /// Invoked after the security token has passed validation and a ClaimsIdentity has been generated.
        /// </summary>
        protected virtual Task OnSecurityTokenValidated(SecurityTokenValidatedNotification<OpenIdConnectMessage, OpenIdConnectAuthenticationOptions> notification)
        {
            var claims = default(List<Claim>);
            if (notification.OwinContext.Request.Environment.ContainsKey(nameof(claims)))
                claims = notification.OwinContext.Request.Environment[nameof(claims)] as List<Claim>;

            if (notification.AuthenticationTicket != null)
            {
                if (claims != null)
                {
                    notification.AuthenticationTicket.Identity.AddClaims(claims);
                }

                var identity = notification.AuthenticationTicket.Identity;
                // Added by Kiran based on other learnings - Try to comment if issues
                notification.AuthenticationTicket = new AuthenticationTicket(identity, notification.AuthenticationTicket.Properties);

                var transformationContext = new TransformationContext(FederatedAuthenticationConfiguration, IdentityProvider);
                identity.ApplyClaimsTransformations(transformationContext);
            }

            return Task.CompletedTask;
        }
    }
}

namespace SCBasics.Project.SCBasics.Pipelines.IdentityProviders
{
    public class MySiteIdentityProvider : Wso2IdentityProvider
    {
        protected override string Name => nameof(MySiteIdentityProvider );

        public override string SiteName => "MySite";

        public MySiteIdentityProvider (FederatedAuthenticationConfiguration federatedAuthenticationConfiguration, ICookieManager cookieManager, BaseSettings settings) : base(federatedAuthenticationConfiguration, cookieManager, settings)
        {
        }
    }
}

Sitecore pipeline configuration to register your IDProvider: `

`
kpatil-hi commented 3 years ago

@Tratcher 👍 Thanks for correcting format - Haven't used git comments for code highlighting!

Hope above information helps. Please let me know if you need any other information.

Tratcher commented 3 years ago

I'm still missing the part where that code is called at startup to wire everything up? What's calling IAppBuilder.Map?

kpatil-hi commented 3 years ago

One more thing we also using UseWhen as you might have noticed in ProcessCore method : https://docs.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.builder.usewhenextensions.usewhen?view=aspnetcore-5.0

namespace SCBasics.Foundation.Security.Extensions
{
    using AppFunc = Func<IDictionary<string, object>, Task>;
    using Predicate = Func<IOwinContext, bool>;

    /// <summary>
    /// Extension methods for <see cref="IAppBuilder"/>.
    /// </summary>
    public static class UseWhenExtensions
    {
        /// <summary>
        /// Conditionally creates a branch in the request pipeline that is rejoined to the main pipeline.
        /// </summary>
        /// <param name="app"></param>
        /// <param name="predicate">Invoked with the request environment to determine if the branch should be taken</param>
        /// <param name="configuration">Configures a branch to take</param>
        /// <returns></returns>
        public static IAppBuilder UseWhen(this IAppBuilder app, Predicate predicate, Action<IAppBuilder> configuration)
        {
            if (app == null)
            {
                throw new ArgumentNullException(nameof(app));
            }

            if (predicate == null)
            {
                throw new ArgumentNullException(nameof(predicate));
            }

            if (configuration == null)
            {
                throw new ArgumentNullException(nameof(configuration));
            }

            // Create and configure the branch builder right away; otherwise,
            // we would end up running our branch after all the components
            // that were subsequently added to the main builder.
            var branchBuilder = app.New();
            configuration(branchBuilder);

            return app.Use(new Func<AppFunc, AppFunc>(main =>
            {
                // This is called only when the main application builder 
                // is built, not per request.
                branchBuilder.Run(context => main(context.Environment));
                var branch = (AppFunc)branchBuilder.Build(typeof(AppFunc));

                return context =>
                {
                    if (predicate(new OwinContext(context)))
                    {
                        return branch(context);
                    }
                    else
                    {
                        return main(context);
                    }
                };
            }));
        }
    }
}
kpatil-hi commented 3 years ago

Sure - Got your point - Here you go:

Config to register it as per Sitecore pipeline standards:

<?xml version="1.0"?>
<configuration xmlns:patch="http://www.sitecore.net/xmlconfig/">
  <sitecore>
    <pipelines>
      <owin.initialize>
        <processor type="SCBasics.Foundation.Security.Pipelines.OwinInitialize.HandleLoginLink, SCBasics.Foundation.Security"
                   patch:instead="processor[@type='Sitecore.Owin.Authentication.Pipelines.Initialize.HandleLoginLink, Sitecore.Owin.Authentication']" resolve="true"/>
      </owin.initialize>
          </pipelines>
  </sitecore>
</configuration>
namespace SCBasics.Foundation.Security.Pipelines.OwinInitialize
{
    public class HandleLoginLink : Sitecore.Owin.Authentication.Pipelines.Initialize.HandleLoginLink
    {
        public HandleLoginLink(SignInStatusHandler signInStatusHandler, BaseSettings settings, ApplicationUserFactory applicationUserFactory, FederatedAuthenticationConfiguration federatedAuthenticationConfiguration, LoginLinksGenerator loginLinksGenerator, BaseLog log) : base(signInStatusHandler, settings, applicationUserFactory, federatedAuthenticationConfiguration, loginLinksGenerator, log)
        {
        }

        protected override void HandleExternalLoginUrl(InitializeArgs args)
        {
            Assert.ArgumentNotNull(args, "args");
            args.App.Map(string.Concat(Settings.IdentityProcessingPathPrefix().EnsureTrailingSlash(), "externallogin"), SessionStateBehavior.Required, app => app.Run(context =>
            {
                if (!string.Equals(context.Request.Method, "POST", StringComparison.OrdinalIgnoreCase))
                {
                    WebUtil.RedirectToErrorPage("HTTP POST method is required for login requests");
                    return Task.CompletedTask;
                }

                var redirectUri = context.Request.Query.Get("ReturnUrl");
                var str = context.Request.Query.Get("authenticationType");

                // Custom logic to add custom parameters - START
                var queryStringParameters = context.Request.Query;
                var parametersList = new List<string>
                {
                    IdentityHelper.CancelUrlStringKey,
                    IdentityHelper.ComponentQueryStringKey,
                    IdentityHelper.LayoutIdQueryStringKey,
                    IdentityHelper.ThemeIdQueryStringKey,
                    IdentityHelper.IbeUrlStringKey
                };
                foreach (var parameter in queryStringParameters)
                {
                    if (!parametersList.Contains(parameter.Key))
                        continue;

                    var value = parameter.Value?.FirstOrDefault();
                    if (!string.IsNullOrWhiteSpace(value))
                        context.Set(parameter.Key, value);

                    // 'ibeurl' that we expect here comes from booking widget JS
                    // Update RedirectUri if ibeurl QueryString is present
                    if (parameter.Key == IdentityHelper.IbeUrlStringKey)
                    {
                        // extract all QueryStrings from redirectUri
                        var listQueryString = WebUtil.ParseUrlParameters(redirectUri);

                        // Get inner redirect url from QueryString
                        /*
                         Url Sample:
                         /identity/externallogin?authenticationType=MySiteIdentityProvider&ReturnUrl=%2fidentity%2fexternallogincallback%3fReturnUrl%3d%252f%26sc_site%3dHollywood%26authenticationSource%3dDefault&sc_site=Hollywood&component=guest&layout_id=BBG&theme_id=HWD_1
                         */
                        var returnUrl = listQueryString["ReturnUrl"];

                        // Build ibeurl QueryString
                        var ibeParameter = $"{parameter.Key}={WebUtil.UrlEncode(parameter.Value?.FirstOrDefault() ?? "")}";

                        if (returnUrl.Contains("?"))
                        {
                            // check if return url already has query parameters then add ampersand
                            returnUrl += $"&{ibeParameter}";
                        }
                        else
                        {
                            // check if return url has no query parameters then add question mark
                            returnUrl += $"?{ibeParameter}";
                        }

                        listQueryString["ReturnUrl"] = WebUtil.UrlEncode(returnUrl); // update return url in list

                        // get relative base path
                        var path = WebUtil.GetLocalPath(redirectUri); // expecting path to be '/identity/externallogincallback'
                        redirectUri = $"{path}?{WebUtil.BuildQueryString(listQueryString.ToSafeDictionary(), false)}";
                    }
                }
                // custom logic - END
                var authentication = context.Authentication;
                var authenticationProperty = new AuthenticationProperties { RedirectUri = redirectUri };
                authentication.Challenge(authenticationProperty, str);
                return Task.CompletedTask;
            }));
        }
    }
}
Tratcher commented 3 years ago

The Map looks OK, but the stack implies it's being called 7 times?

Eventually you end up in an AuthenticationMiddleware which implies you're not taking the map branch. In hindsight Map could have been implemented more efficiently not to create an async state machine for the negative branch. If you have a lot of calls to Map this might be worth doing locally. https://github.com/aspnet/AspNetKatana/blob/fae38fae683bb0977bcaf7c3ffce244aced82135/src/Microsoft.Owin/Mapping/MapMiddleware.cs#L51-L74

kpatil-hi commented 3 years ago

The Map looks OK, but the stack implies it's being called 7 times?

Thanks for checking @Tratcher - Yeah that's what this SO stack-trace indicates. Unfortunately proactive crash monitoring doesn't provide raw dump file. To investigate this further. And when we use Crash monitoring application doesn't crash at all. We've MSFT support ticket already opened for this : TrackingID#2106160040003513 Is there any other way to capture crash dump for such issues? Do you think Enabling OWIN logging will help? https://www.tugberkugurlu.com/archive/logging-in-the-owin-world-with-microsoft-owin--introduction

So, as per Stack-trace "Microsoft.Owin.Mapping.MapMiddleware+d__3.MoveNext()" this invokes HandleLoginLink.HandleExternalLoginUrl function? And somehow it is getting called multiple times? Is that right understanding?

Eventually you end up in an AuthenticationMiddleware which implies you're not taking the map branch. In hindsight Map could have been implemented more efficiently not to create an async state machine for the negative branch. If you have a lot of calls to Map this might be worth doing locally.

I will go through your reply and let you know if I've more questions

Thanks for your quick help - really appreciate!

Tratcher commented 3 years ago

Do you think Enabling OWIN logging will help?

Unlikely. Stepping through the startup logic in the debugger might help explain why it looks like there are multiple maps added to the pipeline.

So, as per Stack-trace "Microsoft.Owin.Mapping.MapMiddleware+d__3.MoveNext()" this invokes HandleLoginLink.HandleExternalLoginUrl function? And somehow it is getting called multiple times? Is that right understanding?

The other way around, it seems like HandleExternalLoginUrl is called multiple times so Map gets called multiple times and builds up a long app processing pipeline.

kpatil-hi commented 3 years ago

Unlikely. Stepping through the startup logic in the debugger might help explain why it looks like there are multiple maps added to the pipeline.

I debugged it in my local and what I found is for each login request this map gets called. For example whenever user clicks on login following POST request gets triggered and it calls HandleExternalLoginUrl every time - Is that normal behavior?

/identity/externallogin?authenticationType=SieIdentityProvider&ReturnUrl=%2fidentity%2fexternallogincallback%3fReturnUrl%3d%252f%253fdrawer%253dlogin%26sc_site%3dTampa%26authenticationSource%3dDefau >>The other way around, it seems like HandleExternalLoginUrl is called multiple times so Map gets called multiple times and builds up a long app processing pipeline. You are right and as I understand that's not normal behavior - correct? If yes, then I will have to work with Sitecore CMS Dev Team to figure this out.
Tratcher commented 3 years ago

I suspect you're not supposed to change the app pipeline in HandleExternalLoginUrl if it's called multiple times.

kpatil-hi commented 3 years ago

Sure - This is OOTB code of HandleExternalLoginUrl. So, I don't see big change. But will confirm with Sitecore team image

So, @Tratcher what else we should check to pin point this issue? Shall I do load test staging app using this URL? "/identity/externallogin?authenticationType=SieIdentityProvider&ReturnUrl=%2fidentity%2fexternallogincallback%3fReturnUrl%3d%252f%253fdrawer%253dlogin%26sc_site%3dTampa%26authenticationSource%3dDefau"

Tratcher commented 3 years ago

You'll need to ask Sitecore to clarify the expected usage. It's strange to call Map after Startup.

kpatil-hi commented 3 years ago

Sure Sir - Will keep you posted!

kpatil-hi commented 3 years ago

@Tratcher : Quick update, We've been able to capture Crash dump after tweaking Crash monitoring configurations

System.Uri.CreateThis(System.String, Boolean, System.UriKind)
SCBasics.Foundation.Security.Pipelines.OwinInitialize.HandleIbeUrl.UpdateLoggedInIbeUrlIfNeeded(System.String)
SCBasics.Foundation.Security.Pipelines.OwinInitialize.HandleIbeUrl.<HandleExternalIbeUrl>b__8_1(Microsoft.Owin.IOwinContext)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage+<RunApp>d__5.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(System.Threading.Tasks.Task)
Microsoft.Owin.Mapping.MapMiddleware+<Invoke>d__3.MoveNext()

Here's code: image

Here ibeUrlParam value was coming as "undefined" in one of the specific scenario. Which we are fixing by adding graceful exception handling. But can you please help us understand why URI Format exception can cause SO exception? Is it because callback delegate function runs on different thread?

Thank you so much for your help!

Tratcher commented 3 years ago

I doubt URI Format is causing the SO directly, it's just the code that was running when you finally ran out of space. Looks like Map was called 23 times? Is HandleExternalIbeUrl called per request? Map should not be called per request.

Tratcher commented 9 months ago

Closing as idle.