Timeout Exception in Rewrite Middleware

[Eilon]

From @MarcDrexler on February 23, 2018 8:26

I am using the rewrite middleware to filter out requests with specific user agents. My rules are coming from a text file, and I use the AddApacheModRewrite method to apply them.

My Startup.cs looks like this:

using (StreamReader apacheModRewriteStreamReader = File.OpenText("ApacheModRewriteRules.txt"))
{
    var options = new RewriteOptions()
                    .AddApacheModRewrite(apacheModRewriteStreamReader)

    app.UseRewriter(options);
}

And the text file contains a lot of rules in this format:

RewriteEngine on
RewriteCond %{HTTP_USER_AGENT} almaden [OR]
RewriteCond %{HTTP_USER_AGENT} ^Anarchie [OR]
RewriteCond %{HTTP_USER_AGENT} ^ASPSeek [OR]
RewriteCond %{HTTP_USER_AGENT} ^attach [OR]
RewriteCond %{HTTP_USER_AGENT} ^autoemailspider [OR]

...... and 1700 lines more in the same format .....

RewriteCond %{HTTP_USER_AGENT} ^Zeus
RewriteRule ^.* - [F,L]

The filter works, but some requests fail with the following exception:

System.Text.RegularExpressions.RegexMatchTimeoutException: The RegEx engine has timed out while trying to match a pattern to an input string. This can occur for many reasons, including very large inputs or excessive backtracking caused by nested quantifiers, back-references and other factors.
   at System.Text.RegularExpressions.RegexRunner.DoCheckTimeout()
   at System.Text.RegularExpressions.RegexRunner.CheckTimeout()
   at Go60(RegexRunner )
   at System.Text.RegularExpressions.CompiledRegexRunner.Go()
   at System.Text.RegularExpressions.RegexRunner.Scan(Regex regex, String text, Int32 textbeg, Int32 textend, Int32 textstart, Int32 prevlen, Boolean quick, TimeSpan timeout)
   at System.Text.RegularExpressions.Regex.Run(Boolean quick, Int32 prevlen, String input, Int32 beginning, Int32 length, Int32 startat)
   at System.Text.RegularExpressions.Regex.Match(String input, Int32 startat)
   at System.Text.RegularExpressions.Regex.Match(String input)
   at Microsoft.AspNetCore.Rewrite.Internal.UrlMatches.RegexMatch.Evaluate(String pattern, RewriteContext context)
   at Microsoft.AspNetCore.Rewrite.Internal.ApacheModRewrite.ConditionEvaluator.Evaluate(IEnumerable`1 conditions, RewriteContext context, BackReferenceCollection backReferences, Boolean trackAllCaptures)
   at Microsoft.AspNetCore.Rewrite.Internal.ApacheModRewrite.ApacheModRewriteRule.ApplyRule(RewriteContext context)
   at Microsoft.AspNetCore.Rewrite.RewriteMiddleware.Invoke(HttpContext context)
   at Microsoft.AspNetCore.Server.IISIntegration.IISMiddleware.<Invoke>d__11.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.Hosting.Internal.RequestServicesContainerMiddleware.<Invoke>d__3.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Frame`1.<ProcessRequestsAsync>d__2.MoveNext()

I believe these timeouts happen because of that many rules (~1700) and how they are written. Is there another/more efficient way to write these rules so the rewrite engine can work faster without having a timeout? Otherwise I need the ability to increase the timeout.

Copied from original issue: aspnet/Home#2899

[Tratcher]

Likely a duplicate of https://github.com/aspnet/BasicMiddleware/issues/279

[jkotalik]

We already merged something into dev to increase the timeout. We would still need to create a feature to make it configurable.

[muratg]

What @jkotalik said. If increased timeouts are still causing issues, please let us know!

(Also, #279 is tracking the feature to make timeouts configurable.)