Web crawlers cause ArgumentException: Illegal characters in path and The path must start with a '/' followed by one or more characters.

[abatishchev]

Stack trace:

System.ArgumentException: The path must start with a '/' followed by one or more characters.
Parameter name: value
   at Microsoft.Owin.PathString..ctor(String value)
   at Microsoft.Owin.OwinRequest.get_Path()
   at Microsoft.Owin.OwinRequest.get_Uri()
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.CreateRequestMessage(IOwinRequest owinRequest, HttpContent requestContent)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

and

System.ArgumentException: Illegal characters in path.
   at System.IO.Path.CheckInvalidPathChars(String path, Boolean checkAdditional)
   at System.IO.Path.GetExtension(String path)
   at Microsoft.Owin.StaticFiles.ContentTypes.FileExtensionContentTypeProvider.TryGetContentType(String subpath, String& contentType)
   at Microsoft.Owin.StaticFiles.StaticFileContext.LookupContentType()
   at Microsoft.Owin.StaticFiles.StaticFileMiddleware.Invoke(IDictionary`2 environment)
   at Microsoft.Owin.StaticFiles.DefaultFilesMiddleware.Invoke(IDictionary`2 environment)
   at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__20.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

Environment:

.NET Framework 4.6.2
Azure Service Fabric

Packages:

<PackageReference Include="Microsoft.Owin" Version="4.2.1" />
<PackageReference Include="Microsoft.Owin.FileSystems" Version="4.2.1" />
<PackageReference Include="Microsoft.Owin.Host.HttpListener" Version="4.2.1" />
<PackageReference Include="Microsoft.Owin.Hosting" Version="4.21" />
<PackageReference Include="Microsoft.Owin.Security" Version="4.2.1" />
<PackageReference Include="Microsoft.Owin.Security.ActiveDirectory" Version="4.2.10" />
<PackageReference Include="Microsoft.Owin.StaticFiles" Version="4.2.1" />

[Tratcher]

What's the request, something like OPTIONS * HTTP/1.1 where the * ended up in the path? These old frameworks didn't support those kinds of requests. You should be able to filter them out with middleware like:

app.Use((context, next) =>
{
  if (string.Equals(context["owin.RequestPath"], "*"))
  {
    context.Response.StatusCode = 400;
    return Task.FromResult(0);
  }
  return next();
});

[abatishchev]

I'm not sure unfortunately what's the exact request. The service is old, not modern and doesn't log failed requests such as these.

What would work better:

string.Equals(context.Request.Path.Value, "*", StringComparison.InvariantCulture)

or

string.Equals((string)context.Environment["owin.RequestPath"], "*", StringComparison.InvariantCulture)

[Tratcher]

You can't use Request.Path without triggering the same exception as above, you'll need to read the Environment path key directly.