Inconsistent server response when using JWT bearer tokens to authenticate

I have been noticing inconsistent behavior with my web api ever since upgrading it to asp.net core. The relevant components of the project are

An asp.net core authentication server (implemented via identity server 4)
A web api (recently migrated from .net framework 4.5.1 to asp.net core)

Use Case 1

Client sends an unauthenticated request to the resource server without a token
Server responds with a 401 and no content

Use Case 2

Client retrieves a valid token using the client credentials grant flow from the auth server
Client sends an authenticated request to the resource server with the token
Server responds with a 200 and access to the protected resource

This worked consistently when the web api was on .net framework. However since the switch to .net core, the behavior is as follows

Client sends an unauthenticated request to the resource server without a token
Server responds with either:
1. 200 and no content
2. 401 and no content
Client retrieves token using the client credentials grant flow from the auth server
Client sends an authenticated request to the resource server with the token
Server responds with either:
1. 200 and access to the protected resource
2. 401 and no content
3. 200 and no content (odd because the api would never return no content)

There doesn't seem to be any sort of pattern to the server responses.

The startup is posted below. I have tried replacing the .AddIdentityServerAuthentication with .AddJwtBearer, but the behavior was not affected.

I do not believe this is an issue with the authentication server project because no code was changed and the exact same steps to obtain a token were used on the .net framework and the asp.net core versions of the web api project.

I do not believe this is an environmental issue as I have seen it in a dev environment, UAT environment, and on my local machine.

The only logs I have been able to grab so far has been from our transaction log because I don't believe any exception gets thrown

I realize this a very generic issue, but I am having a hard time debugging it because it appears most of this is happening in the middleware. If there is any other info I can provide please let me know. Thank you in advance for your time.

dotnet --info
.NET Core SDK (reflecting any global.json):
 Version:   2.1.301
 Commit:    59524873d6

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.17134
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.1.301\

Host (useful for support):
  Version: 2.1.1
  Commit:  6985b9f684

.NET Core SDKs installed:
  1.1.0 [C:\Program Files\dotnet\sdk]
  2.1.4 [C:\Program Files\dotnet\sdk]
  2.1.104 [C:\Program Files\dotnet\sdk]
  2.1.200 [C:\Program Files\dotnet\sdk]
  2.1.301 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.5 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.6 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.0.7 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.1 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

To install additional .NET Core runtimes or SDKs:
  https://aka.ms/dotnet-download

//                     LOGS                                  //

**no token 200**

2018-09-20 10:24:24.954 -05:00 [INF] Request starting HTTP/1.1 GET http://localhost:50107/api/v4/category  
2018-09-20 10:24:24.954 -05:00 [INF] Route matched with {action = "GetAll", controller = "CategoryV4", page = ""}. Executing action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost)
2018-09-20 10:24:24.955 -05:00 [INF] Authorization failed.
2018-09-20 10:24:24.955 -05:00 [INF] Authorization failed for the request at filter 'Microsoft.AspNetCore.Mvc.Authorization.AuthorizeFilter'.
2018-09-20 10:24:24.955 -05:00 [INF] Executing ChallengeResult with authentication schemes ([]).
2018-09-20 10:24:24.955 -05:00 [INF] AuthenticationScheme: Bearer was challenged.
2018-09-20 10:24:24.955 -05:00 [INF] Executed action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost) in 0.5895ms
2018-09-20 10:24:24.955 -05:00 [INF] Request finished in 1.17ms 200 

**no token 401**

2018-09-20 10:24:23.478 -05:00 [INF] Request starting HTTP/1.1 GET http://localhost:50107/api/v4/category  
2018-09-20 10:24:23.532 -05:00 [INF] Route matched with {action = "GetAll", controller = "CategoryV4", page = ""}. Executing action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost)
2018-09-20 10:24:23.544 -05:00 [INF] Authorization failed.
2018-09-20 10:24:23.545 -05:00 [INF] Authorization failed for the request at filter 'Microsoft.AspNetCore.Mvc.Authorization.AuthorizeFilter'.
2018-09-20 10:24:23.548 -05:00 [INF] Executing ChallengeResult with authentication schemes ([]).
2018-09-20 10:24:23.912 -05:00 [INF] AuthenticationScheme: BearerIdentityServerAuthenticationJwt was challenged.
2018-09-20 10:24:23.912 -05:00 [INF] AuthenticationScheme: Bearer was challenged.
2018-09-20 10:24:23.915 -05:00 [INF] Executed action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost) in 380.02320000000003ms
2018-09-20 10:24:23.917 -05:00 [INF] Request finished in 439.4776ms 401 

**with token 401**

2018-09-20 10:25:20.177 -05:00 [INF] Request starting HTTP/1.1 GET http://localhost:50107/api/v4/category  
2018-09-20 10:25:20.178 -05:00 [INF] Route matched with {action = "GetAll", controller = "CategoryV4", page = ""}. Executing action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost)
2018-09-20 10:25:20.178 -05:00 [INF] Authorization failed.
2018-09-20 10:25:20.178 -05:00 [INF] Authorization failed for the request at filter 'Microsoft.AspNetCore.Mvc.Authorization.AuthorizeFilter'.
2018-09-20 10:25:20.178 -05:00 [INF] Executing ChallengeResult with authentication schemes ([]).
2018-09-20 10:25:20.178 -05:00 [INF] AuthenticationScheme: Bearer was challenged.
2018-09-20 10:25:20.178 -05:00 [INF] Executed action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost) in 0.4509ms
2018-09-20 10:25:20.178 -05:00 [INF] Request finished in 0.773ms 401 

**with token 200 with content**

2018-09-20 10:25:53.738 -05:00 [INF] Request starting HTTP/1.1 GET http://localhost:50107/api/v4/category  
2018-09-20 10:25:54.203 -05:00 [INF] Successfully validated the token.
2018-09-20 10:25:54.211 -05:00 [INF] Route matched with {action = "GetAll", controller = "CategoryV4", page = ""}. Executing action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost)
2018-09-20 10:25:54.212 -05:00 [INF] Authorization was successful.
2018-09-20 10:25:54.267 -05:00 [INF] Executing action method Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost) - Validation state: "Valid"
2018-09-20 10:25:54.346 -05:00 [INF] Executed action method Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost), returned result Microsoft.AspNetCore.Mvc.ObjectResult in 74.5571ms.
2018-09-20 10:25:54.357 -05:00 [INF] Executing ObjectResult, writing value of type 'System.Linq.Enumerable+SelectArrayIterator`2[[System.Object, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e],[Clarity.Api.v4.Category_v4, Clarity.Api, Version=3.19.0.0, Culture=neutral, PublicKeyToken=null]]'.
2018-09-20 10:25:54.394 -05:00 [INF] Executed action Clarity.WebHost.Controllers.v4.CategoryV4Controller.GetAll (Clarity.WebHost) in 183.63160000000002ms
2018-09-20 10:25:54.396 -05:00 [INF] Request finished in 658.2099ms 200 application/json; charset=utf-8

//                   STARTUP                                     //

Startup
public class Startup
{
    public IConfiguration Configuration { get; }

    public IConfigSettings ConfigSettings { get; set; }

    public Startup(IConfiguration configuration)
    {
        Configuration = configuration;

        ConfigSettings = configuration.BuildClarityConfigSettings();
    }

    // This method gets called by the runtime. Use this method to add services to the container.
    public virtual IServiceProvider ConfigureServices(IServiceCollection services)
    {
        services.AddAuthentication(options =>
            {
                options.DefaultScheme = IdentityServerAuthenticationDefaults.AuthenticationScheme;
                options.DefaultAuthenticateScheme = IdentityServerAuthenticationDefaults.AuthenticationScheme;
                options.DefaultChallengeScheme = IdentityServerAuthenticationDefaults.AuthenticationScheme;

            })
            .AddIdentityServerAuthentication(options =>
            {
                options.Authority = ConfigSettings.AuthServerAddress;
                options.RequireHttpsMetadata = false;
                //options.ApiName = "clarity-stock";
            });

        services
            .AddMvc();

        return services.BuildClarityServiceProvider(ConfigSettings);
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public virtual void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
    {
        app.UseAuthentication();

        app.UseMvc();

        //initialize static stuff that depends on service registration
        if (app.ApplicationServices.GetService<WorkerConfig>().ChannelFactoryType == typeof(DirectChannelFactory))
        {
            app.ApplicationServices.GetService<IWorkerServer>().Start();
        }

        EventDispatcher.Raise(new ApplicationStartedEvent(app.ApplicationServices)).GetAwaiter().GetResult();
    }
}

aspnet / Identity

Inconsistent server response when using JWT bearer tokens to authenticate #1971