Bad Request: "Malformed request: invalid headers" with Umlaut in Header

[Flyingdot]

Describe the bug

I've noticed you're allowing UTF-8 HTTP Headers since 2.2. (https://github.com/aspnet/KestrelHttpServer/issues/1144). Unfortunately, I still get HTTP 400 back with "Malformed request: invalid headers." for headers with eg. umlauts.

To Reproduce

Steps to reproduce the behavior:

1. dotnet new webapi
2. curl -I 'https://localhost:5001/api/values' -H 'test: ä' -X GET -k
3. See error

   HTTP/1.1 400 Bad Request
   Connection: close
   Date: Tue, 19 Feb 2019 14:49:32 GMT
   Server: Kestrel
   Content-Length: 0

   info: Microsoft.AspNetCore.Server.Kestrel[17]
     Connection id "0HLKMF1OH82SC" bad request data: "Malformed request: invalid headers."
   Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Malformed request: invalid headers.
   at Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException.Throw(RequestRejectionReason reason)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.Http1Connection.TryParseRequest(ReadResult result, Boolean& endConnection)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequests[TContext](IHttpApplication`1 application)
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http.HttpProtocol.ProcessRequestsAsync[TContext](IHttpApplication`1 application)

   Surprisingly, I tried the same thing on macos and there it worked as expected (same sdk version).

Expected behavior

Kestrel accepts headers with Umlauts.

Additional context

I get the error with Kestrel directly, without any reverse proxy configured.

.NET Core SDK (reflecting any global.json):
 Version:   2.2.104
 Commit:    73f036d4ac

Runtime Environment:
 OS Name:     Windows
 OS Version:  10.0.16299
 OS Platform: Windows
 RID:         win10-x64
 Base Path:   C:\Program Files\dotnet\sdk\2.2.104\

Host (useful for support):
  Version: 2.2.2
  Commit:  a4fd7b2c84

.NET Core SDKs installed:
  1.1.10 [C:\Program Files\dotnet\sdk]
  2.1.302 [C:\Program Files\dotnet\sdk]
  2.1.400 [C:\Program Files\dotnet\sdk]
  2.1.402 [C:\Program Files\dotnet\sdk]
  2.2.104 [C:\Program Files\dotnet\sdk]

.NET Core runtimes installed:
  Microsoft.AspNetCore.All 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.All 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
  Microsoft.AspNetCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.AspNetCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 1.0.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 1.1.9 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.1.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
  Microsoft.NETCore.App 2.2.2 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]

[blowdart]

The fact that MacOS accepts it is more concerning

@JunTaoLuo @Tratcher

[Tratcher]

Can you get a Wireshark capture? Is it really utf8 on the wire?

[Flyingdot]

@Tratcher Wireshark and I aren't friends, but I tried... and I think the header is encoded as ISO 8859-1. The package dump:

4745 5420 2f61 7069 2f76 616c 7565 7320
4854 5450 2f31 2e31 0d0a 486f 7374 3a20
6865 6164 6572 2d74 6573 742e 7363 6170
702d 636f 7270 2e73 7769 7373 636f 6d2e
636f 6d0d 0a55 7365 722d 4167 656e 743a
2063 7572 6c2f 372e 3631 2e31 0d0a 4163
6365 7074 3a20 2a2f 2a0d 0a74 6573 743a
20e4 0d0a 0d0a 

I can read the header correctly as ISO 8859-1 but cannot as UTF-8. Sorry for that, I somehow configured my terminal to use UTF-8, but that doesn't seem to work obviously. So I think your side works as expected.

Is there any workaround to get kestrel to accept a ISO 8859-1 encoded header as well? Because that's what we get in the end from our reverse proxy in production.

[Tratcher]

No there's not.