aspnet / KestrelHttpServer

[Archived] A cross platform web server for ASP.NET Core. Project moved to https://github.com/aspnet/AspNetCore
Apache License 2.0
2.63k stars 528 forks source link

chrome Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING #1858

Closed amirjalali1 closed 6 years ago

amirjalali1 commented 7 years ago

asp.net core 1.1.2 behind iis arr 3 and behind cloudflare getting Failed to load resource: net::ERR_INCOMPLETE_CHUNKED_ENCODING on chrome

checked with different chrome browser on different machines.

checked chrome extenstions

but i still get the error

any suggestion on how to investigate error more. or how to deal with this error?

halter73 commented 7 years ago

ERR_INCOMPLETE_CHUNKED_ENCODING in chrome usually means that an uncaught exception was thrown from the application in the middle of writing to the response body. If the response body hadn't be written to prior the the exception being thrown, Kestrel would respond with a 500 status code instead of sending an incomplete response.

This can happen for a bunch of reasons. Some examples include throwing from the middle of a razor view, throwing from a property of a object being returned as JsonResult, etc...

The best way to find the exact cause is to look at the asp.net logs. Look for exceptions written to the log after "An unhandled exception was thrown by the application". Unfortunately, the developer exception page doesn't help track down this exception, because the exception is thrown after the response is partially written at which point it's too late to write the exception page to the response.

jmacentee commented 7 years ago

I am also experiencing an intermittent error with net::ERR_INCOMPLETE_CHUNKED_ENCODING

the error is shown in Fiddler as: HTTP/1.1 504 Fiddler - Receive Failure [Fiddler] ReadResponse() failed: The server did not return a complete response for this request. Server returned 1,740 bytes.

We have asp.net logging turned on, but no error is being logged. I am going to look into if we have it configured properly.

I'm not actually sure what possible exception could be getting thrown. On success, we are just returning the contents of index.html without any processing needed. (The route causing the trouble is a deep link into a SPA.)

Here is a very odd thing, though -- I can see all HTTP requests in the IIS logs, unless this error occurs. For this error, there is nothing. Is that the expected behavior when we have "an uncaught exception was thrown from the application in the middle of writing to the response body"?

Thanks for any advice or further insight.

halter73 commented 7 years ago

The expected behavior from Kestrel in this case is to close the connection mid response. This leads to an invalid response because there is no chunked terminator. I don't know why IIS/ANCM logs this the way it does (or doesn't), but to find out what's going on, you should look at the ASP.NET logs, not the IIS logs.

amirjalali1 commented 7 years ago

the problem is that there is no asp.net logs when this error happens.

jmacentee commented 7 years ago

halter73 -- thank you for your reply. 1amirjalai -- that is the same in my case, too. Although, I have not given up on finding a way to increase the logging level, yet.

amirjalali1 commented 7 years ago

is there any way to get better logging for this error?

halter73 commented 7 years ago

@1amirjalai There should be an error logged containing the text "An unhandled exception was thrown by the application". It might be helpful to see the logs you are currently getting. Even at the default levels, there should be something logged for the request.

If you want to enable more verbose logging, you can see how here. LogLevel.Trace is the most verbose level.

jmacentee commented 7 years ago

I got my logging working (nlog was incorrectly configured on this server) and found the underlying error:

System.InvalidOperationException: Headers are read-only, response has already started. at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameHeaders.ThrowHeadersReadOnlyException() at Microsoft.AspNetCore.Server.Kestrel.Internal.Http.FrameHeaders.Microsoft.AspNetCore.Http.IHeaderDictionary.set_Item(String key, StringValues value) at Microsoft.AspNetCore.Http.Internal.DefaultHttpResponse.set_ContentType(String value) at AOD.Common.ErrorHandlingMiddleware.d__5.MoveNext() in d:\a\1\s\trunk\AOD\Middlewares\ErrorHandlingMiddleware.cs:line 52

So, now that I know what I'm looking for, I can fix it. Thank you for the assistance.

cesarblum commented 7 years ago

This means ErrorHandlingMiddleware is trying to set a header (Content-Type) after the response has already started (i.e. headers have already been sent, so you can't change them).

amirjalali1 commented 7 years ago

@halter73 i know how to get log and i have enabled logging at different levels and sections but the problem here is that i do't see no log when this error (which happens a lot) in chrome happens

halter73 commented 7 years ago

@1amirjalai Can you post the logs you did get? A repro app would be better yet.

muratg commented 7 years ago

ping

amirjalali1 commented 7 years ago

@muratg @halter73 i made a har file and sent it to cloudflare support they think the error is because of my anti-virus. but i couldn't find any info about anti-virus on the har file

QuantumHive commented 7 years ago

I've been scratching my head an entire evening. In Chrome 59, I'm sending a POST request (with CORS enabled) to my aspnet core api and I'm getting this error in the Chrome console:

POST http://localhost:3100/api/users net::ERR_INCOMPLETE_CHUNKED_ENCODING

The odd thing is that when I try to reproduce the same request in Firefox, no error is been thrown and the client application does not fail.

My client first sends a preflight request with:

OPTIONS /api/users HTTP/1.1
Host: localhost:3100
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
Access-Control-Request-Method: POST
Origin: http://localhost:3000
User-Agent: Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Mobile Safari/537.36
Access-Control-Request-Headers: content-type
Accept: */*
DNT: 1
Referer: http://localhost:3000/settings/users
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8

Kestrel responds:

HTTP/1.1 204 No Content
Date: Thu, 29 Jun 2017 21:25:26 GMT
Server: Kestrel
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: content-type
Access-Control-Allow-Origin: http://localhost:3000

Then the actual POST request:

POST /api/users HTTP/1.1
Host: localhost:3100
Connection: keep-alive
Content-Length: 201
Pragma: no-cache
Cache-Control: no-cache
Origin: http://localhost:3000
User-Agent: Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Mobile Safari/537.36
content-type: application/json
Accept: */*
DNT: 1
Referer: http://localhost:3000/settings/users
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8
Cookie: io=ZAnqCzeuGggcfQxNAAAG

and Kestrel responds with:

HTTP/1.1 201 Created
Date: Thu, 29 Jun 2017 21:25:26 GMT
Content-Type: application/json; charset=utf-8
Server: Kestrel
Transfer-Encoding: chunked
Location: http://localhost:3100/api/users/2
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: http://localhost:3000
Access-Control-Expose-Headers: *

I've configured my server like this:

public void ConfigureServices(IServiceCollection services)
{
    services.AddCors();
    services.AddMvc();
}

public void Configure(IApplicationBuilder app, IHostingEnvironment environment)
{
    if (environment.IsDevelopment())
    {
        app.UseDeveloperExceptionPage();
    }

    app.UseCors(builder =>
    {
        builder.WithOrigins("http://localhost:3000");
        builder.AllowAnyMethod();
        builder.AllowAnyHeader();
        builder.AllowCredentials();
        builder.WithExposedHeaders("*");
    });

    app.UseMvc();
}

And the controller returns a CreatedAtRoute as described in the web-api tutorial in the docs.

While searching for an answer I've found an old issue at chromium. I'm not sure if I understand the issue correctly, but as I'm understanding it, it seems that they accuse Kestrel (in this case) as a faulty server? Is Kestrel timing-out the socket? It seems that Chrome throws an error when the server does that?

Also when trying to find an answer for this solution, I came across multiple answers saying that an anti-virus is the cause. But I don't have any third party AV. Only thing running on my machine is Windows Defender. And disabling it, doesn't resolve the issue.

halter73 commented 7 years ago

@QuantumHive What does your api/users POST controller action look like?

The key difference between the 204 OPTIONS response and the 201 POST response is that the POST response has a Transfer-Encoding: chunked header.

I noticed that you didn't include the response body to your POST response. Is there a response body? If there isn't, that would definitely explain the ERR_INCOMPLETE_CHUNKED_ENCODING error from Chrome.

QuantumHive commented 7 years ago

@halter73

The action method is similar to the one in the web api tutorial:

[HttpPost]
public IActionResult Create([FromBody] User user)
{
    //create model with EF
    return CreatedAtRoute("GetUser", new { id = user.Id }, user);
}

First of all, when POSTing, there is a request body, which is a JSON format of the user. The good thing is that both in Firefox and Chrome, the POST controller action method is actually getting executed and the entity gets saved into the database. In Firefox the response body is the JSON format returned by the action method from the server. This is what Firefox gets after the HTTP POST request:

Date: Fri, 30 Jun 2017 06:02:16 GMT
Content-Type: application/json; charset=utf-8
Server: Kestrel
Transfer-Encoding: chunked
Location: http://localhost:3100/api/users/4
access-control-allow-credentials: true
Access-Control-Allow-Origin: http://localhost:3000
access-control-expose-headers: *

As you can see, exactly the same as the response header of Chrome. However the key difference is that Firefox does receive a response body:

{
    "id": 4,
    "firstName": "John",
    "lastName": "Doe",
    "email": "JohnDoe@mail.com",
    "mobile": "123456789",
    "birthdate": "1990-01-01T00:00:00",
    "sex": 1
}

whereas Chrome does not, because the response failed.

Again, in this Chromium issue, they seem to say that other browser vendors handle the Transfer-Encoding: chunked differently by ignoring the time-out that the server forced. But Chrome does not ignore it and throws an error, which leads to that the request fails. So does Kestrel force a time-out when sending the chunk?

halter73 commented 7 years ago

@QuantumHive I wouldn't get too hung up on that Chromium issue. This is far from the first report we've received about incomplete chunked responses, and it won't be the last. The 4 issues I linked to might not seem like a lot, but you need to remember that most reporters search for any open issue containing "ERR_INCOMPLETE_CHUNKED_ENCODING" and comment on that instead of opening a new issue even though the root cause is usually different.

All ERR_INCOMPLETE_CHUNKED_ENCODING means is that Chrome did not receive a complete chunked response body or more specifically Chrome never received a terminal "0\r\n\r\n" chunk before some timeout or the TCP connection closing. This can happen for a number of reasons. The most common reason though, is throwing an exception in the middle of serializing the response.

If I were you, I would first take a look at a the response in Fiddler's "Raw" response view to verify that the "0\r\n\r\n" terminal chunk is being sent to Firefox and not Chrome. Note that even Fiddler's Raw view displays "\r\n" as a new line instead of showing escape characters. The fact that Firefox and Chrome are receiving different response bodies is the most peculiar part of what you describe, so I'd want to be extra certain that's actually the case.

Next, I would take a look at the ASP.NET Core logs captured while making the failed request like I mentioned multiple times earlier in this thread. As I mention in those previous comments, this will likely show the unhanded exception that caused the response writing to be cut short.

If your logs don't show an unhandled exception, capture the most verbose log output you can, put it in a gist and link it to here along with the raw request and response copied from Fiddler. We'd also likely need the source for an App that reproduces the behavior, not just a link to a tutorial.

shnepper commented 6 years ago

Clear cookies for site helped me.

halter73 commented 6 years ago

@shnepper Do you know what cookie was causing an incomplete chunked response and why?

shnepper commented 6 years ago

Sorry, I just looked for quick resolution. Cannot reproduce right now. If will happen again - I'll try check little bit deeply. Problem was with WebEx site.

aspnet-hello commented 6 years ago

We periodically close 'discussion' issues that have not been updated in a long period of time.

We apologize if this causes any inconvenience. We ask that if you are still encountering an issue, please log a new issue with updated information and we will investigate.