search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.49k stars 10.04k forks source link

Kestrel will not accept new connection. #55599

Open Ivony opened 6 months ago

Ivony commented 6 months ago

Is there an existing issue for this?

Describe the bug

send request to Kestrel server on Linux get an error: connection reset by peer. netstat shows TCP Connection status is Listening, Recv-Q is 257 and no changes in long term.

in TCP Listening status, Recv-Q means how many connection wait for accept.

memory dump information shows: image

TransportManager._transports[0].AcceptLoopTask.Status is RanToCompletion.

TransportManager._transports is the transport collection of endpoint binding.

diagnosis source code:

in this code will set AcceptLoopTask.Status to RanToCompletion: https://github.com/dotnet/aspnetcore/blob/d6f154cca3863703cf87c8b840eea9cbe20229b2/src/Servers/Kestrel/Core/src/Internal/ConnectionDispatcher.cs#L74

and when this Task completed( break from accept loop ). Kestrel will not accept any connection, looks like go hibernation.

and not found critical error log in log file. so, it's not breaking in https://github.com/dotnet/aspnetcore/blob/d6f154cca3863703cf87c8b840eea9cbe20229b2/src/Servers/Kestrel/Core/src/Internal/ConnectionDispatcher.cs#L70

The presumed path is: https://github.com/dotnet/aspnetcore/blob/d6f154cca3863703cf87c8b840eea9cbe20229b2/src/Servers/Kestrel/Core/src/Internal/ConnectionDispatcher.cs#L51

whatever,

There is possible of go out from accept loop and not quit or crush.

Expected Behavior

No response

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

dotnet 6.0.16

Anything else?

No response

Ivony commented 6 months ago

Proposal:

  1. When leaving accept loop, Kestrel must unbind endpoint or stop listen.
  2. When leaving accept loop, Kestrel should be send notification, by event or otherwise.
  3. Public accept loop state.
amcasey commented 6 months ago

Sorry, I'm having a little trouble understanding what behavior you're seeing. Are you saying that all requests to your kestrel server are rejected with "connection reset by peer"? Or maybe you're saying that the port is still in use after kestrel exits? A sample program demonstrating the problem or even the server logs (ideally, level trace from "Microsoft.AspNetCore") would be very helpful.

785468931 commented 6 months ago

Sorry, I'm having a little trouble understanding what behavior you're seeing. Are you saying that all requests to your kestrel server are rejected with "connection reset by peer"? Or maybe you're saying that the port is still in use after kestrel exits? A sample program demonstrating the problem or even the server logs (ideally, level trace from "Microsoft.AspNetCore") would be very helpful.

curl http://ip:5000/heal All requests return curl: (56) Recv failure: Connection reset by peer

davidfowl commented 6 months ago

@Ivony did you run into this?

amcasey commented 6 months ago

@785468931 Is it possible the connection is being blocked by the firewall on the server? Do you have some reason to believe the request is reaching kestrel and then being rejected?

785468931 commented 6 months ago

@785468931 Is it possible the connection is being blocked by the firewall on the server? Do you have some reason to believe the request is reaching kestrel and then being rejected?

I confirmed that there is no problem with the local firewall. I also got the same error response when using curl localhost（curl http://localhost:5000/heal）.

amcasey commented 6 months ago

@785468931 I think the next step would be to collect a server log so we can see whether it's receiving the connection and, if so, why it's rejecting it. A pcap might also be useful.

amcasey commented 6 months ago

I can't tell whether @785468931 and @Ivony have related problems because I'm still not sure I understand @Ivony's concern/request.