Open haludi opened 1 year ago
Hi, Is there any additional information that we can provide?
Sorry for the delay getting to this. This is interesting... I'm fairly sure none of the changes between 7.0.1 and 7.0.4 (at least on the Kestrel side) would affect this. I see you mentioned that downgrading to 6.0 solved it. How confident are you that 7.0.1 doesn't have the issue?
It might be helpful to see 'Trace' level logs for "Microsoft.AspNetCore". And maybe a dump while the server is in the bad state?
Hi, thank you for the reply.
We had two incidents of that in two separate customers.
One of them upgraded our product from a version with .Net6 and the other upgraded from a version with .Net7.0.1
For both customers, the issue happened only in the production environment and has not happened in the test environment.
For both of them when we installed our latest version with .Net6 the issue was resolved.
Because this is a production environment a dump is not possible.
Next time it will happen I will collect Trace
for Microsoft.AspNetCore
.
To ask our customer to reproduce the issue on purpose to collect the data can be problematic (we already asked one time) but I can check.
Just to be sure, you want to collect 'Trace' for all sub logger of Microsoft.AspNetCore
, right?
Is there any additional information that we can provide? This has caused us to downgrade several customers to .NET 6.0, which is not a long term strategy, obviously.
As mentioned above, Trace level logs for "Microsoft.AspNetCore" would be the best next step (since @haludi said dumps are not going to be possible).
Also curious whether you can test on .NET 8
There is a plan to do so but since we saw that only in production environments and we couldn't reproduce it in a test environment we are very limited
In many/most cases the update is just flip the target framework and rebuild. Do you have the ability to do that, and deploy (perhaps temporarily and limited, just enough to see whether it's fixed)?
That would give another data point but depending on your processes and limitations might be a way to get the fix.
Currently, our customers running on dotnet 6. We created a custom build for them. Our stable currently running on dotnet 7 and should be upgraded to dotnet 8 in the upcoming 2 months. We created a custom build for them to work around until we find another solution. Our customers don't experience issues in dotnet 6. Requesting them to upgrade to dotnet 8 and potentially break their production is problematic. But we will gather more information if another accident happens.
Is there an existing issue for this?
Describe the bug
Our product is a distributed system consisting of multiple ASP servers. There are TCP connections open between the servers. And the servers also communicate with HTTP (version 1.1).
The issue:
The server doesn’t accept HTTP requests between the servers. Even not from the server itself. No errors on Microsoft logs (the configuration below) on the target server. In the source server, we got:
(full stack trace below) The issue starts between 1 hour to a couple of hours after a restart. Azure firewall and OS firewall were checked.
Some details:
One of our customers had an issue in a production environment when upgrading our product. This customer tested the upgrade in a test environment but no issue there. We upgraded the servers again to collect more information. We collected Microsoft logs (the configuration below) and also
tcpdump
. In the Microsoft logs, we saw no error. Ontcpdump
that was collected (for 3 minutes) for the target server, we see all HTTPS packages to the port the server is listening to has no response packages[Conversation completeness: Incomplete, SYN_SENT (1)]
. We didn’t have any changes regarding the server communication handling between our product versions besides upgrading.Net
from.Net6
to.Net7
so we created the same build with one difference, we used.Net6
instead and the issue was solved. Since this is a production we deployed the.Net7
version twice after the first issue to collect the logs and the issue happened again on both times. We also saw the issue again in another customer system, this time the .Net upgrade was between .NET 7.0.1 to .NET 7.0.4 - again using.Net6
build solved the issue.Both customers’ servers were on:
Addition finding:
Microsoft.AspNetCore.Hosting.Diagnostics
, we see only HTTP/2 (this is external communication typically when you access the server from a browser).tcpdump
there are no packages that go out from the HTTPS port even though there are packages that go in.tcpdump
from this time).tcpdump
was collected for 3 minutes while the issue happenedMicrosoft log configuration:
Expected Behavior
No response
Steps To Reproduce
We couldn't reproduce
Exceptions (if any)
Exception Stack Trace:
.NET Version
.NET 7.0.4
Anything else?
No response