search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.23k stars 9.95k forks source link

Phantom app_offline.htm Detection Events and Application Shutdowns #55939

Open twinmind opened 3 months ago

twinmind commented 3 months ago

Is there an existing issue for this?

Describe the bug

We run ASP.NET Core web application on dotnet 6 in IIS 10 with in-process ANCM hosting model on Windows Server 2019 Datacenter instances in AWS Elastic Beanstalk. We started seeing 1-2 application shutdowns per machine within 48 hour period causing 503 errors on the application load balancer. Our servers live only 48 hours at most as we replace them with new VMs using autoscaling rules in our Beanstalk environment.

After investigation it was found that there are events with ID 1012 and event data looks like this:

Application 'C:\inetpub\wwwroot\' was recycled after detecting app_offline.htm.

We don't perform any application deployments during the time when we see these events and we don't see any other ways how app_offline.htm file could appear in C:\inetpub\wwwroot\. We are not using Web Deploy for deployments and from the logs it looks like Beanstalk wasn't deploying anything at the time when we see those events. The downtime usually lasts less than a minute and then application comes back online. On the screenshot you can see we got event ID 1032 Application 'C:\inetpub\wwwroot\' started successfully. in 20 seconds after app_offline.htm detection event.

Screenshot 2024-05-28 at 7 41 46 PM

We also don't see any system metrics anomalies during these events, CPU and memory look as usual.

Please advice what could be adding app_offline.htm to application directory and whether it can be a phantom message.

Symptom wise it seems very similar to this https://github.com/dotnet/aspnetcore/issues/24428

Expected Behavior

No app_offline.htm file is expected in site directory C:\inetpub\wwwroot\.

Steps To Reproduce

No response

Exceptions (if any)

No response

.NET Version

6.0.29

Anything else?

.NET SDK: Version: 8.0.204 Commit: c338c7548c Workload version: 8.0.200-manifests.9f663350

Runtime Environment: OS Name: Windows OS Version: 10.0.17763 OS Platform: Windows RID: win-x64 Base Path: C:\Program Files\dotnet\sdk\8.0.204\

.NET workloads installed: There are no installed workloads to display.

Host: Version: 8.0.4 Architecture: x64 Commit: 2d7eea2529

.NET SDKs installed: 6.0.421 [C:\Program Files\dotnet\sdk] 8.0.204 [C:\Program Files\dotnet\sdk]

.NET runtimes installed: Microsoft.AspNetCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.AspNetCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App] Microsoft.NETCore.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.NETCore.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App] Microsoft.WindowsDesktop.App 6.0.29 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App] Microsoft.WindowsDesktop.App 8.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found: x86 [C:\Program Files (x86)\dotnet] registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables: Not set

global.json file: Not found

BrennanConroy commented 1 week ago

We found a few cases where app shutdown could be triggered incorrectly including false notifications of app_offline.htm. See https://github.com/dotnet/aspnetcore/pull/57735

Since this kind of issue is hard to reproduce and diagnose we can't say for sure it'll fix the problem you're having unless you test it out and don't see phantom app_offline notifications anymore.