search

dotnet / aspnetcore

ASP.NET Core is a cross-platform .NET framework for building modern cloud-based web applications on Windows, Mac, or Linux.
https://asp.net
MIT License
35.23k stars 9.95k forks source link

Missing information about ASP.NET Core worker process recycling requirements and setup options on GNU/Linux servers #42897

Open progmars opened 2 years ago

progmars commented 2 years ago

As many developers and web server administrators know, IIS has a built-in feature for application pool recycling. Here is a refresher for those who might have forgotten about this.

Recycling means that the worker process that handles requests for that application pool is terminated and a new one is started. This is generally done to avoid unstable states that can lead to application crashes, hangs, or memory leaks. By default IIS uses the overlapped recycle method, which keeps the old process up until the current requests are finished processing (or a set timeout elapses) while the new process handles new requests. This ensures service continuity so that you usually do not notice a recycle.

When planning to deploy my ASP.NET Core web apps on Linux servers, I was worried about this mechanism on Linux. I already have one customer complaining that Kestrel process on their server seems to be consuming more and more memory, leading to swapping, while there seem to be no such issues on IIS since the web app gets recycled preemptively.

When setting my ASP.NET Core app on Linux, I followed this article: https://docs.microsoft.com/en-us/aspnet/core/host-and-deploy/linux-nginx?view=aspnetcore-6.0

It describes the recommended Kestrel recovery mechanism using Restart=always and RestartSec=10, which is great. However, it has no mention of recycling requirements and does not provide "the officially recommended setup", or at least some reasonable starting point to achieve something similar to IIS pool recycling. I also could not find any information on the internet regarding this. I don't think that Kestrel on Linux is "magical" and can keep the same process alive for ages without accumulating any "debris" that would benefit from preemptive recycling, so I assume we need something similar to IIS app pool recycling. Correct me if I'm wrong here and Kestrel can and should be left alone running for months without recycling and fears of memory debris.

It would be great for someone experienced to look into this and improve that article or write a new article to clear people's doubts regarding worker process recycle management, so that we have more confidence recommending ASP.NET Core web apps for customers who want to host on Linux servers. Is it at all possible to achieve similar zero-downtime background recycle on Linux with Nginx and Kestrel the same way as it works on IIS? Is it necessary at all? I wouldn't want to reinvent a wheel here, so it would be great to have something time-tested and officially recommended, following best practices and playing nicely with general Kestrel lifecycle mechanics, which we, mere mortal developers, might not know enough to be able to build a reliable solution from scratch.

adityamandaleeka commented 2 years ago

@progmars There's some info here in a prior similar discussion: https://github.com/dotnet/aspnetcore/issues/40147

To answer your other question, in general, it's not necessary to recycle Kestrel processes. There are some cases where you may want to (for instance, if you have a surge in load which causes Kestrel to allocate more in its memory pool, that memory isn't returned when the load drops back down -- we have an issue to address this in the future https://github.com/dotnet/aspnetcore/issues/27394). And of course, it's not uncommon to see people addressing memory leaks in their own server-side code by restarting servers periodically.

We should improve the docs to explain this and show the recommended pattern for doing it on Linux for folks who need/want to.