High memory usage observed during Kestrel stress test

[CristianVladescu]

I have set up a little VS solution to test if Kestrel is long term stable. The solution has 2 projects in it, KestrelMemoryLeak, and StressW. KestrelMemoryLeak is a default RazorPages project listening on port 80 using Kestrel and StressW is a console application used to send a large number of requests to Kestrel, and it's automatically launched when KestrelMemoryLeak is launched.

To reproduce the issue, simply download KestrelMemoryLeak VS solution and press Start debugging. If you monitor memory consumption, you will notice that the memory usage will rapidly raise to 1 GB, after which growing rate will be reduced, but still growing.

I plan to run a small web server on a small linux VM (600 MB RAM) and this makes it worrying. The server won't see this kind of traffic, but I want to make sure is stable in worst case scenario. Is it possible to limit memory usage?

[Tratcher]

@sebastienros

[sebastienros]

Some suggestions:

1- Measure in Release mode, not with Visual Studio attached to the application 2- Measure on the actual environment, not on your dev machine which I assume has a lot of memory available, and the GC doesn't feel any pressure. 3- Run it for hours and you will probably see that it's stabilizing

If you care more about memory than CPU then you can try to disable the Server GC.

I'd also suggest you read this document which will explain some of these points: https://github.com/sebastienros/memoryleak

[davidfowl]

Lets turn that into documentation @sebastienros .

If you care more about memory than CPU then you can try to disable the Server GC.

You should also update your guidance based on this https://blogs.msdn.microsoft.com/maoni/2018/10/02/middle-ground-between-server-and-workstation-gc/

[CristianVladescu]

Ok, I've made a boo-boo in the stress code. I didn't realize HttpClient required to be disposed in order to release unmanaged resources (sockets). I even tried previously disposing it, but for some reason I did not notice the improvement, so I thought is not necessary. The high memory usage was simply because TCP connections were not closed by client so the server had to keep the context for each connection. I might have to set MaxConcurrentConnections for safety.

Now it is very stable, consuming ~200 MB at 2000 req/s.

root@instance-1:~/linux# ./KestrelMemoryLeak
True
Working set after 00:00:00.0000035: 29 MB
Hosting environment: Production
Content root path: /root/linux
Now listening on: http://0.0.0.0:80
Application started. Press Ctrl+C to shut down.
Working set after 00:00:10.0292948: 65 MB
Working set after 00:00:20.0285680: 65 MB
Working set after 00:00:30.0648904: 103 MB
Working set after 00:00:40.0831060: 206 MB
Working set after 00:00:50.1109374: 252 MB
Working set after 00:01:00.4817558: 265 MB
Working set after 00:01:10.4889591: 265 MB
Working set after 00:01:20.6958120: 265 MB
Working set after 00:01:30.7251412: 236 MB
Working set after 00:01:40.7291597: 236 MB
Working set after 00:01:50.9472629: 254 MB
Working set after 00:02:00.9541108: 254 MB
Working set after 00:02:10.9606993: 255 MB
Working set after 00:02:20.9770853: 256 MB
Working set after 00:02:31.1943625: 256 MB
Working set after 00:02:41.4542211: 262 MB
Working set after 00:02:51.6956184: 230 MB
Working set after 00:03:01.7398390: 231 MB
Working set after 00:03:12.1949608: 232 MB
Working set after 00:03:22.2051110: 233 MB
Working set after 00:03:32.2211097: 152 MB
Working set after 00:03:42.2333594: 164 MB
Working set after 00:03:52.2376151: 179 MB
Working set after 00:04:02.4602626: 161 MB
Working set after 00:04:12.7152519: 153 MB
Working set after 00:04:22.7299073: 195 MB
Working set after 00:04:32.7330647: 196 MB
Working set after 00:04:42.9935075: 196 MB
Working set after 00:04:53.4454896: 196 MB
Working set after 00:05:03.4698178: 197 MB
Working set after 00:05:13.4832675: 149 MB
Working set after 00:05:23.4832247: 141 MB
Working set after 00:05:33.7091249: 153 MB
Working set after 00:05:43.9609102: 181 MB
Working set after 00:05:53.9874365: 182 MB
Working set after 00:06:04.4456692: 182 MB
Working set after 00:06:14.6950851: 194 MB
Working set after 00:06:24.7339129: 225 MB
Working set after 00:06:35.2170733: 226 MB
Working set after 00:06:45.4477000: 227 MB
Working set after 00:06:55.4449381: 227 MB
Working set after 00:07:05.4610696: 227 MB
Working set after 00:07:15.4889332: 228 MB
Working set after 00:07:25.7105798: 228 MB
Working set after 00:07:35.7183447: 229 MB
Working set after 00:07:45.7411766: 237 MB
Working set after 00:07:55.9848852: 237 MB
Working set after 00:08:06.2183681: 237 MB
Working set after 00:08:16.9474056: 235 MB
Working set after 00:08:26.9933585: 235 MB
Working set after 00:08:37.4875986: 235 MB
Working set after 00:08:48.6952019: 236 MB
Working set after 00:08:58.7034716: 163 MB
Working set after 00:09:09.2076423: 167 MB
Working set after 00:09:19.2078509: 167 MB
Working set after 00:09:29.4485046: 177 MB
Working set after 00:09:39.4868101: 178 MB
Working set after 00:09:49.9825049: 178 MB
Working set after 00:10:00.7041203: 149 MB

[sebastienros]

Don't dispose the HttpClient because you should actually use a single one, as Static is fine. Use the HttpFactory if you are not sure.