High shmem usage in dotnet 7 runtime docker

arnecornillie-gantner commented 1 year ago

Description

We are running several microservices in an AWS ECS cluster. We are moving from .net 6 to .net 7, but we see the memory usage increased a lot.

Configuration

Create an asp.net core 7 application
Build the application
Use .net 7 docker image from dockerhub ex: "mcr.microsoft.com/dotnet/aspnet:7.0"

Analysis

Compared with .net 6, our docker containers are using more memory. To run our .net6 applications, we are using the 6.0 version of the mentioned docker container.

I did some analysis by exploring the cgroup metrics & I see a big difference in "shmem" parameter.

Dotnet 6: payment_user@80a0abe5eb10:/sys/fs/cgroup/memory$ cat memory.stat cache 61927424 rss 96665600 rss_huge 2097152 **shmem 12288** mapped_file 45645824 dirty 28672 writeback 0 swap 0 pgpgin 49249 pgpgout 11046 pgfault 30832 pgmajfault 536 inactive_anon 98697216 active_anon 20480 inactive_file 57602048 active_file 4313088 unevictable 0 hierarchical_memory_limit 9223372036854771712 hierarchical_memsw_limit 9223372036854771712 total_cache 61927424 total_rss 96665600 total_rss_huge 2097152 total_shmem 12288 total_mapped_file 45645824 total_dirty 28672 total_writeback 0 total_swap 0 total_pgpgin 49249 total_pgpgout 11046 total_pgfault 30832 total_pgmajfault 536 total_inactive_anon 98697216 total_active_anon 20480 total_inactive_file 57602048 total_active_file 4313088 total_unevictable 0

-> shmem 12288

Dotnet 7 payment_user@3c57bc105917:/sys/fs/cgroup/memory$ cat memory.stat cache 48123904 rss 116064256 rss_huge 8388608 **shmem 47308800** mapped_file 47874048 dirty 53248 writeback 0 swap 0 pgpgin 40473 pgpgout 2437 pgfault 71286 pgmajfault 8 inactive_anon 159879168 active_anon 5521408 inactive_file 815104 active_file 0 unevictable 0 hierarchical_memory_limit 9223372036854771712 hierarchical_memsw_limit 9223372036854771712 total_cache 48123904 total_rss 116064256 total_rss_huge 8388608 total_shmem 47308800 total_mapped_file 47874048 total_dirty 53248 total_writeback 0 total_swap 0 total_pgpgin 40473 total_pgpgout 2437 total_pgfault 71286 total_pgmajfault 8 total_inactive_anon 159879168 total_active_anon 5521408 total_inactive_file 815104 total_active_file 0 total_unevictable 0

--> shmem 47308800

Is this an issue in the dotnet 7 runtime? Or any tips to get rid of this behavior?

Thanks!

ghost commented 1 year ago

Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.

Issue Details

### Description We are running several microservices in an AWS ECS cluster. We are moving from .net 6 to .net 7, but we see the memory usage increased a lot. ### Configuration * Create an asp.net core 7 application * Build the application * Use .net 7 docker image from dockerhub ex: "mcr.microsoft.com/dotnet/aspnet:7.0" ### Analysis Compared with .net 6, our docker containers are using more memory. To run our .net6 applications, we are using the 6.0 version of the mentioned docker container. I did some analysis by exploring the cgroup metrics & I see a big difference in "shmem" parameter. Dotnet 6: `payment_user@80a0abe5eb10:/sys/fs/cgroup/memory$ cat memory.stat cache 61927424 rss 96665600 rss_huge 2097152 **shmem 12288** mapped_file 45645824 dirty 28672 writeback 0 swap 0 pgpgin 49249 pgpgout 11046 pgfault 30832 pgmajfault 536 inactive_anon 98697216 active_anon 20480 inactive_file 57602048 active_file 4313088 unevictable 0 hierarchical_memory_limit 9223372036854771712 hierarchical_memsw_limit 9223372036854771712 total_cache 61927424 total_rss 96665600 total_rss_huge 2097152 total_shmem 12288 total_mapped_file 45645824 total_dirty 28672 total_writeback 0 total_swap 0 total_pgpgin 49249 total_pgpgout 11046 total_pgfault 30832 total_pgmajfault 536 total_inactive_anon 98697216 total_active_anon 20480 total_inactive_file 57602048 total_active_file 4313088 total_unevictable 0` -> **shmem 12288** Dotnet 7 `payment_user@3c57bc105917:/sys/fs/cgroup/memory$ cat memory.stat cache 48123904 rss 116064256 rss_huge 8388608 **shmem 47308800** mapped_file 47874048 dirty 53248 writeback 0 swap 0 pgpgin 40473 pgpgout 2437 pgfault 71286 pgmajfault 8 inactive_anon 159879168 active_anon 5521408 inactive_file 815104 active_file 0 unevictable 0 hierarchical_memory_limit 9223372036854771712 hierarchical_memsw_limit 9223372036854771712 total_cache 48123904 total_rss 116064256 total_rss_huge 8388608 total_shmem 47308800 total_mapped_file 47874048 total_dirty 53248 total_writeback 0 total_swap 0 total_pgpgin 40473 total_pgpgout 2437 total_pgfault 71286 total_pgmajfault 8 total_inactive_anon 159879168 total_active_anon 5521408 total_inactive_file 815104 total_active_file 0 total_unevictable 0` --> **shmem 47308800** Is this an issue in the dotnet 7 runtime? Or any tips to get rid of this behavior? Thanks!

Author:	arnecornillie-gantner
Assignees:	-
Labels:	`question`, `tenet-performance`, `area-GC-coreclr`, `untriaged`, `needs-area-label`
Milestone:	-

Maoni0 commented 1 year ago

share memory usage is not from the GC heap (so I'm removing the GC label). @janvorli reminded me of the shared memory usage of W^X. if you disable W^X by setting the DOTNET_EnableWriteXorExecute env var to 0 and the shmem drops to the .net 6 level, you would have confirmed it's from W^X.

arnecornillie-gantner commented 1 year ago

Hi @Maoni0

By setting the "DOTNET_EnableWriteXorExecute" variable to "0" the shared memory is back at the level of .net6.

Thanks!

dotnet / runtime