Consider despawning idle SRV GC threads when DATAS is in use

[jkotas]

Discussed in https://github.com/dotnet/runtime/discussions/102093

^{Originally posted by **neon-sunset** May 10, 2024} Currently, SRV GC + DATAS will nicely scale down heapcount to 1/2/3 heaps on light to moderate long-running allocation traffic, keeping memory usage lower and throughput higher than WKS GC. This is particularly useful and important for lightweight NativeAOT deployments, especially when they are published as native libraries (e.g. https://github.com/dotnet/runtime/discussions/102048) However, as DATAS scales down the heapcount, it does not scale down the respective GC threads, keeping the entire `Environment.ProcessorCount` set of threads alive throughout the lifetime of the application. This leads to a scenario where only one GC thread is active when GC is triggered and all others sit idle without doing useful work. This can be particularly problematic as modern systems often come with lots of cores - due to SMT, even 8-core systems have at least 16 GC threads alive, 32 threads alive for 16C and so on and so forth. This will only keep getting worse as manufacturers add E-cores. It is also already problematic on `osx-*` RIDs as they also have additional autorelease pool thread per core. Has despawning of idle threads that do not have a heap associated with them been considered for DATAS? If so, is there a reasoning, besides "we just did not get to this yet" to keep `.ProcessorCount` of GC threads alive at all times?

[dotnet-policy-service[bot]]

Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.

[mangod9]

Thanks for creating this issue. This is something we have considered and will investigate once current DATAS tuning work is complete.