Allow Background GC Thread Affinization

[ddrinka]

Our application is latency-sensitive, and we're using CPU isolation and thread affinitization on Linux to reduce interruption of our core threads. Garbage Collection is rare for our application, but we'd like to make sure when it is happening, it's happening according to the requirements we've put in place for core utilization.

Assume cores 0 and 1 are housekeeping cores, and 2-8 are isolated, meaning the kernel schedules no tasks for those cores except if the tasks are explicitly affinitized to those cores. We'd like Foreground GC to happen with high priority on cores 2-8, as all .Net threads will be stopped during the Foreground GC anyway. We'd like Background GC to occur with low priority on the housekeeping cores.

Currently, .Net does precisely the opposite of this. Foreground GC occurs with high priority on cores 0 and 1, while all threads are paused on cores 2-8. I believe this can be resolved with careful use of System.GC.HeapAffinitizeRanges (though the default isn't what I would have expected and could potentially be reconsidered).

I haven't identified an option to allow affinitization of the Background GC threads. Is this because those threads must run on the same core as the heap they're servicing? If not, we'd like the ability to configure the cores where Background GC occurs to avoid context switching and work stealing on our isolated cores.

[ghost]

Tagging subscribers to this area: @dotnet/gc See info in area-owners.md if you want to be subscribed.

Issue Details

---

Our application is latency-sensitive, and we're using CPU isolation and thread affinitization on Linux to reduce interruption of our core threads. Garbage Collection is rare for our application, but we'd like to make sure when it is happening, it's happening according to the requirements we've put in place for core utilization. Assume cores 0 and 1 are housekeeping cores, and 2-8 are isolated, meaning the kernel schedules no tasks for those cores except if the tasks are explicitly affinitized to those cores. We'd like Foreground GC to happen with high priority on cores 2-8, as all .Net threads will be stopped during the Foreground GC anyway. We'd like Background GC to occur with low priority on the housekeeping cores. Currently, .Net does precisely the opposite of this. Foreground GC occurs with high priority on cores 0 and 1, while all threads are paused on cores 2-8. I believe this can be resolved with careful use of `System.GC.HeapAffinitizeRanges` (though the default isn't what I would have expected and could potentially be reconsidered). I haven't identified an option to allow affinitization of the Background GC threads. Is this because those threads must run on the same core as the heap they're servicing? If not, we'd like the ability to configure the cores where Background GC occurs to avoid context switching and work stealing on our isolated cores.

Author:	ddrinka
Assignees:	-
Labels:	`area-GC-coreclr`, `untriaged`
Milestone:	-

[Maoni0]

if you set core 2-8 as isolated, are you specifically affinitizing your application threads with these cores to make them run on these cores? my understanding is if you don't do that then none of your threads will run on those cores either. so it sounds like you are doing specific affinitizing yourself. if you could please describe exactly how you are doing this (eg, you are doing this after the runtime started, and used some API to affinitize, or used a tool to affinitize before the process started?), that would be helpful.

by setting core 2-8 as isolated I wonder if it means they are excluded from the process's affintity mask, then by design GC threads will not run on them since we don't run on a core that's simply not allowed by the process. the puzzling part to me is it sounds like you are saying background GC threads are running on core 2-8. that would seem very strange since we don't explicitly affinitize these threads at all.

[ddrinka]

so it sounds like you are doing specific affinitizing yourself. if you could please describe exactly how you are doing this

@Maoni0, you're correct, for the latency-critical threads we're calling Thread.BeginThreadAffinity(); (in case that ever becomes necessary) and then calling sched_setaffinity via pinvoke to work around limitations in .Net on Linux for setting thread affinity.

by design GC threads will not run on them since we don't run on a core that's simply not allowed by the process

This would be the goal of this card--to be able to assign explicit cores for GC threads that are outside of the automatically chosen cores. We could also extend the process affinity to include the isolated cores, but we'd still want to restrict which cores the Background GC can run on within the process affinity mask.

background GC threads are running on core 2-8

This is what we've seen here. Reading the source it looks like those Background GC threads are created and destroyed over the lifetime of the application rather than all at the beginning? If that's the case, then perhaps they're getting created within the affinitized threads.

[Maoni0]

thanks for your explanation.

If that's the case, then perhaps they're getting created within the affinitized threads.

just making sure, you are saying this means if a thread is created by a thread that gets affinitized to one of the isolated cores, it will inherit that attribute and automated be scheduled on any isolated cores?

sounds like you are asking for 2 things -

1) have Server GC threads (the ones that are currently affinitized to core 0/1 run on the isolated cores; and 2) have BGC Server GC threads (the ones that are currently running on core 2-8) run on the housekeeping ones;

please let me know if this is incorrect. these sound like good defaults to me.

[ddrinka]

if a thread is created by a thread that gets affinitized to one of the isolated cores, it will inherit that attribute

Yes, my understanding is that thread affinity masks carry over to new thread creation the same way process affinity carries over to child processes.

sounds like you are asking for 2 things

You've understood my request correctly.

The reason I believe Server GC should happen on any core that has a .Net thread running on it is that the goal of Server GC is to complete as quickly as possible using all available resources. While Server GC is executing, all .Net threads are paused anyway, so it makes sense to use the open space on any core to complete the GC work. For our use case I would prefer to avoid having Server GC run on the housekeeping cores because it runs with elevated priority. We have time synchronization and other OS functions running on the housekeeping cores that shouldn't be deemed "less important" than the GC of one .Net application. But I feel comfortable carving those cores out via configuration. I think the default should be to run on every core that has a .Net thread on it, or to allow explicit configuration of the cores where Server GC should run.

The reason I believe Background GC should happen on the housekeeping cores is that we have elected to isolate certain cores from kernel scheduling to avoid context switches, interrupts, L3 cache invalidation, etc. Running a background task on those cores impacts those benefits. Since Background GC doesn't need to happen urgently, it should compete for space on the housekeeping cores.

[Maoni0]

if you actually have .net threads running on isolated cores, then yes, it's reasonable for the Server GC threads to run on them. but we don't keep track which cores the .net threads are running on, especially if you've done the affinitization yourself.

it seems like a better way is just to provide some way for you to do the affinitization for the GC threads yourself, including the BGC threads. do you agree?

@mangod9 FYI - something that we'd want to look into providing.

[ddrinka]

Yes, particularly given potential adjustments to threads and affinities over the life of the application, it makes the most sense to provide runtime configuration settings to allow customization at startup.

Note that as I dug into System.GC.HeapAffinitizeRanges I mentioned above, I learned that that configuration setting does not actually accomplish the goal of affinitizing the Server GC threads to isolated cores. It only allows further restricting the affinitized ranges down from the startup default. So if I've isolated cores 2-8, I can only set HeapAffinitizeRanges to 0 or 1, or leave it as its default 0 & 1.

[Maoni0]

yeah, as I mentioned above, currently the Server GC threads are only allowed to run on a core that your process is allowed to run on. so clearly this would require some changes in the runtime.