runtime: let idle OS threads exit

[brianmario]

1. What version of Go are you using (go version)? 1.5.3, 1.6
2. What operating system and processor architecture are you using (go env)? x86_64 - OSX and Linux
3. What did you do? Any golang program will create a new OS thread when it needs to if things are blocked. But these threads aren't ever destroyed. For example, a program using 7 goroutines might have 40+ OS threads hanging around. The numbers will surely get much higher as traffic fluctuates against a golang server process throughout the day.
4. What did you expect to see? Once an OS thread has been idle long enough, I would have expected it to be destroyed. Being recreated if needed. Expanding and contracting like the relationship between the heap and GC.
5. What did you see instead? The many OS threads that were created hang around even with an idle program and very few goroutines.

---

After doing some reading into other (closed) issues on this repo dating back to 2012 - including the one where SetMaxThreads was introduced - I'm curious, why keep the OS threads around instead of cleaning them up?

[Qubitium]

Cleaning up the idles would benefit mobile (android) deployments since threads are very precious resource there. Perhaps a settings to toggle cleanup idle threads or not is appropriate for those that would trade absolute performance for reduced resource usage.

[bradfitz]

I don't think this has anything to do with GC. It sounds like a general runtime issue instead. Assigning to @aclements to triage for Go 1.8.

[brianmario]

I don't think this has anything to do with GC.

Cleaning up new threads right after they've been used seemed potentially wasteful, so obviously some form of periodic cleanup felt more realistic. GC is a timeframe the runtime is already setting aside for periodic cleanup, so that's what I went with initially. But I agree that it doesn't necessarily need to be tied to the GC for any particular reason.

[brianmario]

I know there's plenty of other work on everyone's respective plates but has anyone had a chance to think about this some more?

[ayanamist]

As i know, GC will return allocated heap memory to system after 5 minutes idle, can it also exit idle threads?

[aclements]

This is certainly technically possible, but the devil's in the details. For example, we often manipulate pointers to Ms (the structure representing an OS thread) in contexts that don't interact with stop-the-world and can't have write barriers, which means it's difficult to know when we're actually done with an M and can safely recycle it. Even if we don't release the M structure itself and just release the OS thread, we have to know when we can safely reuse the M for another OS thread to avoid races.

[ianlancetaylor]

I'm probably missing some subtlety, but it seems to me that in stopm we could change the notesleep to a notetsleep that sleeps for five minutes. If that sleep times out, we discard the thread and go back to a regular notesleep. If we're woken up from that, we spin up a new thread.

I don't know whether this is actually worth doing--does an idle thread really use significant resources?--but I think it seems feasible to do it.

[aclements]

@ianlancetaylor, I'm not sure I understand your suggestion. If we discard the thread, who's calling notesleep?

[ayanamist]

@ianlancetaylor Pure go code will not be affected by these kind of idle threads since there won't be many idle threads, however, cgo codes or blocking syscalls (like cgo dns resolve or disk io related operations) will make a lot of threads, which may slow down the machine.

[ianlancetaylor]

@aclements You're right, of course. That was dumb.

[bradfitz]

Punting to Go 1.10.

[gopherbot]

CL https://golang.org/cl/46037 mentions this issue.

[aclements]

Issue #22439 has an example of a problem caused by not exiting idle Ms (and freeing their stack memory).

[alexbrainman]

I just discovered that exiting threads might affect IO. From WSASendTo documentation https://msdn.microsoft.com/en-us/library/windows/desktop/ms741693(v=vs.85).aspx - "... Note All I/O initiated by a given thread is canceled when that thread exits. For overlapped sockets, pending asynchronous operations can fail if the thread is closed before the operations complete. See ExitThread for more information. ..."

Alex

[aclements]

I just discovered that exiting threads might affect IO.

Oh, geez. Good call.

Could this be a problem even for the current thread exiting we do? Something like:

1. G1 on M1 starts an overlapped I/O operation.
2. G1 migrates to M2.
3. G2 runs on M1, calls runtime.LockOSThread, and then exits, causing M1 to exit.

Is there any way to query whether there are pending I/O operations on a thread? I'm wondering if we could just block a thread from exiting until all of its pending I/Os are done.

[alexbrainman]

Could this be a problem even for the current thread exiting we do? Something like:

G1 on M1 starts an overlapped I/O operation. G1 migrates to M2. G2 runs on M1, calls runtime.LockOSThread, and then exits, causing M1 to exit.

I did not know that OS threads are exiting at this moment. Is there a way to verify your scenario above? What would be the program to do that? I can do 1. by reading from a TCP connection, but how would do I force 2. and 3?

Is there any way to query whether there are pending I/O operations on a thread?

I do not know of any. You could call GetCurrentThreadId Windows API to get current thread id, but you, probably, can already work that information out of whatever you have recorded for Gs and Ms. Surely we could mark Ms when IO starts, and remove that mark once IO completes.

Alex

[gopherbot]

Change https://golang.org/cl/85662 mentions this issue: runtime: remove special handling of g0 stack

[kolayne]

Is this fixed in some go version?

[ianlancetaylor]

@kolayne Not yet.

[superajun-wsj]

maybe we could reduce M number by using runtime.LockOSThread(), and here is a demo to illustrate how it works.

when a LockOSThread goroutine exit without calling UnLockOSThread, the corresponding M will exit.

[ianlancetaylor]

@superajun-wsj Please add text as text, not as an image. Actual text is much easier to read than text in an image. Thanks.

[fuweid]

when a LockOSThread goroutine exit without calling UnLockOSThread, the corresponding M will exit.

Hi @superajun-wsj Not sure that it is good solution. When the child process is created by one thread called A with PdeathSignal: SIGKILL and the thread A becomes idle, if the thread A exits, the child process will receive KILL signal. So I think UnLockOSThread might introduce other issues. just my two cents.

[greenstatic]

Any news regarding this?

In my particular case I am developing for IoT hardware running Linux that only has 1 vCPU and 128 MB of memory. From my testing I am estimating that a single thread in my case has a 80 KiB memory overhead (which is strange in itself since a Ubuntu 20.04 LTS amd64 server I also tested on approx. has an overhead of only 8 KiB). My program after running for a week uses on average 40 goroutines at any given time, however the thread count after a week is 220+. This would mean the remaining threads (if each goroutine is scheduled on it's own thread) memory overhead is over 14 MiB (220 - 40 = 180 threads * 80 KiB ~= 14 MiB)! I know that doesn't seem like a lot however on these memory constrained devices that run Linux it's still an unnecessary resource hog.

I would be perfectly happy if there was a function that we could call in the runtime package to trigger a GC-like thread cleanup.

[LeGamerDc]

Any news regarding this?

In my particular case I am developing for IoT hardware running Linux that only has 1 vCPU and 128 MB of memory. From my testing I am estimating that a single thread in my case has a 80 KiB memory overhead (which is strange in itself since a Ubuntu 20.04 LTS amd64 server I also tested on approx. has an overhead of only 8 KiB). My program after running for a week uses on average 40 goroutines at any given time, however the thread count after a week is 220+. This would mean the remaining threads (if each goroutine is scheduled on it's own thread) memory overhead is over 14 MiB (220 - 40 = 180 threads * 80 KiB ~= 14 MiB)! I know that doesn't seem like a lot however on these memory constrained devices that run Linux it's still an unnecessary resource hog.

I would be perfectly happy if there was a function that we could call in the runtime package to trigger a GC-like thread cleanup.

if you go process did not create thread itself(like by set block to socket), you could use debug.SetMaxThreads to limit max number of threads go create

[omar391]

Any update on this?

[ianlancetaylor]

There is no update.

[hxzhouh]

There is no update.