os: add a semaphore in front of the filesystem?

[bradfitz]

I'd like to investigate putting a semaphore in front of the pkg/os filesystem
operations, so massive numbers of goroutines attempting filesystem I/O don't create
threads each and make the operating system cry.

There are deadlock concerns, but perhaps we can do it per filesystem id, or learn the
limits, etc.

[dvyukov]

Comment 1:

What filesystem operations do you mean? There is an opened bug to make Read/Write async.

[bradfitz]

Comment 2:

All of them. Open, Close, Stat, LStat, Read, Write, Link, ...

[dvyukov]

Comment 3:

A common solution to prevent deadlocks is to throttle threads instead of bound them.
That is, you create threads instantly up to some limit; then you start throttling them a
bit; the more threads the more aggressive is the throttling.
I think that scalability will be important here. That is, if we cap fast cached
(in-memory) filesystem operations on a global chan-based semaphore, it can significantly
affect performance.

[rsc]

Comment 4:

Too late for 1.4. Marking Release-None until there's a plan.

Labels changed: added release-none, removed release-go1.4.

[bcmills]

Isn't it generally up to users to apply reasonable flow-control to limit expensive resources and/or bound the footprint of their program?

What makes "using a thread for a filesystem call" significantly different from "using a bunch of live memory in the Go heap" (which we are not - and, I would argue, should not be - proposing to guard with a semaphore or otherwise restrict artificially)?

[rsc]

The operating system no longer cries, as in Brad's earlier message, because we added a fixed limit on the number of threads in the program. Now the Go program crashes instead, which is better than taking down the OS.

I agree with Bryan that we probably don't want to do more here. We chose to crash on thread count instead of trying to do some kind of thread limit exactly for concerns about deadlocks, and those concerns would exist for any kind of file system semaphore as well.

Unless Brad objects, I think we should probably close this.

[bradfitz]

I think I still object. This was the first problem I hit with Go and I continue to hit it. I'd hope we can do better here.

The problem is that with "reasonable" flow control, reasonable often isn't known and implementations of interfaces can be selected at runtime with totally different properties. Camlistore uses a package to do a VFS where different paths are mounted with different file implementations (like google3 files). What's reasonable for /memfile/foo is very different from what's reasonable from /mnt/my-ext3-filesystem/.

[rsc]

OK, but a FS semaphore will cause deadlocks when someone does an FS operation that does block. That's not better.

[bradfitz]

I acknowledged the deadlock issue in the original issue text. @dvyukov discussed it too in "Comment 3".

Maybe the answer isn't a semaphore in the filesystem but instead a way to provide more information to programs creating work to tell them the current situation. We know how many goroutines there are, but can we cheaply answer how many goroutines are in system calls? Or how many threads there are? That might be enough.

[bcmills]

@bradfitz:

The problem is that with "reasonable" flow control, reasonable often isn't known

There are two or three directions you can go with that, but they're ~all application-side: there is no change you can make to the runtime that fixes the problem that "reasonable often isn't known". The problem is fundamentally one of balancing resource predictability against peak resource usage, and that depends very much on your execution environment: a reasonable "peak usage" on a small end-user desktop running many programs is fundamentally very different from a large server with strong resource isolation and/or only a few tasks.

and implementations of interfaces can be selected at runtime with totally different properties. Camlistore uses a package to do a VFS where different paths are mounted with different file implementations

That's an application design problem: do you want to maximize throughput, optimize for resource predictability, or both? If you want to optimize for predictability, then you need some kind of estimate of the usage that each implementation is potentially going to need at peak (not just what it is currently using), and then you can implement a global throttle where each implementation says "I'm going to need X resources" and the throttle blocks it until those resources are available. Presumably the /memfile implementation would use a different value for "X" than the ext3 one.

But that's all something you can only apply at the application layer: only you know what the peak resource consumption of each implementation is going to be like.

---

There is certainly a broad design space for "static analysis of peak resource consumption of a function call", but Go is not exactly designed to be amenable to that kind of analysis.

If you want to do the equivalent dynamic analysis, you basically only have three options: failure (the current behavior when we exhaust the thread limit), blocking (with the associated risks of deadlock and/or priority inversion), or forcible cancellation of pending work (e.g. causing an arbitrary goroutine to panic when we run out of resources, but most Go code is not written to be panic-safe and it's not clear how you choose which goroutine to kill without risking livelock anyway). None of those three options can be applied safely to arbitrary Go programs. All of those options would violate Go 1 compatibility: they could cause previously correct-but-resource-intensive programs to become "incorrect" programs.

The only viable solution for Go as it is today is for application programmers who care about resource footprints to carefully apply their own flow-control at the application layer.

[gopherbot]

CL https://golang.org/cl/36799 mentions this issue.

[gopherbot]

CL https://golang.org/cl/36800 mentions this issue.

[Lewiscowles1986]

Can this be closed now?

Now that we have GOMAXPROCS I don't understand why anything else is needed to limit from a user perspective.

Could be a very naive take.

[ianlancetaylor]

This doesn't actually have anything to do with GOMAXPROCS. A goroutine that is waiting for file I/O to complete is not running and does not count against GOMAXPROCS.

[Lewiscowles1986]

A goroutine that is waiting for file I/O to complete is not running and does not count against GOMAXPROCS.

wait what?

Any goroutine should count againt GOMAXPROCS; but if a tonne of them are idle, it is as little overhead as you should get.

This might be some learning for me, but does any IO pausing awaiting input put the go-routine to sleep, therefore not counting it against active threads?

[ianlancetaylor]

GOMAXPROCS is the maximum number of goroutines that can be executing concurrently. It does not count goroutines that are not executing, such as those that are waiting for I/O, or waiting for a channel operation, and so forth.

That said, this is not the place for this discussion. Please do not reply here. Please use a forum instead. See https://go.dev/wiki/Questions. Thanks.