runtime: automatically bump RLIMIT_NOFILE on Unix

[bradfitz]

I just read http://0pointer.net/blog/file-descriptor-limits.html which in a nutshell says:

• don't use select
• systemd sets the RLIMIT_NOFILE soft limit to 1024 for compatibility reasons, to not break select users
• systemd sets the RLIMIT_NOFILE hard limit 512K, for programs that want more (without escalation), but by raising their soft limit past 1024, they're implicitly acknowledging that select won't work.

I realize that since Go doesn't use select, the Go runtime could automatically do this fd soft limit bumping on Linux.

We do have a Select wrapper at https://pkg.go.dev/golang.org/x/sys/unix#Select, though, so perhaps we could do the same thing we did for #42347 in 18510ae88ffcb9c4a914805fde3e613539f9b6dc (https://go-review.googlesource.com/c/go/+/299671) and do the bumping conditionally based on whether the unix.Select func is in the binary. Or cgo too, I suppose.

I suspect many users are unaware of this 512K hard limit that's free to bump up to. I certainly was unaware. (I normally have to go in and manual tweak my systemd limits instead, usually in response to problems once I hit the limit...) I think fixing it automatically would help more users than it'd hurt. (I actually can't think how it'd hurt anybody?)

I don't think we need it as a backpressure mechanism. As the blog post mentions, memory limits are already that mechanism.

/cc @ianlancetaylor @aclements @rsc @randall77

[gopherbot]

Change https://go.dev/cl/539435 mentions this issue: syscall: copy rlimit.go's build constraint to rlimit_test.go

[gopherbot]

Change https://go.dev/cl/540615 mentions this issue: [release-branch.go1.21] syscall: copy rlimit.go's build constraint to rlimit_test.go