KyleSanderson
commented
4 hours ago (the magic number of I/O might be 2 less than thread count on Write, but 1 on Read, and 0 on hard core number crunching)
Open KyleSanderson opened 4 hours ago
KyleSanderson
commented
4 hours ago (the magic number of I/O might be 2 less than thread count on Write, but 1 on Read, and 0 on hard core number crunching)
When I first started learning go, the question came up, (naturally with others, I would expect) do I just blindly call go for every invocation (100,000 times) and use a waitgroup, do I spin a magic number like 7 on a 8 thread processor, or what do I do (channels! who calls close on send, oh, one per worker? oh... channels...).
There's since been hints like GOMAXPROCS and similar added, with other more neat libraries like reading cgroup limits and setting that environment variable. This is great stuff, I want to use it for some large jobs, but I don't know what to put in for the number, and from the examples they appear quite haphazard.
If GOMAXPROCS is set I'd prefer there be some function I can call in rill to guess the magic number of workers (maybe a param can be the type of workload I think it is), and if it isn't set Rill estimate what would be a correct amount of concurrency. Obviously there's no right answer, but 2 on a 4 thread box is very different from 2 on a 64 thread box, and this is the crux of the issue.
I'd normally keep it tight and vague, but tried to explain the who/what/where/when/why of why I'd want something like this, and a gap I think can be filled easily. Uber has a library that can get this information from cgroups and similar, maybe even just repackaging that until there's a reason to fork is a way forward.