ecordell
commented
2 years ago The kube scheduler can change the number of CPUs visible to a running container (without restarting it). I didn't find explicit docs on this, but this comment seems to confirm that behavior.
Strawman scenario that I believe can happen and would be bad with automaxprocs:
- SpiceDB container starts and detects 2 cpus, runs with GOMAXPROCS=2
- A "gauranteed" QoS class container gets scheduled on the same node, which steals 1 whole cpu away
- SpiceDB container is running with maxprocs=2, but actually only has 1 core available
- SpiceDB has tons of contention and performs badly
I think we could just take advantage of this ourselves: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/
By setting requests=limits=<a whole number of CPUs> on spicedb containers, we should get dedicated CPUs for spicedb to use. But then it's not super valuable to use automaxprocs (just set GOMAXPROCS env on the pod to equal the cpu request).
I'm not 100% sure how available cpu management features are in various cloud flavors of kube or if there are more caveats to be aware of.
There are tools like https://github.com/uber-go/automaxprocs that enable setting GOMAXPROCS dynamically based on the deployment environment.
From talking to Uber engineers, I get the impression that his library might not be maintained much in the future (cgroups v2) and the more promising solution is to wait for https://github.com/golang/go/issues/33803