MAXPROCS and CPU limits

[gouthamve]

I've read somewhere that Go's GC that normally uses 25% of the GOMAXPROCS threads. Now which means if we run containers on a 32-core machine, and have CPULimit of 2cores, Go might use up all CPU for GC itself!

And in the case compute intensive workloads, it makes sense to explore setting the number of GOMAXPROCS to the cpu limits which can be done automatically using: https://github.com/uber-go/automaxprocs

[bboreham]

I'd like a citation for that GC claim. What I've read is that the GC continuously paces itself to aim for a certain budget.

~Note that cgroups CPU limits aren't hard limits, but adjust the priorities when multiple tasks are competing for all available CPU. If you have free CPU a process can go above its cgroup limit. Therefore GOMAXPROCS is a much harder limit than the cgroup limit.~ see later

I find automaxprocs interesting, but I think I would start by simply setting GOMAXPROCS.

[cboggs]

To use the Kubernetes parlance, I think what you say @bboreham is true for requests, but not for limits. Limits are indeed a hard cap on number of periods allowed to be used per second per core as enforced by the Linux CFS. So in the case of a proper limit, if the GC does indeed use the host's core count to determine its budget, it could definitely max out what CPU quota its been granted and not get any work done.

That said, I don't pretend at being a GC expert (or even a novice, yet), so I can't speak to the accuracy of that bit. :-)

[bboreham]

Limits are indeed a hard cap on number of periods allowed to be used per second per core as enforced by the Linux CFS.

I'm going to have to call "citation required" on that one.

I'm looking at, for instance: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/cm/node_container_manager.go#L159 , where CpuShares is set. To get closer to what you said, it should be setting CpuQuota.

There is also a beta feature which uses CPU sets, which would perhaps obviate this discussion: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies

[bboreham]

Update: following some Slack discussion, I was pointed at https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/kuberuntime/kuberuntime_container_linux.go#L68 (the bool option cpuCFSQuota defaults on)

So you're right, it is a hard cap.

[embano1]

Not sure if this is a stale discussion here (issue still open). Regarding GC and CPU quota (limits) in containerized deployments: Monzo hit that issue really hard (with Go GC in an older version, might still persist because it's not so much about the GC, rather the OS/kernel), which led them and other Kubernetes users to disable CPU quotas globally (per node/Kubelet). All details and references in this comment: https://github.com/uber-go/automaxprocs/issues/12

Hope it's still useful.

[bboreham]

Thanks for the pointer. I would say this issue is still open because we don't have a lot of evidence either way. For the Cortex installation I look after, we don't set any CPU limits and adjust requests so that machines are not avoid 90% CPU utilisation, so I don't expect we hit the issue.

[bboreham]

Well now. I tried setting GOMAXPROCS on our distributors, and saw a 15% reduction in CPU usage. We run on 16-core machines, and the distributors typically take 2-4 cores each with compression on which generates a lot of garbage. Same load same latency.

So colour me stunned.

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[friedrichg]

Using GOMAXPROCS is the known method that still works till today in cortex. But it's certainly another hassle for operators to keep updating their GOMAXPROCS whenever cpu resources are changed, leave alone advanced automatic resource allocation...

Use automaxprocs as first suggested seems like the best way forward (Which falls back to GOMAXPROCS if configured).

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.

[friedrichg]

Turns out it is possible today to configure GOMAXPROCS and GOMEMLIMITS appropiately in kubernetes downward API https://github.com/cortexproject/cortex-jsonnet/commit/524c3b3401fbe6efdb34339772cff63c05eeee03

This is no longer needed