owncloud / ocis

:atom_symbol: ownCloud Infinite Scale Stack
https://doc.owncloud.com/ocis/next/
Apache License 2.0
1.4k stars 183 forks source link

kubernetes cpu limits are not respected by services using the reva runtime #8451

Open butonic opened 9 months ago

butonic commented 9 months ago

Running the loadtests I saw this in a group service log:

{"level":"info","service":"groups","time":"2024-02-14T12:51:10Z","line":"github.com/cs3org/reva/v2@v2.18.1-0.20240129131717-cff0a2eeb959/cmd/revad/runtime/runtime.go:178","message":"running on 8 cpus"} 

But the pod is configured to use

     Requests:                                                                                                                                                                                                              
       cpu:     15m                                                                                                                                                                                                         
       memory:  50Mi

ocis is not changing the GOMAXPROCS but reva does:

// adjustCPU parses string cpu and sets GOMAXPROCS
// according to its value. It accepts either
// a number (e.g. 3) or a percent (e.g. 50%).
// Default is to use all available cores.
func adjustCPU(cpu string) (int, error) {
    var numCPU int

    availCPU := runtime.NumCPU()

    if cpu != "" {
        if strings.HasSuffix(cpu, "%") {
            // Percent
            var percent float32
            pctStr := cpu[:len(cpu)-1]
            pctInt, err := strconv.Atoi(pctStr)
            if err != nil || pctInt < 1 || pctInt > 100 {
                return 0, fmt.Errorf("invalid CPU value: percentage must be between 1-100")
            }
            percent = float32(pctInt) / 100
            numCPU = int(float32(availCPU) * percent)
        } else {
            // Number
            num, err := strconv.Atoi(cpu)
            if err != nil || num < 1 {
                return 0, fmt.Errorf("invalid CPU value: provide a number or percent greater than 0")
            }
            numCPU = num
        }
    } else {
        numCPU = availCPU
    }

    if numCPU > availCPU || numCPU == 0 {
        numCPU = availCPU
    }

    runtime.GOMAXPROCS(numCPU)
    return numCPU, nil
}

I don't know how much of a prolem this is. Needs investigation. Maybe switch to https://github.com/uber-go/automaxprocs to correct the calculation of available CPUs. See https://nemre.medium.com/is-your-go-application-really-using-the-correct-number-of-cpu-cores-20915d2b6ccb

Further reading: https://engineering.grab.com/performance-bottlenecks-go-apps

butonic commented 9 months ago

cc @wkloucek