availableCores returns wrong number of cores on Posit Workbench Session with Kubernetes Cluster

[CGlemser]

The Issue

Since version 1.39.0, parallelly::availableCores() no longer returns the correct number of available cores on a Kubernetes Cluster on the Posit Workbench (here: 32 instead of the 2 available to my session)

Reproducing example Running parallelly::availableCores() on Kubernetes Cluster on the Posit Workbench.

Expected behavior In version 1.38.0, running debugonce shows that cgroups.cpuquota was filled and correctly returned as the number of available cores, as seen here on a small 2-core session: .

Since the new version, when debugging, cgroups.cpuquota is now NA and system is returned, (which is equivalent to parallel::detectCores, and has always been wrong on the Kubernetes cluster). If I explicitly try to select the cgroups.cpuquota method, it returns 1 (I presume, because it is the minimal number of cores possible).

In case it is helpful, this returns the correct number of cores:

as.numeric(system(command = "cat /sys/fs/cgroup/cpu/cpu.shares", intern = TRUE)) / 1024

Session information

R version 4.3.2 (2023-10-31)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 22.04.4 LTS

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3 
LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so;  LAPACK version 3.10.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

time zone: cet
tzcode source: system (glibc)

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] parallelly_1.39.0

loaded via a namespace (and not attached):
 [1] compiler_4.3.2    fastmap_1.2.0     cli_3.6.3         parallel_4.3.2    htmltools_0.5.8.1 tools_4.3.2      
 [7] rstudioapi_0.16.0 yaml_2.3.10       rmarkdown_2.28    knitr_1.48        xfun_0.46         digest_0.6.36    
[13] rlang_1.1.4       evaluate_0.24.0 

[HenrikBengtsson]

Thanks for reporting. I clearly broke things when I tried to improve CGroups in v1.39.0. What does that:

$ cat /sys/fs/cgroup/cpu/cpu.shares

output as-is?

FYI, you can do:

parallelly::availableCores(which = "all")

to get everything that it picks up, i.e. no need to use debugging tools.

PS. Please cut'n'paste from the terminal instead of using screenshots, because of reasons, e.g. we've got users with screen readers (here and elsewhere), it's not possible to search for text in images, and it's not possible to cut'n'paste from them.