IDEA: Query the "current" CPU load

[HenrikBengtsson]

(Adopted from an old 2019-09-12 note of mine)

The below is a proof of concept on how to query the "current" CPU load on a Linux machine:

cpu_load <- function() {
  if (file.exists("/proc/loadavg")) {
    res <- readLines("/proc/loadavg", n=1L)
    res <- strsplit(res, split=" ", fixed=TRUE)[[1]][1:3]
    res <- as.numeric(res)
  } else {
    res <- rep(NA_real_, times = 3L)
  }
  names(res) <- c("1min", "5min", "15min")
  res
}

Example:

> parallel::detectCores()
[1] 8
> cpu_load()
 1min  5min 15min 
 0.59  0.55  0.68

The value of cpu_load() can serve as a guide on how much of the CPU resources are "free", e.g. parallel::detectCores() - cpu_load().

This could potentially be incorporated into availableCores() as an option, e.g. ncores <- availableCores(methods = "5min-load").

[HenrikBengtsson]

More from the same notes:

cpu_load <- function() {
  if (file.exists("/proc/loadavg")) {
    res <- readLines("/proc/loadavg", n=1L)
    res <- strsplit(res, split=" ", fixed=TRUE)[[1]][1:3]
    res <- as.numeric(res)
  } else {
    res <- rep(NA_real_, times = 3L)
  }
  names(res) <- c("1min", "5min", "15min")
  res
}

free_cores <- function(memory = c("5min", "15min", "1min"), fraction = 0.90, default = 1L) {
  stop_if_not <- parallelly:::stop_if_not
  memory <- match.arg(memory, choices = c("5min", "15min", "1min"))
  stop_if_not(!is.na(fraction), fraction > 0, fraction <= 1)
  default <- as.integer(default)
  stop_if_not(!is.na(default), default >= 1L)
  load <- cpu_load()[memory]
  if (is.na(load)) {
    oopts <- options(future.availableCores.custom = NULL)
    on.exit(options(oopts))
    return(parallelly::availableCores())
  }
  ncores <- availableCores(methods = c("system", "fallback"))
  if (ncores == 1L) return(1L)
  free_cores <- max(1L, floor(fraction * (ncores - load)))
  attr(free_cores, "load") <- load
  attr(free_cores, "max_cores") <- ncores
  attr(free_cores, "fraction") <- fraction
  free_cores
}

Example:

> library(parallelly)
> availableCores()
system 
     8
> options(future.availableCores.custom = free_cores)
> availableCores()
custom 
     6

[HenrikBengtsson]

Now this is implemented in the 'develop' branch:

• Add cpuLoad() for querying the "average" system load on Unix-like systems.
• Add freeCores() for estimating the average number of unused cores based on the average system load as given by cpuLoad().