HenrikBengtsson / parallelly

R package: parallelly - Enhancing the 'parallel' Package
https://parallelly.futureverse.org
130 stars 7 forks source link

Please pass "logical" to the function availableCores() method = "system" and parallel::detectCores(). #38

Closed AndreMikulec closed 3 years ago

AndreMikulec commented 3 years ago

Henrik,

For me to do the correct work:

On my computer (Intel chip of December 2016) and doing parallel work, using four(4) cores is the correct answer.

> parallel::detectCores() # wrong answer
[1] 8
> parallel::detectCores(logical = T) # wrong answer
[1] 8
> parallel::detectCores(logical = F) # right answer
[1] 4

I need to use just 4 parallel processes.

I am suggesting passing to availableCores the parameter logical = FALSE that is eventually passed:

from (old) Number of cores available according to parallel::detectCores()

n <- detectCores()

to (new) Number of cores available according to parallel::detectCores(locical)

n <- detectCores(logical)

I am suggesting from this (old)

availableCores <- function(constraints = NULL, methods = getOption2("future.availableCores.methods", c("system", "nproc", "mc.cores", "_R_CHECK_LIMIT_CORES_", "PBS", "SGE", "Slurm", "LSF", "fallback", "custom")), na.rm = TRUE, default = c(current = 1L), which = c("min", "max", "all")
  )

to this (new)

availableCores <- function(constraints = NULL, methods = getOption2("future.availableCores.methods", c("system", "nproc", "mc.cores", "_R_CHECK_LIMIT_CORES_", "PBS", "SGE", "Slurm", "LSF", "fallback", "custom")), na.rm = TRUE, default = c(current = 1L), which = c("min", "max", "all"),
  logical = TRUE)

I am suggesting from this (old)

# no parameter

to add the parameter definition (new)

#' @param logical if methods = "system" and ignored otherwise, the parameter "logical" is passed to parallel::detectCores.  If possible, use the number of physical CPUs/cores (if FALSE) or logical CPUs (if TRUE). Currently this is honoured only on macOS, Solaris and Windows.  Default is FALSE.

So, to do my work (and not break the current code implementation) , I (personally) would call

availableCores(methods = "system", logical = FALSE)

Thanks, Andre Mikulec

HenrikBengtsson commented 3 years ago

Good idea - yes, adding argument logical is definitely a start so you can use it as you proposed.

It could also be that the default should be logical = getOptionOrEnvVar("parallelly.availableCores.system.logical", TRUE). This would allow the user to set the value that is best suited on their system, e.g.

## Alt 1. In ~/.Rprofile:
options(parallelly.availableCores.system.logical = FALSE)

## Alt 2. In ~/.Renviron:
R_PARALLELLY_AVAILABLECORES_SYSTEM_LOGICAL=FALSE

FYI, you can use:

options(parallelly.availableCores.custom = function() parallel::detectCores(logical = FALSE))

This will cause availableCores() to acknowledge that as well.

Finally, a question:

Q. I've just rolled out parallelly 1.22.0, which if you're on a Unix system, will cause availableCores() to also query the nproc tool. What does parallelly::availableCores(methods = "nproc") give on your platform?

AndreMikulec commented 3 years ago

Henrik,

I am not on UNIX. I am on Windows . . .

> library(parallelly)
> parallelly::availableCores(methods = "nproc")
current
      1

Also (ironically about the same topic), Some time ago, I found this discussion . . .

Re: [Rd] Get Logical processor count correctly whether NUMA is enabled or disabled
Tomas Kalibera <tomas.kalibera@@gmail.com>
Mon 9/3/2018, 8:07 AM
A summary for reference: the new detectCores() for Windows in R-devel.
Get Logical processor count correctly whether NUMA is enabled or disabled.

https://r.789695.n4.nabble.com/Get-Logical-processor-count-correctly-whether-NUMA-is-enabled-or-disabled-td4751774.html

Mine (So, I do not have to program in C, C++, or Fortran to get this information):

C:\Users\ComputerUser>wmic cpu get numberofcores
NumberOfCores
4

From the link above, Tomas Kalibera, wrote some code (system call and regular expression). For my personal use (from Tomas Kalibera's code), I made this function. (You can have it. I am giving it away for free. It is the MIT license or whatever license you want). (I did not see "wmic cpu get numberofcores" in your code.)

detectTrueCores <- function() {

  if (.Platform$OS.type == "windows") {
    out   <- system("wmic cpu get numberofcores", intern=TRUE)
    Cores <- as.integer(sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))))
  } else {
    Cores <- parallel::detectCores(logical = FALSE)
  }
  Cores
}

So I run it and get this.

detectTrueCores()
[1] 4

Top of . . .

CPU-Z TXT Report
-------------------------------------------------------------------------

Binaries
-------------------------------------------------------------------------

CPU-Z version           1.74.0.x64

Processors
-------------------------------------------------------------------------

Number of processors        1
Number of threads       8

APICs
-------------------------------------------------------------------------

Processor 0 
    -- Core 0   
        -- Thread 0 0
        -- Thread 1 1
    -- Core 1   
        -- Thread 0 2
        -- Thread 1 3
    -- Core 2   
        -- Thread 0 4
        -- Thread 1 5
    -- Core 3   
        -- Thread 0 6
        -- Thread 1 7

Timers
-------------------------------------------------------------------------

    Perf timer      10.000 MHz
    Sys timer       1.000 KHz
Processors Information
-------------------------------------------------------------------------

Processor 1         ID = 0
    Number of cores     4 (max 8)
    Number of threads   8 (max 16)
    Name            Intel Core i3/i5/i7
    Codename        Sandy Bridge
    Specification       Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz
    Package         
    CPUID           6.A.7
    Extended CPUID      6.2A
    Core Stepping       D2
    Technology      32 nm
    Tjmax           100.0 °C
    Core Speed      2394.7 MHz
    Stock frequency     2400 MHz
    Instructions sets   MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, EM64T, VT-x, AES, AVX
    L1 Data cache       4 x 32 KBytes, 8-way set associative, 64-byte line size
    L1 Instruction cache    4 x 32 KBytes, 8-way set associative, 64-byte line size
    L2 cache        4 x 256 KBytes, 8-way set associative, 64-byte line size
    L3 cache        6 MBytes, 12-way set associative, 64-byte line size
    FID/VID Control     yes

    Turbo Mode      not supported

Bottom

Software
-------------------------------------------------------------------------

Windows Version         Microsoft Windows 10 (10.0) Professional 64-bit   (Build 19041) 
DirectX Version         12.0
 devtools::session_info()
- Session info ---------------------------------------------------------------
 setting  value
 version  R version 4.0.3 (2020-10-10)
 os       Windows 10 x64
 system   x86_64, mingw32
 ui       RTerm
 language (EN)
 collate  English_United States.1252
 ctype    English_United States.1252
 tz       America/Chicago
 date     2020-12-14

- Packages -------------------------------------------------------------------
 package     * version     date       lib source
. . . 
 parallelly  * 1.22.0-9000 2020-12-14 [1] local
. . .
HenrikBengtsson commented 3 years ago

FYI, I've added argument `logical' to the next release, which is in the 'develop' branch;

NEW FEATURES:

HenrikBengtsson commented 3 years ago

Thanks for the pointer to R-devel thread 'Get Logical processor count correctly whether NUMA is enabled or disabled', 2018-08-17. It looks like Tomas Kalibera implemented support for that on Windows in SVN r75198 (2018-08-27), which made it into R 3.6.0 (2019-04-26). As far as I understand that thread, that bug fix is to support Windows machines with more than 64 cores. Also, Tomas writes in a follow up in https://stat.ethz.ch/pipermail/r-devel/2018-September/076743.html:

summary for reference: the new detectCores() for Windows in R-devel seems to be working both for logical and physical cores on systems with

64 logical processors (thanks to Arun for testing!). If the feature is important for anyone particularly using an older version of Windows and/or on a system with >64 logical processors, it would be nice if you could test and report any possible problem.

As I mentioned earlier, in older versions of R [edit: i.e. R (<= 3.6.0)] one can as a workaround use "wmic" to detect the number of processors on systems with >64 logical processors (with appropriate error handling added as needed):

# detectCores()
out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE)
sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, 
value=TRUE))))
#detectCores(logical=FALSE)
out <- system("wmic cpu get numberofcores", intern=TRUE)
sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, 
value=TRUE))))

@AndreMikulec, in https://github.com/HenrikBengtsson/parallelly/issues/38#issuecomment-744612819 you mention your detectTrueCores() based on the latter example of Tomas and suggests this could be added to parallelly. However, it's not clear to me why this is need given that detectCores(logical = FALSE) should work on Windows in R (>= 3.6.0). Are you suggest, it could be added to parallelly for backward compatibility for users still running R (< 3.6.0) on Windows?

AndreMikulec commented 3 years ago

Henrik,

Here are some cases for advantages. Opinions may vary.

wmic.exe is (or can be) pushed down to end-users every month during the monthly Microsoft 2nd Tuesday patch night. Users get a fixed version every time, without waiting for an R developer to to write some C or Fortran code (or fix a problem).

Another advantage to using wmic.exe is that it is a command line executable. For a developer, the situation is a fast and easy API access to everything (else) wmic can do (without writing C or Fortran code): E.g., I just run wmic.exe through the R function system() (or system2()), then parse the output, and then I am done.

HenrikBengtsson commented 3 years ago

I will skip the wmic fallback for now because (i) it requires robust tests to make sure it works as expected and won't break when/if wmic is updated, (ii) I don't have a convenient way to validate it, and (iii) it AFAIU the current R implementation meets the needs.

Argument logical is added to availableCores() in the next release.