Closed AndreMikulec closed 3 years ago
Good idea - yes, adding argument logical
is definitely a start so you can use it as you proposed.
It could also be that the default should be logical = getOptionOrEnvVar("parallelly.availableCores.system.logical", TRUE)
. This would allow the user to set the value that is best suited on their system, e.g.
## Alt 1. In ~/.Rprofile:
options(parallelly.availableCores.system.logical = FALSE)
## Alt 2. In ~/.Renviron:
R_PARALLELLY_AVAILABLECORES_SYSTEM_LOGICAL=FALSE
FYI, you can use:
options(parallelly.availableCores.custom = function() parallel::detectCores(logical = FALSE))
This will cause availableCores()
to acknowledge that as well.
Finally, a question:
Q. I've just rolled out parallelly 1.22.0, which if you're on a Unix system, will cause availableCores()
to also query the nproc
tool. What does parallelly::availableCores(methods = "nproc")
give on your platform?
Henrik,
I am not on UNIX. I am on Windows . . .
> library(parallelly)
> parallelly::availableCores(methods = "nproc")
current
1
Also (ironically about the same topic), Some time ago, I found this discussion . . .
Re: [Rd] Get Logical processor count correctly whether NUMA is enabled or disabled
Tomas Kalibera <tomas.kalibera@@gmail.com>
Mon 9/3/2018, 8:07 AM
A summary for reference: the new detectCores() for Windows in R-devel.
Get Logical processor count correctly whether NUMA is enabled or disabled.
Mine (So, I do not have to program in C, C++, or Fortran to get this information):
C:\Users\ComputerUser>wmic cpu get numberofcores
NumberOfCores
4
From the link above, Tomas Kalibera, wrote some code (system call and regular expression). For my personal use (from Tomas Kalibera's code), I made this function. (You can have it. I am giving it away for free. It is the MIT license or whatever license you want). (I did not see "wmic cpu get numberofcores" in your code.)
detectTrueCores <- function() {
if (.Platform$OS.type == "windows") {
out <- system("wmic cpu get numberofcores", intern=TRUE)
Cores <- as.integer(sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE)))))
} else {
Cores <- parallel::detectCores(logical = FALSE)
}
Cores
}
So I run it and get this.
detectTrueCores()
[1] 4
Top of . . .
CPU-Z TXT Report
-------------------------------------------------------------------------
Binaries
-------------------------------------------------------------------------
CPU-Z version 1.74.0.x64
Processors
-------------------------------------------------------------------------
Number of processors 1
Number of threads 8
APICs
-------------------------------------------------------------------------
Processor 0
-- Core 0
-- Thread 0 0
-- Thread 1 1
-- Core 1
-- Thread 0 2
-- Thread 1 3
-- Core 2
-- Thread 0 4
-- Thread 1 5
-- Core 3
-- Thread 0 6
-- Thread 1 7
Timers
-------------------------------------------------------------------------
Perf timer 10.000 MHz
Sys timer 1.000 KHz
Processors Information
-------------------------------------------------------------------------
Processor 1 ID = 0
Number of cores 4 (max 8)
Number of threads 8 (max 16)
Name Intel Core i3/i5/i7
Codename Sandy Bridge
Specification Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz
Package
CPUID 6.A.7
Extended CPUID 6.2A
Core Stepping D2
Technology 32 nm
Tjmax 100.0 °C
Core Speed 2394.7 MHz
Stock frequency 2400 MHz
Instructions sets MMX, SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, EM64T, VT-x, AES, AVX
L1 Data cache 4 x 32 KBytes, 8-way set associative, 64-byte line size
L1 Instruction cache 4 x 32 KBytes, 8-way set associative, 64-byte line size
L2 cache 4 x 256 KBytes, 8-way set associative, 64-byte line size
L3 cache 6 MBytes, 12-way set associative, 64-byte line size
FID/VID Control yes
Turbo Mode not supported
Bottom
Software
-------------------------------------------------------------------------
Windows Version Microsoft Windows 10 (10.0) Professional 64-bit (Build 19041)
DirectX Version 12.0
devtools::session_info()
- Session info ---------------------------------------------------------------
setting value
version R version 4.0.3 (2020-10-10)
os Windows 10 x64
system x86_64, mingw32
ui RTerm
language (EN)
collate English_United States.1252
ctype English_United States.1252
tz America/Chicago
date 2020-12-14
- Packages -------------------------------------------------------------------
package * version date lib source
. . .
parallelly * 1.22.0-9000 2020-12-14 [1] local
. . .
FYI, I've added argument `logical' to the next release, which is in the 'develop' branch;
NEW FEATURES:
Thanks for the pointer to R-devel thread 'Get Logical processor count correctly whether NUMA is enabled or disabled', 2018-08-17. It looks like Tomas Kalibera implemented support for that on Windows in SVN r75198 (2018-08-27), which made it into R 3.6.0 (2019-04-26). As far as I understand that thread, that bug fix is to support Windows machines with more than 64 cores. Also, Tomas writes in a follow up in https://stat.ethz.ch/pipermail/r-devel/2018-September/076743.html:
summary for reference: the new
detectCores()
for Windows in R-devel seems to be working both for logical and physical cores on systems with64 logical processors (thanks to Arun for testing!). If the feature is important for anyone particularly using an older version of Windows and/or on a system with >64 logical processors, it would be nice if you could test and report any possible problem.
As I mentioned earlier, in older versions of R [edit: i.e. R (<= 3.6.0)] one can as a workaround use "wmic" to detect the number of processors on systems with >64 logical processors (with appropriate error handling added as needed):
# detectCores() out <- system("wmic cpu get numberoflogicalprocessors", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE))))
#detectCores(logical=FALSE) out <- system("wmic cpu get numberofcores", intern=TRUE) sum(as.numeric(gsub("([0-9]+).*", "\\1", grep("[0-9]+[ \t]*", out, value=TRUE))))
@AndreMikulec, in https://github.com/HenrikBengtsson/parallelly/issues/38#issuecomment-744612819 you mention your detectTrueCores()
based on the latter example of Tomas and suggests this could be added to parallelly. However, it's not clear to me why this is need given that detectCores(logical = FALSE)
should work on Windows in R (>= 3.6.0). Are you suggest, it could be added to parallelly for backward compatibility for users still running R (< 3.6.0) on Windows?
Henrik,
Here are some cases for advantages. Opinions may vary.
wmic.exe is (or can be) pushed down to end-users every month during the monthly Microsoft 2nd Tuesday patch night. Users get a fixed version every time, without waiting for an R developer to to write some C or Fortran code (or fix a problem).
Another advantage to using wmic.exe is that it is a command line executable. For a developer, the situation is a fast and easy API access to everything (else) wmic can do (without writing C or Fortran code): E.g., I just run wmic.exe through the R function system() (or system2()), then parse the output, and then I am done.
I will skip the wmic
fallback for now because (i) it requires robust tests to make sure it works as expected and won't break when/if wmic
is updated, (ii) I don't have a convenient way to validate it, and (iii) it AFAIU the current R implementation meets the needs.
Argument logical
is added to availableCores()
in the next release.
Henrik,
For me to do the correct work:
On my computer (Intel chip of December 2016) and doing parallel work, using four(4) cores is the correct answer.
I need to use just 4 parallel processes.
I am suggesting passing to
availableCores
the parameterlogical = FALSE
that is eventually passed:from (old) Number of cores available according to parallel::detectCores()
to (new) Number of cores available according to parallel::detectCores(locical)
I am suggesting from this (old)
to this (new)
I am suggesting from this (old)
to add the parameter definition (new)
So, to do my work (and not break the current code implementation) , I (personally) would call
Thanks, Andre Mikulec