rocker/rstudio etc, but not r-base, leads to 36 cores at 100% when doing psych::fa amongst other things

[mark-andrews]

The following R code will usually be completed in less than one second and does not (as far as I can) need to use parallel::mclapply or other parallel tools.

data(bfi, package = 'psych')
M <- psych::fa(bfi[,1:15], nfactors = 5, fm="ml", rotate = 'none')

When I run it in rocker/rstudio:latest, all 36 cores of a Xeon workstation are processing at 100% for about 30 seconds. The sha256 of the rocker/rstudio:latest image is as follows.

rocker/rstudio@sha256:45b1e0a675aad9417ebb3bb63819c6e775efa77484fbabd0a228e797e7d6cc25

I can recreate this by

docker run -it --rm rocker/rstudio:latest bash 

In the bash shell in the container type R and then

install.packages("psych")
data(bfi, package = 'psych')
M <- psych::fa(bfi[,1:15], nfactors = 5, fm="ml", rotate = 'none')

When I try this in rocker/r-base:latest, everything is fine.

In my title, I say "amongst other things". I have not tracked down all the details yet, but all I can say is that I was running a RMarkdown script using psych::fa and lavaan::sem and it run at 36 cores for around 30 minutes before I killed it. On a regular, non-docker non-rocker but up to date R installation, it took around 30 seconds to render the document.

• Host operating system Arch Linux

> uname -a
Linux valhalla 5.7.2-arch1-1 #1 SMP PREEMPT Wed, 10 Jun 2020 20:36:24 +0000 x86_64 GNU/Linux

• Docker version

> docker --version
Docker version 19.03.11-ce, build 42e35e61f3

• docker logs provides no information

  
  > docker ps               
  CONTAINER ID        IMAGE                   COMMAND             CREATED              STATUS              PORTS                    NAMES
  0b24f5e85ab5        rocker/rstudio:latest   "bash"              About a minute ago   Up About a minute   8787/tcp                 blissful_mendeleev

docker logs 0b24f5e85ab5

* docker image id

docker images -q rocker/rstudio f4d862a97fdf

[eddelbuettel]

It's OpenBLAS which will grab all cores, which, when coupled with parallel code, leads to a cross-product -- see #412.

[mark-andrews]

@eddelbuettel Thank you. I have set environment variables

export OMP_NUM_THREADS=1
export OPENBLAS_NUM_THREADS=1

And that solves it.

For completeness, in one container, I tried just

export OMP_NUM_THREADS=1

And that seemed to be sufficient to solve the problem above.

In another container, I did

export OPENBLAS_NUM_THREADS=1

but that did not seem to be sufficient.

[cboettig]

@mark-andrews thanks for the report on this. @eddelbuettel @noamross do you think we should reconfigure the default openblas settings? This doesn't happen on the older openblas on the debian:stretch containers. Might be worth seeing if we swap libopenblas-openmp-dev for one of the other flavors of libopenblas that was now introduced?

https://github.com/rocker-org/rocker-versioned2/blob/master/scripts/install_R.sh#L22

[eddelbuettel]

It is my recollection that this has always been that way with OpenBLAS so we could throttle it, say with a max of four, in Renviron.site. That would be a CRAN-alike compromise to give the better library but prevent overscaling. Of course we'd have to tell power users (in non-forked/non-parallel use) how to comment it out. Or we could go back to a non-OpenBLAS.