search

rocker-org / rocker

R configurations for Docker
https://rocker-project.org
GNU General Public License v2.0
1.45k stars 273 forks source link

OpenBLAS warning in r-base #498

Closed denrou closed 1 year ago

denrou commented 2 years ago

If I run a simple R command using the latest r-base image, here is what I get:

docker run --rm r-base:latest Rscript -e '1'
OpenBLAS WARNING - could not determine the L2 cache size on this system, assuming 256k
[1] 1

However, if I remove the libopenblas0-pthread beforehand, the warning disappear:

docker run --rm r-base:latest bash -c "apt-get -y update >/dev/null && apt-get remove -y libopenblas0-pthread >/dev/null && Rscript -e '1'"
[1] 1

I tried to get any information on this OpenBLAS warning but couldn't get very far. And this warning is very annoying to us because it appears quit a lot and pollute our logs even though it doesn't seem to cause any side effect on our results.

docker --version
Docker version 20.10.16, build aa7e414

docker images -q r-base
05eb7a258125

Running docker with colima

eddelbuettel commented 2 years ago

:shrug: -- Maintainer here.

I cannot reproduce here on a vanilla i7 desktop (or laptop):

$ docker run --rm r-base Rscript -e '1'
[1] 1
$

I rely on the proper behavior of the underlying Debian package -- for which I am also the maintainer.

Which relies on the proper behavior of the underlying Debian package, here OpenBLAS. I suggest you try to get in touch with the OpenBLAS team and see if that is a known issue, and you may want to give them a wee bit more detail on where you deploy. Is it a VM farm? Is it non-standard hardware?

Picking a LAPACK/BLAS that works for all is not trivial. We are generally happy with OpenBLAS and would be somewhat reluctant to move on. However, you can by starting from our Dockerfile and creating your own variant with reference BLAS or Atlas or even MKL.

denrou commented 2 years ago

Thanks for your comment, I will try to see with OpenBLAS team.

And for additional informations, I have the warning on my macOS laptop on an Intel(R) Core(TM) i9-9980HK CPU

eddelbuettel commented 2 years ago

Ah, that is helpful. It may be due to the VM layer not passing that on.

r2evans commented 1 year ago

I'm finding it in rocker/shiny-verse:4.2.2 but not in 4.2.1. While I don't know for sure where the true cause is (openblas or here), it does seem meaningful that some difference between either the image builds or the R version underneath is causing the issue.

# docker pull rocker/shiny-verse:4.2.2
4.2.2: Pulling from rocker/shiny-verse
Digest: sha256:0a749de2110c4b0ddb33f37e28f9a0db36d8d39e17051a37977e78616d37d2e7
Status: Image is up to date for rocker/shiny-verse:4.2.2
docker.io/rocker/shiny-verse:4.2.2
root@franz:/srv/docker/postgres# docker run -it --rm rocker/shiny-verse:4.2.2 R
OpenBLAS blas_thread_init: pthread_create failed for thread 1 of 40: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
OpenBLAS blas_thread_init: pthread_create failed for thread 2 of 40: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max
...
OpenBLAS blas_thread_init: pthread_create failed for thread 39 of 40: Operation not permitted
OpenBLAS blas_thread_init: RLIMIT_NPROC -1 current, -1 max

R version 4.2.2 (2022-10-31) -- "Innocent and Trusting"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

During startup - Warning messages:
1: package ‘utils’ in options("defaultPackages") was not found
2: package ‘stats’ in options("defaultPackages") was not found
> q('no')

When used in my gitlab ci, in a first step attempting to update dep packages, it errors/fails with

Error: .onLoad failed in loadNamespace() for 'utils', details:
  call: system(paste(which, shQuote(names[i])), intern = TRUE, ignore.stderr = TRUE)
  error: cannot popen '/usr/bin/which 'uname' 2>/dev/null', probable reason 'Cannot allocate memory'
Execution halted

When I do this with rocker/shiny-verse:4.2.1, I get no errors, and the CI passes without warning.

# docker pull rocker/shiny-verse:4.2.1
4.2.1: Pulling from rocker/shiny-verse
eaead16dc43b: Already exists
ba99059561c4: Pull complete
f3d27f46eecc: Pull complete
ca4910f4958b: Pull complete
c2d005b3d0b9: Pull complete
f4e0915bd63e: Pull complete
dffca296d880: Pull complete
Digest: sha256:befeae376697b68b0fbd92149abd80925deac27053f4ef5f0bdb506dd1421b84
Status: Downloaded newer image for rocker/shiny-verse:4.2.1
docker.io/rocker/shiny-verse:4.2.1
root@franz:/srv/docker/postgres# docker run -it --rm rocker/shiny-verse:4.2.1 R

R version 4.2.1 (2022-06-23) -- "Funny-Looking Kid"
Copyright (C) 2022 The R Foundation for Statistical Computing
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

>q('no')

In case it is related to the docker version,

# docker version
Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:47 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:58 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
cboettig commented 1 year ago

weird, can't reproduce. Can you try upgrading your docker version? I'm on

docker version
Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.17.3
 Git commit:        20.10.12-0ubuntu4
 Built:             Mon Mar  7 17:10:06 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.3
  Git commit:       20.10.12-0ubuntu4
  Built:            Mon Mar  7 15:57:50 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.9-0ubuntu3.1
  GitCommit:        
 runc:
  Version:          1.1.0-0ubuntu1.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:
r2evans commented 1 year ago

Can you try upgrading your docker version?

Yes, that's what it is, it resolves with 20.10.21. (Unfortunately for me the CI system is in desperate need of update (ubuntu 16), and 20.10.7 is the latest available.) Thanks!