search

MichaelChirico / r-bugs

A ⚠️read-only⚠️mirror of https://bugs.r-project.org/
20 stars 0 forks source link

[BUGZILLA #17210] Taking determinant of a matrix of NAs results in intermittent memory corruption #6385

Closed MichaelChirico closed 4 years ago

MichaelChirico commented 4 years ago

If I start R from the command line with --vanilla, then repeatedly execute the following line:

det(matrix(nrow=10,ncol=10))

... I eventually get a crash, with error:

Error in `/usr/lib/R/bin/exec/R': malloc(): memory corruption: 0x0000000002399400

The specific address in memory that is referenced varies. The number of times I need to execute the above line before getting a crash also varies.

This occurs with a wide range of matrix dimensions; 10x10 is not the only size that causes this issue.

output of R.version: platform x86_64-pc-linux-gnu
arch x86_64
os linux-gnu
system x86_64, linux-gnu
status
major 3
minor 3.2
year 2016
month 10
day 31
svn rev 71607
language R
version.string R version 3.3.2 (2016-10-31) nickname Sincere Pumpkin Patch

I am running Linux Mint 17.3; my CPU is an Intel Core i7-2620m (Sandy Bridge). My RAM is non-ECC. My R binary is from the CRAN ubuntu repository. (r-base version 3.3.2-1trusty0 )

I also see this issue running R within Emacs, as well as Rstudio.

This issue may be related to https://bugs.r-project.org/bugzilla/show_bug.cgi?id=16862 but I'm not certain.

METADATA

MichaelChirico commented 4 years ago

As the older report you mention, I strongly guess this is not reproducible with a version of R that uses R's own BLAS and Lapack.... which we Debian-based (=> Ubuntu, Mint) versions of R typically do not (or did not in the past.

As my version of R does use R's own BLAS + Lapack (instead of an accelerated one), I can run the following (or longer versions of this) without any problem :

for(i in 1:100000) { d <- det(matrix(, 10,10)); stopifnot(identical(d, NAreal)) }

What does La_version() give for you ? ... and what does 

system(paste("lsof -p", Sys.getpid(), "| grep -iE '(blas|lapack)'"))

I guess it will show external libraries, e.g., 'atlas' for the BLAS version... and that may be the culprit. We would tend to only consider this a bug in R, when the problem comes from our own (non-optimized but safe) versions of BLAS & Lapack.

Can you please post to R-SIG-Debian@<::CENSORED -- SEE ORIGINAL ON BUGZILLA::>-project.org and ask for confirmation (of your problem) there? Together with the same info from the libraries used?

Thank you for the report, and further feedback!

METADATA

MichaelChirico commented 4 years ago

Greetings Martin, thanks for your attention.

The output of La_version() for me is: [1] "3.5.0"

The output of: system(paste("lsof -p", Sys.getpid(), "| grep -iE '(blas|lapack)'")) is: R 3636 mem REG 252,0 39272 11930369 /usr/lib/R/modules/lapack.so R 3636 mem REG 252,0 5882272 11933488 /usr/lib/lapack/liblapack.so.3.0 R 3636 mem REG 252,0 23108112 11929607 /usr/lib/openblas-base/libopenblas.so.0

I will send a message to R-SIG-Debian next.

I had assumed that since I was using the CRAN DEB repositories, I would be using R's own versions of BLAS/Lapack, but if this is not the case, is there a way to force R to do so?

METADATA

MichaelChirico commented 4 years ago

This is now also a thread on r-sig-debian at https://stat.ethz.ch/pipermail/r-sig-debian/2017-January/thread.html#2640 with two more participants failing to replicate.

METADATA

MichaelChirico commented 4 years ago

I tried booting off a flash drive loaded with (the most recent) Linux Mint 18.1, enabled a CRAN Ubuntu mirror as a software source, installed the latest R from CRAN, and was unable to reproduce the issue.

La_version() gave me 3.6.0, and the output of

system(paste("lsof -p", Sys.getpid(), "| grep -iE '(blas|lapack)'"))

...also indicated that R was using the 3.6.0 versions of liblapack.so and libopenblas.so

Martin, I agree that it looks like it was an issue with older versions of the shared libraries as you suspected.

Before submitting this bug report, I had assumed that R uses its own versions of these libraries by default, so I've learned something new as well.

Thanks Martin also for the troubleshooting tips on how to determine what libraries are being used by R.

METADATA

MichaelChirico commented 4 years ago

(In reply to Ian Erickson from comment #4) ................

Before submitting this bug report, I had assumed that R uses its own
versions of these libraries by default, so I've learned something new as
well.

A very reasonable assumption. By default, if you compile R from the sources without any non-default configuration, this is indeed what happens. Also, e.g., the Redhat/Fedora/.. distribution does keep that default. Debian/Ubuntu/Mint/... chose a different, partly more flexible route: You can relatively easily change which versions of Lapack/BLAS your R should be using (*) and the default uses openblas which can be considerably faster than R's version... but comes with rare bugs, as we have seen here... and similarly for lapack.

and in the end, I think we are grateful for report, becaused it highlighted some issues.

*) However, you can only choose betwen system wide libraries, typically "system" (blas/lapack), ATLAS (..), and openblas but you can NOT choose to use R's own blas/lapack libraries, because they are not even built with that setup.

I think this is because R's own configure scripts and Makefiles currently don't make it easy to both build R's own BLAS + Lapack and allow "at run time" (without rebuilding) to switch BLAS/Lapack libraries (R's own with externals). .. I may be wrong, but if I'm correct, somebody could open a "Wishlist" R bug(zilla) item, for allowing the above.

METADATA

MichaelChirico commented 4 years ago

(In reply to Martin Maechler from comment #5)

*) However, you can only choose betwen system wide libraries, typically
"system" (blas/lapack),  ATLAS (..), and openblas 
but you can NOT choose to use R's own blas/lapack libraries, because they
are not 
even built with that setup.

I think this is because R's own configure scripts and Makefiles currently
don't make it easy to both build R's own
BLAS + Lapack _and_ allow "at run time"  (without rebuilding) to switch
BLAS/Lapack libraries (R's own with externals).
.. I may be wrong, but if I'm correct, somebody could open a "Wishlist" R
bug(zilla) item, for allowing the above.

It seems pretty clear from the corresponding reference manual, "R Installation and Administration" (short "R-admin") https://cloud.r-project.org/doc/manuals/r-devel/R-admin.html#Linear-algebra , that I am correct and one needs to (at least partly) rebuild R to allow switching from R's own BLAS/Lapack to using external ones.
People insterested in this topic, should read the R-admin section indicated above.

METADATA

MichaelChirico commented 4 years ago

(In reply to Martin Maechler from comment #6)

It seems pretty clear from the corresponding reference manual,
"R Installation and Administration" (short "R-admin")

https://cloud.r-project.org/doc/manuals/r-devel/R-admin.html#Linear-algebra ,
that I am correct and one needs to (at least partly) rebuild R to allow
switching from R's own BLAS/Lapack  to using external ones.  
People insterested in this topic, should read the R-admin section indicated
above.

See https://cloud.r-project.org/doc/manuals/r-devel/R-admin.html#Shared-BLAS .

METADATA