Memory leak in bigmemory?

[YaohuiZeng]

Hi bigmemory authors,

I am resubmitting to CRAN my R package biglasso, which depends on bigmemory, and noticed that the Memtest with clang-UBSAN reveals some strange runtime errors related to BigMatrix object, though I didn't encounter any errors with R CMD check --as-cran, nor with win-builder.

I attached some output as below. Detailed output can be found here.

Could you clarify me how to fix this? In addition, I have my package depend on bigmemory (>= 4.0.0), should I change the dependency to the latest version, say >=4.5.0 ?

> cleanEx()
> nameEx("biglasso")
> ### * biglasso
> 
> flush(stderr()); flush(stdout())
> 
> ### Name: biglasso
> ### Title: Fit lasso penalized regression path for big data
> ### Aliases: biglasso
> 
> ### ** Examples
> 
> ## Linear regression
> data(colon)
> X <- colon$X
> y <- colon$y
> X.bm <- as.big.matrix(X)
> # lasso, default
> par(mfrow=c(1,2))
> fit.lasso <- biglasso(X.bm, y, family = 'gaussian')
gaussian_hsr.cpp:87:17: runtime error: member call on address 0x6120001bd540 which does not point to an object of type 'BigMatrix'
0x6120001bd540: note: object is of type 'SharedMemoryBigMatrix'
 36 00 80 0f  38 c1 ae 75 ad 7f 00 00  d0 07 00 00 00 00 00 00  3e 00 00 00 00 00 00 00  3e 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'SharedMemoryBigMatrix'
SUMMARY: AddressSanitizer: undefined-behavior gaussian_hsr.cpp:87:17 in 
/data/gannet/ripley/R/test-clang/bigmemory/include/bigmemory/BigMatrix.h:44:37: runtime error: member access within address 0x6120001bd540 which does not point to an object of type 'const BigMatrix'
0x6120001bd540: note: object is of type 'SharedMemoryBigMatrix'
 36 00 80 0f  38 c1 ae 75 ad 7f 00 00  d0 07 00 00 00 00 00 00  3e 00 00 00 00 00 00 00  3e 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'SharedMemoryBigMatrix'
SUMMARY: AddressSanitizer: undefined-behavior /data/gannet/ripley/R/test-clang/bigmemory/include/bigmemory/BigMatrix.h:44:37 in 
/data/gannet/ripley/R/test-clang/bigmemory/include/bigmemory/MatrixAccessor.hpp:37:39: runtime error: member call on address 0x6120001bd540 which does not point to an object of type 'BigMatrix'
0x6120001bd540: note: object is of type 'SharedMemoryBigMatrix'
 36 00 80 0f  38 c1 ae 75 ad 7f 00 00  d0 07 00 00 00 00 00 00  3e 00 00 00 00 00 00 00  3e 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'SharedMemoryBigMatrix'
SUMMARY: AddressSanitizer: undefined-behavior /data/gannet/ripley/R/test-clang/bigmemory/include/bigmemory/MatrixAccessor.hpp:37:39 in 
/data/gannet/ripley/R/test-clang/bigmemory/include/bigmemory/BigMatrix.h:41:28: runtime error: member access within address 0x6120001bd540 which does not point to an object of type 'BigMatrix'
0x6120001bd540: note: object is of type 'SharedMemoryBigMatrix'
 36 00 80 0f  38 c1 ae 75 ad 7f 00 00  d0 07 00 00 00 00 00 00  3e 00 00 00 00 00 00 00  3e 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'SharedMemoryBigMatrix'
SUMMARY: AddressSanitizer: undefined-behavior /data/gannet/ripley/R/test-clang/bigmemory/include/bigmemory/BigMatrix.h:41:28 in 
/data/gannet/ripley/R/test-clang/bigmemory/include/bigmemory/MatrixAccessor.hpp:38:23: runtime error: member call on address 0x6120001bd540 which does not point to an object of type 'BigMatrix'
0x6120001bd540: note: object is of type 'SharedMemoryBigMatrix'
 36 00 80 0f  38 c1 ae 75 ad 7f 00 00  d0 07 00 00 00 00 00 00  3e 00 00 00 00 00 00 00  3e 00 00 00
              ^~~~~~~~~~~~~~~~~~~~~~~
              vptr for 'SharedMemoryBigMatrix'

[kaneplusplus]

The SharedBigMatrix class inherits from BigMatrix - including the ncol function the error references. Is the object you created that is causing the error a pointer?

[YaohuiZeng]

@kaneplusplus I am not sure I understand, but basically I just use as.big.matrix to create the big.matrix object. Also, my package didn't touch any c++ source code in bigmemory, nor did it implement any new subclasses. It just accesses big.matrix object using MatrixAccessor in bigmemory.

[kaneplusplus]

Nevermind, I found the problem. It looks like clang UBSan was not compiled with RTTI (similar issue here). The sanitizer was not built to handle inheritance and when you call xMat->ncol() on line 87 of gaussian_hsr.cpp the compiler can't find ncol because it can't find methods of the parent class.

[YaohuiZeng]

Thanks a lot for debugging @kaneplusplus ! Then what is the best way to fix this? Should I just get ncol inside R and then pass it to c++ code?

Also, if the errors are caused by xMat->ncol(), presumably checking bigmemory with clang UBSan would also give that kind of error.

[kaneplusplus]

It would be best if they fixed their sanitizer ;-)

The previous link prescribes using the -fno-sanitize=vptr option but I think this will cause a warning in your build since you are fiddling with compiler flags. We should probably send an email to whoever is in charge of the builder. Do you know if it's Uwe?

[YaohuiZeng]

I know Uwe and Kurt help maintain R packages, but it's Kurt who brought up the memory leak issue. I just sent out an email to both and have you copied on the list. Hope they will take care of it. Thanks, @kaneplusplus .

[jaredhuling]

@YaohuiZeng Did you ever get this issue resolved? If so what was the fix?

[YaohuiZeng]

@jaredhuling not yet. Uwe pointed out that the issue seems caused by bugs in LLVM. I haven't got a chance to reach out the author of LLVM. Let me try that within these days.