gcc-ASAN error with OpenMP and column access

[helske]

The following code causes an error when compiled with gcc-ASAN configuration which tries to emulate the CRAN settings.

#include "RcppArmadillo.h"

// [[Rcpp::depends(RcppArmadillo)]]
// [[Rcpp::plugins(openmp)]]
// [[Rcpp::export]]
void foo() {
#pragma omp parallel for num_threads(1)
  for (int k = 0; k < 10; k++) {
    arma::mat y(2,2,arma::fill::ones);
    y.col(0);
  }
}`

R-devel (r73733) was compiled with gcc 7.2 using settings from BDR, and using the corresponding Makevars.

This is the error I am seeing:

> Rcpp::sourceCpp("asanbug/foo.cpp")
attributes.cpp:167:11: runtime error: load of value 112, which is not a valid value for type 'bool'
attributes.cpp:167:11: runtime error: load of value 15, which is not a valid value for type 'bool'
> foo()
=================================================================
==20848==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffe30a78980 at pc 0x7f476d173ad1 bp 0x7ffe30a787f0 sp 0x7ffe30a787e0
WRITE of size 8 at 0x7ffe30a78980 thread T0
    #0 0x7f476d173ad0 in foo() [clone ._omp_fn.0] /home/jouni/R/x86_64-pc-linux-gnu-library/3.5/RcppArmadillo/include/armadillo_bits/subview_meat.hpp:37
    #1 0x7f4777d16d7e in GOMP_parallel (/usr/lib/x86_64-linux-gnu/libgomp.so.1+0xdd7e)
    #2 0x7f476d1749a2 in sourceCpp_1_foo /tmp/RtmpVTgH5V/sourceCpp-x86_64-pc-linux-gnu-0.12.13/sourcecpp_517036776ded/foo.cpp:21
    #3 0x564f2b636fc0 in do_dotcall /home/jouni/R-devel/src/main/dotcode.c:1252
    #4 0x564f2b706f04 in Rf_eval /home/jouni/R-devel/src/main/eval.c:728
    #5 0x564f2b712845 in Rf_evalList /home/jouni/R-devel/src/main/eval.c:2698
    #6 0x564f2b706b08 in Rf_eval /home/jouni/R-devel/src/main/eval.c:719
    #7 0x564f2b70afc4 in R_execClosure /home/jouni/R-devel/src/main/eval.c:1617
    #8 0x564f2b7061ea in Rf_eval /home/jouni/R-devel/src/main/eval.c:747
    #9 0x564f2b77b21e in Rf_ReplIteration /home/jouni/R-devel/src/main/main.c:258
    #10 0x564f2b77bc30 in R_ReplConsole /home/jouni/R-devel/src/main/main.c:308
    #11 0x564f2b77d421 in run_Rmainloop /home/jouni/R-devel/src/main/main.c:1082
    #12 0x564f2b4daa1d in main /home/jouni/R-devel/src/main/Rmain.c:29
    #13 0x7f47777443f0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x203f0)
    #14 0x564f2b4dd139 in _start (/usr/local/lib/R/bin/exec/R+0x131139)

Address 0x7ffe30a78980 is located in stack of thread T0 at offset 288 in frame
    #0 0x7f476d172ebf in foo() [clone ._omp_fn.0] /tmp/RtmpVTgH5V/sourceCpp-x86_64-pc-linux-gnu-0.12.13/sourcecpp_517036776ded/foo.cpp:7

  This frame has 6 object(s):
    [32, 40) 'dest'
    [96, 104) 'error_message'
    [160, 168) '<unknown>'
    [224, 232) '<unknown>'
    [288, 336) '<unknown>' <== Memory access at offset 288 is inside this variable
    [384, 544) 'y'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-scope /home/jouni/R/x86_64-pc-linux-gnu-library/3.5/RcppArmadillo/include/armadillo_bits/subview_meat.hpp:37 in foo() [clone ._omp_fn.0]
Shadow bytes around the buggy address:
  0x1000461470e0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000461470f0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100046147100: 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1 f1 f1
  0x100046147110: 00 f2 f2 f2 f2 f2 f2 f2 f8 f2 f2 f2 f2 f2 f2 f2
  0x100046147120: 00 f2 f2 f2 f2 f2 f2 f2 00 f2 f2 f2 f2 f2 f2 f2
=>0x100046147130:[f8]f8 f8 f8 f8 f8 f2 f2 f2 f2 f2 f2 00 00 00 00
  0x100046147140: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x100046147150: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
  0x100046147160: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 01 f2 f2 f2
  0x100046147170: f2 f2 f2 f2 01 f2 f2 f2 f2 f2 f2 f2 00 00 00 00
  0x100046147180: f3 f3 f3 f3 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==20848==ABORTING

> sessionInfo()
R Under development (unstable) (2017-11-15 r73733)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 17.04

Matrix products: default
BLAS: /usr/local/lib/R/lib/libRblas.so
LAPACK: /usr/local/lib/R/lib/libRlapack.so

locale:
 [1] LC_CTYPE=en_GB.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_GB.UTF-8        LC_COLLATE=en_GB.UTF-8    
 [5] LC_MONETARY=en_GB.UTF-8    LC_MESSAGES=en_GB.UTF-8   
 [7] LC_PAPER=en_GB.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_GB.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] Rcpp_0.12.13

loaded via a namespace (and not attached):
[1] compiler_3.5.0

(similar issue is seen in CRAN as well in case of seqHMM: https://www.stats.ox.ac.uk/pub/bdr/memtests/gcc-ASAN/seqHMM/00check.log

[helske]

I tried both with the CRAN versions of Rcpp and RcppArmadillo as well as Github-versions.

Removing the openmp pragma "fixes" the issue.

[eddelbuettel]

Well... that would make it a 'Conrad et al' rather than 'Dirk et al' issue, no? Paging @conradsnicta ...

[helske]

Yes you are of course right. Same issue is also seen with row and submat methods, but not with direct element access as well as with cubes. Also with k<1 there is no error.

[eddelbuettel]

It's a good minimal example by the way. Only pain the darn exact calibration of the compiler for ASAN/UBSAN/whateverSAN switches available. I once built a Docker container for this which an enterprising soul could update...

[helske]

Yes I actually first tried to reproduce the issue with r-hub and then directly with your rocker/r-devel-san, but noticed that openMP is disabled in those R configurations. And enabling that caused all kinds of weird compilation issues so I had to do everything locally...

[eddelbuettel]

@conradsnicta : the example by @helske was "pure". See the code:

#include "RcppArmadillo.h"

// [[Rcpp::depends(RcppArmadillo)]]
// [[Rcpp::plugins(openmp)]]
// [[Rcpp::export]]
void foo() {
#pragma omp parallel for num_threads(1)
  for (int k = 0; k < 10; k++) {
    arma::mat y(2,2,arma::fill::ones);
    y.col(0);
  }
}`

That is a void function with zero arguments. Jouni is "just" using the R build environment, and then the SAN-instrumented compiler set up to match one of the R test beds.

Jouni, can you maybe show the compiler invocation and whatever else may be pertinent?

[eddelbuettel]

@conradsnicta I think we're actually saying the same thing. We want a minimally reproducible example, so by compiling just those lines in a main() directly with the same g++ and its *SAN setting we should see this.

@helske Could you do that?

[helske]

Yes I can do that. I actually miswrote earlier, I don't have g++ 7.2 but g++ 7.0. I can try to upgrade if you feel that is necessary.

#include <armadillo>
#include <omp.h>
int main() {
#pragma omp parallel for num_threads(1)
  for (int k = 0; k < 10; k++) {
    arma::mat y(2,2,arma::fill::ones);
    y.col(0);
  }
  std::cout<<"success."<<std::endl;
  return 0;
}

 g++ asan.cpp -o asan -fsanitize=address,undefined,bounds-strict -fno-omit-frame-pointer -std=gnu++11 -fopenmp -larmadillo -v
Using built-in specs.
COLLECT_GCC=g++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 7-20170407-0ubuntu2' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.0.1 20170407 (experimental) [trunk revision 246759] (Ubuntu 7-20170407-0ubuntu2) 
COLLECT_GCC_OPTIONS='-o' 'asan' '-fsanitize=address,undefined,bounds-strict' '-fno-omit-frame-pointer' '-std=gnu++11' '-fopenmp' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-pthread'
 /usr/lib/gcc/x86_64-linux-gnu/7/cc1plus -quiet -v -imultiarch x86_64-linux-gnu -D_GNU_SOURCE -D_REENTRANT asan.cpp -quiet -dumpbase asan.cpp -mtune=generic -march=x86-64 -auxbase asan -std=gnu++11 -version -fsanitize=address,undefined,bounds-strict -fno-omit-frame-pointer -fopenmp -fstack-protector-strong -Wformat -Wformat-security -o /tmp/ccP68a7c.s
GNU C++11 (Ubuntu 7-20170407-0ubuntu2) version 7.0.1 20170407 (experimental) [trunk revision 246759] (x86_64-linux-gnu)
    compiled by GNU C version 7.0.1 20170407 (experimental) [trunk revision 246759], GMP version 6.1.1, MPFR version 3.1.5, MPC version 1.0.3, isl version isl-0.18-GMP

warning: GMP header version 6.1.1 differs from library version 6.1.2.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
ignoring duplicate directory "/usr/include/x86_64-linux-gnu/c++/7"
ignoring nonexistent directory "/usr/local/include/x86_64-linux-gnu"
ignoring nonexistent directory "/usr/lib/gcc/x86_64-linux-gnu/7/../../../../x86_64-linux-gnu/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/include/c++/7
 /usr/include/x86_64-linux-gnu/c++/7
 /usr/include/c++/7/backward
 /usr/lib/gcc/x86_64-linux-gnu/7/include
 /usr/local/include
 /usr/lib/gcc/x86_64-linux-gnu/7/include-fixed
 /usr/include/x86_64-linux-gnu
 /usr/include
End of search list.
GNU C++11 (Ubuntu 7-20170407-0ubuntu2) version 7.0.1 20170407 (experimental) [trunk revision 246759] (x86_64-linux-gnu)
    compiled by GNU C version 7.0.1 20170407 (experimental) [trunk revision 246759], GMP version 6.1.1, MPFR version 3.1.5, MPC version 1.0.3, isl version isl-0.18-GMP

warning: GMP header version 6.1.1 differs from library version 6.1.2.
GGC heuristics: --param ggc-min-expand=100 --param ggc-min-heapsize=131072
Compiler executable checksum: a2492fcd3282591a39f9f5a6e1b7434c
COLLECT_GCC_OPTIONS='-o' 'asan' '-fsanitize=address,undefined,bounds-strict' '-fno-omit-frame-pointer' '-std=gnu++11' '-fopenmp' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-pthread'
 as -v --64 -o /tmp/cccbgTyj.o /tmp/ccP68a7c.s
GNU assembler version 2.28 (x86_64-linux-gnu) using BFD version (GNU Binutils for Ubuntu) 2.28
COMPILER_PATH=/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/
LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/7/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/7/../../../:/lib/:/usr/lib/
Reading specs from /usr/lib/gcc/x86_64-linux-gnu/7/libgomp.spec
COLLECT_GCC_OPTIONS='-o' 'asan' '-fsanitize=address,undefined,bounds-strict' '-fno-omit-frame-pointer' '-std=gnu++11' '-fopenmp' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-pthread'
 /usr/lib/gcc/x86_64-linux-gnu/7/collect2 -plugin /usr/lib/gcc/x86_64-linux-gnu/7/liblto_plugin.so -plugin-opt=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper -plugin-opt=-fresolution=/tmp/ccc41g5p.res -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc -plugin-opt=-pass-through=-lpthread -plugin-opt=-pass-through=-lc -plugin-opt=-pass-through=-lgcc_s -plugin-opt=-pass-through=-lgcc --sysroot=/ --build-id --eh-frame-hdr -m elf_x86_64 --hash-style=gnu --as-needed -dynamic-linker /lib64/ld-linux-x86-64.so.2 -pie -z now -z relro -o asan /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/Scrt1.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crti.o /usr/lib/gcc/x86_64-linux-gnu/7/crtbeginS.o /usr/lib/gcc/x86_64-linux-gnu/7/crtoffloadbegin.o -L/usr/lib/gcc/x86_64-linux-gnu/7 -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu -L/usr/lib/gcc/x86_64-linux-gnu/7/../../../../lib -L/lib/x86_64-linux-gnu -L/lib/../lib -L/usr/lib/x86_64-linux-gnu -L/usr/lib/../lib -L/usr/lib/gcc/x86_64-linux-gnu/7/../../.. /usr/lib/gcc/x86_64-linux-gnu/7/libasan_preinit.o --push-state --no-as-needed -lasan --pop-state /tmp/cccbgTyj.o -larmadillo -lstdc++ -lm -lgomp -lubsan -lgcc_s -lgcc -lpthread -lc -lgcc_s -lgcc /usr/lib/gcc/x86_64-linux-gnu/7/crtendS.o /usr/lib/gcc/x86_64-linux-gnu/7/../../../x86_64-linux-gnu/crtn.o /usr/lib/gcc/x86_64-linux-gnu/7/crtoffloadend.o
COLLECT_GCC_OPTIONS='-o' 'asan' '-fsanitize=address,undefined,bounds-strict' '-fno-omit-frame-pointer' '-std=gnu++11' '-fopenmp' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64' '-pthread'

And the output:

 ./asan
=================================================================
==29398==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffcb2e37190 at pc 0x563af443fa54 bp 0x7ffcb2e370b0 sp 0x7ffcb2e370a0
WRITE of size 8 at 0x7ffcb2e37190 thread T0
    #0 0x563af443fa53 in arma::subview_col<double>::subview_col(arma::Mat<double> const&, unsigned long long) (/home/jouni/repos/asan+0x10a53)
    #1 0x563af443dcd7 in main._omp_fn.0 (/home/jouni/repos/asan+0xecd7)
    #2 0x7f3a167b9d7e in GOMP_parallel (/usr/lib/x86_64-linux-gnu/libgomp.so.1+0xdd7e)
    #3 0x563af443bce0 in main (/home/jouni/repos/asan+0xcce0)
    #4 0x7f3a154cf3f0 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x203f0)
    #5 0x563af443a739 in _start (/home/jouni/repos/asan+0xb739)

Address 0x7ffcb2e37190 is located in stack of thread T0 at offset 32 in frame
    #0 0x563af443daf7 in main._omp_fn.0 (/home/jouni/repos/asan+0xeaf7)

  This frame has 2 object(s):
    [32, 96) '<unknown>' <== Memory access at offset 32 is inside this variable
    [128, 304) 'y'
HINT: this may be a false positive if your program uses some custom stack unwind mechanism or swapcontext
      (longjmp and C++ exceptions *are* supported)
SUMMARY: AddressSanitizer: stack-use-after-scope (/home/jouni/repos/asan+0x10a53) in arma::subview_col<double>::subview_col(arma::Mat<double> const&, unsigned long long)
Shadow bytes around the buggy address:
  0x1000165bede0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000165bedf0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000165bee00: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000165bee10: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000165bee20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 f1 f1
=>0x1000165bee30: f1 f1[f8]f8 f8 f8 f8 f8 f8 f8 f2 f2 f2 f2 00 00
  0x1000165bee40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000165bee50: 00 00 00 00 f2 f2 00 00 00 00 00 00 00 00 00 00
  0x1000165bee60: 00 00 00 00 00 00 00 00 f1 f1 f1 f1 00 00 00 00
  0x1000165bee70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
  0x1000165bee80: 00 00 f2 f2 00 00 00 00 00 00 00 00 00 00 00 00
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07 
  Heap left redzone:       fa
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==29398==ABORTING

So it doesn't look like an R issue?

[eddelbuettel]

That is entirely possible. Especially with *.0 versions.

[psteinb]

Just had a similar issue with asan in gcc 7.3.1. GCC's libgomp yielded a stack-use-after-scope while clang's equivalent (llvm 5.0.0) produced no stack corruption with the same source code. Bottom line: looks like a false positive.

[eddelbuettel]

Is this still an issue or can we close it?

[sgsokol]

personally, I don't have this issue with g++ 8.4.0

[eddelbuettel]

Ok, thanks,

But can you clarify whether you are referring to normal usage, or usage with ASAN-enabled compiler?

[sgsokol]

I have reproduced the example of "pure" armadillo given here before (hence with ASAN-enabled options) and no error popped up.

[eddelbuettel]

Excellent. (And I presume you have used an ASAN-enabled build of R as e.g. provided by those Docker containers. Plain R would not show it.)

[sgsokol]

Originally, no R was involved in my test for the purity of the experiment. It was a standalone C++ code as in the example above. But now, as you asked, I made also the same test in the rdevel-gcc-san container and still no error. Here is the pertinent part of sourceCpp("asan.cpp", verbose=TRUE) output:

/usr/local/lib/R/bin/R CMD SHLIB -o 'sourceCpp_2.so'  'asan.cpp'  
g++ -fsanitize=undefined,bounds-strict -fno-omit-frame-pointer -std=gnu++11 -I"/usr/local/lib/R/include" -DNDEBUG -I../inst/include   -I"/usr/local/lib/R/site-library/Rcpp/include" -I"/usr/local/lib/R/site-library/RcppArmadillo/include" -I"/home/rcpp-pkgs" -I/usr/local/include  -fopenmp -fpic  -g -O2 -Wall -pedantic -mtune=native  -c asan.cpp -o asan.o
g++ -fsanitize=undefined,bounds-strict -fno-omit-frame-pointer -std=gnu++11 -shared -L/usr/local/lib/R/lib -L/usr/local/lib -o sourceCpp_2.so asan.o -L/usr/local/lib/R/lib -lRlapack -L/usr/local/lib/R/lib -lRblas -lgfortran -lm -lubsan -lquadmath -fopenmp -L/usr/local/lib/R/lib -lR
> foo()
> 

Please note that gcc is v9.2.1 in the docker while it was v8.4 in my previous test.

[eddelbuettel]

Thanks so much! Really appreciate it.