Travis CI: Run sanitizers ASAN / UBSAN automatically

[HenrikBengtsson]

Add Travis CI matrix job

    - RVERSION=devel _R_SAN_=TRUE _R_CHECK_FULL_=TRUE 

that performs:

1. Addess Sanitizer (ASAN)
2. "Undefined Behavior Sanitizer" (UBSAN)

UBSAN requires GCC (>= 4.9.0), but Travis CI only provides GCC 4.6.3 for language: c;

$ gcc --version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3

or clang (>= 3.6?), but Travis CI only provides clang 3.4 for language: c;

$ clang --version
clang version 3.4 (tags/RELEASE_34/final) Target: x86_64-unknown-linux-gnu Thread model: posix

So, need to figure out how to install/update to GCC 4.9.x and/or clang 3.6.x.

Other than this, all that should be needed is to add a ~/.R/Makevars file with:

CC = gcc -std=gnu99 -fsanitize=unknown,address -fno-omit-frame-pointer

This should be enough for R CMD INSTALL to build and install with sanitizer checks. With this setup, it should be possible to confirm the USBAN errors reported by BDR for 0.14.1, cf. Issue #48.

[HenrikBengtsson]

Asked for advice at https://github.com/metacran/r-builder/issues/33

[HenrikBengtsson]

Update

I can reproduce the UBSAN runtime errors reported in Issue #48 (matrixStats v0.14.1) running Ubuntu 15.04 Linux on VirtualBox:

$ Rscript tests/x_OP_y.R
[...]
List of 2
 $ x: int [1:2, 1:2] NA NA NA NA
 $ y: int NA
mode='integer', OP='+', na.rm=FALSE
./x_OP_y_TYPE-template.h:234: runtime error: value nan is outside the range of representable values of type 'int'
./x_OP_y_TYPE-template.h:234: runtime error: value nan is outside the range of representable values of type 'int'
./x_OP_y_TYPE-template.h:234: runtime error: value nan is outside the range of representable values of type 'int'
./x_OP_y_TYPE-template.h:234: runtime error: value nan is outside the range of representable values of type 'int'
 int [1:2, 1:2] NA NA NA NA
mode='integer', OP='+', na.rm=TRUE
./x_OP_y_TYPE-template.h:221: runtime error: value nan is outside the range of representable values of type 'int'
 int [1:2, 1:2] NA NA NA NA
[...]

For this to work, the ~/.R/Makevars file should be:

CC = clang -fsanitize=undefined
PKG_LIBS = -lubsan -lasan

It seems that one need to use clang (here v3.6.0), because using gcc (here v4.9.2) there are no errors;

CC = gcc-4.9 -fsanitize=undefined
PKG_LIBS = -lubsan -lasan

Also, these UBSAN errors are not detected by R CMD check --as-cran. Instead, they have to be manually checked for, e.g.

$ grep -F "error: " matrixStats.Rcheck/tests/*.Rout

[HenrikBengtsson]

Need to add -fno-sanitize=float-divide-by-zero, otherwise we get UBSAN errors on division by zero from running the package tests:

$ grep -E "(error|note):" tests/*.Rout | sort -u
tests/meanOver.Rout:./meanOver_TYPE-template.h:106:17: runtime error: division by zero
tests/meanOver.Rout:./meanOver_TYPE-template.h:61:17: runtime error: division by zero
tests/weightedMean.Rout:./weightedMean_TYPE-template.h:60:15: runtime error: division by zero

originating from:

• https://github.com/HenrikBengtsson/matrixStats/blob/master/src/meanOver_TYPE-template.h#L106
• https://github.com/HenrikBengtsson/matrixStats/blob/master/src/meanOver_TYPE-template.h#L61
• https://github.com/HenrikBengtsson/matrixStats/blob/master/src/weightedMean_TYPE-template.h#L60

[krlmlr]

Have you tried dist: trusty?

[HenrikBengtsson]

Thxs. That's the one that was just announced on Travis, correct? Haven't looked into the details, but it sound like a way forward on this. I'll keep an eye on it and hope to find time soon to look into it.

[krlmlr]

I think it's been around for a while, but not sure if and how documented. I'd be interested to know if if works for you.