Sanitizer builds are broken (regressed on travis)

[springmeyer]

https://github.com/travis-ci/travis-ci/issues/8840 notes what we are seeing as of today for node-cpp-skel and all repos based on it: the sanitizer job is crashing due to a regression in travis.

LeakSanitizer has encountered a fatal error

/cc @mapbox/core-tech

[mapsam]

This was fixed! https://travis-ci.org/mapbox/node-cpp-skel/jobs/308170250 is now 🍏 again.

[mapsam]

Unfortunately, this is popping up again 😢 https://travis-ci.org/mapbox/vtquery/jobs/312013091

[springmeyer]

@mapsam - that looks slightly different to me. I also see that the LSAN debugging flag is on still in that build. Can you turn it off, try another build, and then post what you are seeing?

[springmeyer]

Closing, looks like @mapsam resolved the vtquery specific issue by removing LSAN_OPTIONS in https://github.com/mapbox/vtquery/commit/18b682b7f0450f572fd755960b794d52ccce5f6c#diff-354f30a63fb0907d4ad57269548329e3

[springmeyer]

Re-opening. This is now happening again per https://github.com/travis-ci/travis-ci/issues/8836#issuecomment-356362524. Going to keep this open as a way of tracking this for @mapbox/core-tech (since all node-cpp-skel and hpp-skel based repos that leverage the sanitizers will have that job failing).

For now I propose:

• letting the job fail, ignore for now
• we'll wait a week or so for travis to try to fix and close https://github.com/travis-ci/travis-ci/issues/8836#issuecomment-356362524
• if the problem persists then we'll look at applying a workaround (like sudo: true, but ideally we'll not need to do anything)

Also: moving to circleci is another option, but let's not do this for this reason. Rather let's build up knowledge of circleci 2.0 and its merits and not worry about a mass migration yet.

[springmeyer]

if the problem persists then we'll look at applying a workaround (like sudo: true, but ideally we'll not need to do anything)

Problem is persisting. Let's move to applying sudo: required overload for sanitizer jobs. We can keep sudo: false for other jobs. /cc @mapbox/core-tech

[mapsam]

Adding some of the santizer errors to this ticket (instead of links to travis builds) so it's more search friendly:

==18649==AddressSanitizer: failed to intercept '__isoc99_printf'
==18649==AddressSanitizer: failed to intercept '__isoc99_sprintf'
==18649==AddressSanitizer: failed to intercept '__isoc99_snprintf'

==18651==Installed the sigaction for signal 11
==18651==Installed the sigaction for signal 7
==18651==Installed the sigaction for signal 8
==18651==T0: stack [0x7ffcefeff000,0x7ffcf06ff000) size 0x800000; local=0x7ffcf06fc21c
==18651==AddressSanitizer Init done

|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap  ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000

==18667==T6: FakeStack created: 0x7f633d2db000 -- 0x7f633dde4000 stack_size_log: 20; mmapped 11300K, noreserve=0 

==18667==T6 TSDDtor
==18667==T6 exited
==18674==Processing thread 18667.
==18674==Stack at 0x7ffd9a93e000-0x7ffd9b13e000 (SP = 0x7ffd9b138128).
==18674==TLS at 0x7f6349ed4000-0x7f6349ed50c0.

==18667==WARNING: Can't read from symbolizer at fd 10
==18667==WARNING: Can't read from symbolizer at fd 10
==18667==WARNING: Can't read from symbolizer at fd 10
==18667==WARNING: Can't read from symbolizer at fd 10
==18667==WARNING: Failed to use and restart external symbolizer!
=================================================================
==18667==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 1024 byte(s) in 1 object(s) allocated from:

[springmeyer]

@mapsam The diagnostic error is:

=4346==LeakSanitizer has encountered a fatal error.
==4346==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==4346==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)

Which indicates broken ptrace support (ability for an external program to attach to another). I think those other lines you shared are a result of setting LSAN_OPTIONS=verbosity=1:log_threads=1 which is not recommended and does not fix the problem.

[mapsam]

Thanks for including those @springmeyer - looks like you already called me out on this above. Forgot about my past self apparently!