search

theQRL / go-qrl

Go version of QRL node
MIT License
9 stars 7 forks source link

using goqryptonight with AddressSanitizer prevents unit tests from compiling and running #70

Closed randomshinichi closed 6 years ago

randomshinichi commented 6 years ago

The problem:

# github.com/theQRL/go-qrl/pkg/core/block (testmain)
//usr/lib/x86_64-linux-gnu/libasan.so.2: warning: the use of `tmpnam' is dangerous, better use `mkstemp'
//usr/lib/x86_64-linux-gnu/libasan.so.2: warning: the use of `tempnam' is dangerous, better use `mkstemp'
//usr/lib/x86_64-linux-gnu/libasan.so.2: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'
==7847==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
FAIL    github.com/theQRL/go-qrl/pkg/core/block 0.002s

It turns out that if you run LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.4 go test -v ./pkg/core/block then it will fail to compile. Notice the g++ failed: exit status 1. This means that the compilation step failed.

shinichi@ilya:~/source/go/src/github.com/theQRL/go-qrl [asanfix*]$ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.4 go test -c -v ./pkg/core/block
# github.com/theQRL/go-qrl/pkg/core/block (testmain)
/usr/lib/go-1.10/pkg/tool/linux_amd64/link: running g++ failed: exit status 1
//usr/lib/x86_64-linux-gnu/libasan.so.4: warning: the use of `tmpnam' is dangerous, better use `mkstemp'
//usr/lib/x86_64-linux-gnu/libasan.so.4: warning: the use of `tempnam' is dangerous, better use `mkstemp'
//usr/lib/x86_64-linux-gnu/libasan.so.4: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'

=================================================================
==11635==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 10952 byte(s) in 6 object(s) allocated from:
    #0 0x7fb169669b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7fb16928b2e1 in bfd_malloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x502e1)

To get the tests, you have to compile the package in a separate step: go test -c -v ./pkg/core/block. This will produce an executable block.test.

If you compiled goqryptonight with ASAN, you need to run LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.4 ./block.test otherwise it will fail with this error: ==7847==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.

Since there is no way to specify that LD_PRELOAD should NOT be set during compilation but should be set during test running within go-test, the best thing to do is just disable AddressSanitizer for goqryptonight. Perhaps AddressSanitizer should only be used within goqryptonight tests, not in tests for other components that depend on goqryptonight.

Explanation: compilation probably fails because leaks were detected within ld/gcc itself. The following stack trace shows that none of the leak reports actually mentioned our code.

shinichi@ilya:~/source/go/src/github.com/theQRL/go-qrl [master*]$ ASAN_OPTIONS=fast_unwind_on_malloc=0 LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.4 go test -c -v ./pkg/core/block
# github.com/theQRL/go-qrl/pkg/core/block (testmain)
/usr/lib/go-1.10/pkg/tool/linux_amd64/link: running g++ failed: exit status 1
//usr/lib/x86_64-linux-gnu/libasan.so.4: warning: the use of `tmpnam' is dangerous, better use `mkstemp'
//usr/lib/x86_64-linux-gnu/libasan.so.4: warning: the use of `tempnam' is dangerous, better use `mkstemp'
//usr/lib/x86_64-linux-gnu/libasan.so.4: warning: the use of `tmpnam_r' is dangerous, better use `mkstemp'

=================================================================
==23667==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 7228 byte(s) in 3 object(s) allocated from:
    #0 0x7f564c977b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f564c5992e1 in bfd_malloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x502e1)
    #2 0x7f564c5e0334 in _bfd_elf_strtab_save (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x97334)
    #3 0x7f564c5d43dd in bfd_elf_link_add_symbols (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x8b3dd)
    #4 0x55f83c76156b  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1d56b)
    #5 0x55f83c761fef  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1dfef)
    #6 0x55f83c761ec6  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1dec6)
    #7 0x55f83c764270  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x20270)
    #8 0x55f83c7524b3  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe4b3)
    #9 0x7f564bf75b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #10 0x55f83c752b59  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xeb59)

Direct leak of 5600 byte(s) in 1 object(s) allocated from:
    #0 0x7f564c977f40 in realloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdef40)
    #1 0x7f564c643a4c in xrealloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xfaa4c)
    #2 0x55f83c772716  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2e716)
    #3 0x55f83c757c73  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x13c73)
    #4 0x55f83c75238e  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe38e)
    #5 0x7f564bf75b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #6 0x55f83c752b59  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xeb59)

Direct leak of 5536 byte(s) in 1 object(s) allocated from:
    #0 0x7f564c977b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x55f83c757b61  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x13b61)
    #2 0x55f83c75238e  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe38e)
    #3 0x7f564bf75b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #4 0x55f83c752b59  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xeb59)

Direct leak of 3724 byte(s) in 3 object(s) allocated from:
    #0 0x7f564c977b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f564c5992e1 in bfd_malloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x502e1)
    #2 0x7f564c5e0334 in _bfd_elf_strtab_save (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x97334)
    #3 0x7f564c5d43dd in bfd_elf_link_add_symbols (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x8b3dd)
    #4 0x55f83c76156b  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1d56b)
    #5 0x55f83c761fef  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1dfef)
    #6 0x55f83c764270  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x20270)
    #7 0x55f83c7524b3  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe4b3)
    #8 0x7f564bf75b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #9 0x55f83c752b59  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xeb59)

Direct leak of 1871 byte(s) in 31 object(s) allocated from:
    #0 0x7f564c977b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f564c6439c7 in xmalloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xfa9c7)
    #2 0x7f564c63e977 in concat (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xf5977)
    #3 0x55f83c770628  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2c628)
    #4 0x55f83c770fe8  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2cfe8)
    #5 0x55f83c76ddb5  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x29db5)
    #6 0x55f83c76e433  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2a433)
    #7 0x55f83c7611ea  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1d1ea)
    #8 0x55f83c761fef  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1dfef)
    #9 0x55f83c764270  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x20270)
    #10 0x55f83c7524b3  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe4b3)
    #11 0x7f564bf75b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #12 0x55f83c752b59  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xeb59)

Direct leak of 780 byte(s) in 68 object(s) allocated from:
    #0 0x7f564c977b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f564c6439c7 in xmalloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xfa9c7)
    #2 0x7f564c643ad9 in xstrdup (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xfaad9)
    #3 0x55f83c756d44  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x12d44)
    #4 0x55f83c7535b6  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xf5b6)
    #5 0x55f83c752664  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe664)
    #6 0x7f564bf75b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #7 0x55f83c752b59  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xeb59)

Direct leak of 397 byte(s) in 5 object(s) allocated from:
    #0 0x7f564c977b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f564c6439c7 in xmalloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xfa9c7)
    #2 0x7f564c63e977 in concat (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xf5977)
    #3 0x55f83c770628  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2c628)
    #4 0x7f564c62ca74  (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xe3a74)
    #5 0x7f564c597f53 in bfd_check_format_matches (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x4ef53)
    #6 0x55f83c76dcfa  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x29cfa)
    #7 0x55f83c772808  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2e808)
    #8 0x55f83c76e27a  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2a27a)
    #9 0x55f83c76e3bc  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2a3bc)
    #10 0x55f83c7611ea  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1d1ea)
    #11 0x55f83c761fef  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x1dfef)
    #12 0x55f83c764270  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x20270)
    #13 0x55f83c7524b3  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe4b3)
    #14 0x7f564bf75b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #15 0x55f83c752b59  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xeb59)

Direct leak of 320 byte(s) in 1 object(s) allocated from:
    #0 0x7f564c977b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f564c6439c7 in xmalloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0xfa9c7)
    #2 0x55f83c770bd1  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x2cbd1)
    #3 0x55f83c7523a3  (/usr/bin/x86_64-linux-gnu-ld.bfd+0

.......

Indirect leak of 18 byte(s) in 1 object(s) allocated from:
    #0 0x7fa942fceb50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x4720c7  (/usr/bin/x86_64-linux-gnu-g++-7+0x4720c7)
    #2 0x4721d9  (/usr/bin/x86_64-linux-gnu-g++-7+0x4721d9)
    #3 0x40959d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40959d)
    #4 0x409952  (/usr/bin/x86_64-linux-gnu-g++-7+0x409952)
    #5 0x40d63c  (/usr/bin/x86_64-linux-gnu-g++-7+0x40d63c)
    #6 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #7 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #8 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #9 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #10 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #11 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #12 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #13 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #14 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #15 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #16 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #17 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #18 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #19 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #20 0x40fe9d  (/usr/bin/x86_64-linux-gnu-g++-7+0x40fe9d)
    #21 0x40dcfd  (/usr/bin/x86_64-linux-gnu-g++-7+0x40dcfd)
    #22 0x40efe6  (/usr/bin/x86_64-linux-gnu-g++-7+0x40efe6)
    #23 0x411110  (/usr/bin/x86_64-linux-gnu-g++-7+0x411110)
    #24 0x403aaa  (/usr/bin/x86_64-linux-gnu-g++-7+0x403aaa)
    #25 0x403d63  (/usr/bin/x86_64-linux-gnu-g++-7+0x403d63)
    #26 0x7fa942b20b96 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21b96)
    #27 0x403e99  (/usr/bin/x86_64-linux-gnu-g++-7+0x403e99)

SUMMARY: AddressSanitizer: 4472 byte(s) leaked in 23 allocation(s).

shinichi@ilya:~/source/go/src/github.com/theQRL/go-qrl [master*]$

This is what happens when you go LD_PRELOAD go test -v ./... - even fails at AddressState:

shinichi@ilya:~/source/go/src/github.com/theQRL/go-qrl [master*]$ ASAN_OPTIONS=fast_unwind_on_malloc=0 LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libasan.so.4 go test -v ./...
?       github.com/theQRL/go-qrl/cmd/gqrl   [no test files]
?       github.com/theQRL/go-qrl/pkg/config [no test files]
# github.com/theQRL/go-qrl/pkg/core/addressstate (testmain)
/usr/lib/go-1.10/pkg/tool/linux_amd64/link: running g++ failed: exit status 1

=================================================================
==24012==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 757008 byte(s) in 1 object(s) allocated from:
    #0 0x7f45749a8b50 in __interceptor_malloc (/usr/lib/x86_64-linux-gnu/libasan.so.4+0xdeb50)
    #1 0x7f45745ca2e1 in bfd_malloc (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x502e1)
    #2 0x7f457460b4aa in bfd_elf_final_link (/usr/lib/x86_64-linux-gnu/libbfd-2.30-system.so+0x914aa)
    #3 0x55c9aed024ad  (/usr/bin/x86_64-linux-gnu-ld.bfd+0x254ad)
    #4 0x55c9aeceb519  (/usr/bin/x86_64-linux-gnu-ld.bfd+0xe519)
randomshinichi commented 6 years ago

Instead of using AddressSanitizer in this fashion, we could investigate this instead:

//  -msan
//      enable interoperation with memory sanitizer.
//      Supported only on linux/amd64, linux/arm64
//      and only with Clang/LLVM as the host C compiler.

Funnily enough this extra, important piece of info seems to appear and disappear, and it is not in go version 1.10.

Related issue on golang repo