search

DynamoRIO / dynamorio

Dynamic Instrumentation Tool Platform
Other
2.63k stars 557 forks source link

fix failing MacOS tests #1815

Open derekbruening opened 8 years ago

derekbruening commented 8 years ago

Split from #58.

Only recently did we get the test suite to build on Mac (we were blocked by #1409). Now they all build but only a quarter pass:

> egrep '^Test Pass|Testing:' ~/derek/dr/git/build_x86_dbg_tests/Testing/Temporary/LastTest.log | grep -B 1 Pass | grep '^[0-9]' | sort -n 
1/184 Testing: unit_tests
2/184 Testing: samples_proj
3/184 Testing: code_api|common.broadfun
4/184 Testing: code_api|common.fib
7/184 Testing: code_api|common.floatpc
8/184 Testing: code_api|common.floatpc_xl8all
9/184 Testing: code_api|common.getretaddr
11/184 Testing: code_api|libutil.frontend_test
16/184 Testing: code_api|linux.exit
20/184 Testing: code_api|linux.infinite
23/184 Testing: code_api|linux.signal0000
24/184 Testing: code_api|linux.signal0001
25/184 Testing: code_api|linux.signal0010
27/184 Testing: code_api|linux.signal0100
28/184 Testing: code_api|linux.signal0101
29/184 Testing: code_api|linux.signal0110
47/184 Testing: code_api|linux.sigaction
53/184 Testing: code_api|linux.app_tls
55/184 Testing: code_api|linux.fib-conflict
57/184 Testing: code_api|linux.fib-pie
69/184 Testing: code_api|security-common.codemod
71/184 Testing: code_api|security-common.retexisting
72/184 Testing: code_api|security-common.ret_noncall_trace
79/184 Testing: code_api|client.abort
80/184 Testing: code_api|client.crashmsg
84/184 Testing: code_api|client.count-ctis-noopt
88/184 Testing: code_api|client.exception
104/184 Testing: code_api|client.timer
105/184 Testing: code_api|client.syscall-mod
107/184 Testing: code_api|client.cbr-retarget
109/184 Testing: code_api|client.flush
112/184 Testing: code_api|client.truncate
113/184 Testing: code_api|client.dr_options
114/184 Testing: code_api|client.unregister
116/184 Testing: code_api|client.null_instrument
118/184 Testing: code_api|client.option_parse
119/184 Testing: code_api|client.drcontainers-test
129/184 Testing: code_api|client.fcache_shift
132/184 Testing: code_api|client.destructor
150/184 Testing: code_api|sample.signal
163/184 Testing: code_api|tool.drcpusim.simple
167/184 Testing: code_api|tool.drcpusim.cpuid-Banias
168/184 Testing: code_api|tool.drcpusim.cpuid-Prescott
169/184 Testing: code_api|tool.drcpusim.cpuid-Presler
170/184 Testing: code_api|tool.drcpusim.cpuid-Merom
171/184 Testing: code_api|tool.drcpusim.cpuid-Penryn
172/184 Testing: code_api|tool.drcpusim.cpuid-Westmere
173/184 Testing: code_api|tool.drcpusim.cpuid-Nehalem
174/184 Testing: code_api|api.ir
175/184 Testing: code_api|api.dis
177/184 Testing: code_api|api.ir-static
178/184 Testing: code_api|api.static
28% tests passed, 132 tests failed out of 184

The following tests FAILED:
          5 - code_api|common.decode-bad (Failed)
          6 - code_api|common.eflags (Failed)
         10 - code_api|common.segfault (Failed)
         12 - code_api|linux.execve-null (Failed)
         13 - code_api|linux.child-blacklist (Failed)
         14 - code_api|linux.child-whitelist (Failed)
         15 - code_api|linux.execve-rec (Failed)
         17 - code_api|linux.fork (Failed)
         18 - code_api|linux.fork-sleep (Failed)
         19 - code_api|linux.fork-MSR (Failed)
         21 - code_api|linux.longjmp (Failed)
         22 - code_api|linux.mmap (Failed)
         26 - code_api|linux.signal0011 (Failed)
         30 - code_api|linux.signal0111 (Failed)
         31 - code_api|linux.signal1000 (Failed)
         32 - code_api|linux.signal1001 (Failed)
         33 - code_api|linux.signal1010 (Failed)
         34 - code_api|linux.signal1011 (Failed)
         35 - code_api|linux.signal1100 (Failed)
         36 - code_api|linux.signal1101 (Failed)
         37 - code_api|linux.signal1110 (Failed)
         38 - code_api|linux.signal1111 (Failed)
         39 - code_api|linux.sigplain000 (Failed)
         40 - code_api|linux.sigplain001 (Failed)
         41 - code_api|linux.sigplain010 (Failed)
         42 - code_api|linux.sigplain011 (Failed)
         43 - code_api|linux.sigplain100 (Failed)
         44 - code_api|linux.sigplain101 (Failed)
         45 - code_api|linux.sigplain110 (Failed)
         46 - code_api|linux.sigplain111 (Failed)
         48 - code_api|linux.alarm (Failed)
         49 - code_api|linux.eintr (Failed)
         50 - code_api|linux.eintr-noinline (Failed)
         51 - code_api|linux.sigsuspend (Failed)
         52 - code_api|linux.signest (Failed)
         54 - code_api|linux.readlink (Failed)
         56 - code_api|linux.fib-conflict-early (Failed)
         58 - code_api|linux.vfork (Failed)
         59 - code_api|linux.thread-reset (OTHER_FAULT)
         60 - code_api|linux.clone-reset (OTHER_FAULT)
         61 - code_api|pthreads.pthreads (Failed)
         62 - code_api|pthreads.pthreads_exit (Failed)
         63 - code_api|pthreads.ptsig_FLAKY (Failed)
         64 - code_api|pthreads.pthreads_fork_FLAKY (Failed)
         65 - code_api|linux.reset (Failed)
         66 - code_api|linux.freeze_FLAKY (Failed)
         67 - code_api|linux.persist_FLAKY (Failed)
         68 - code_api|linux.persist-use_FLAKY (Failed)
         70 - code_api|security-common.decode-bad-stack (Failed)
         73 - code_api|security-common.retnonexisting (Failed)
         74 - code_api|security-common.selfmod2 (Failed)
         75 - code_api|security-common.selfmod (Failed)
         76 - code_api|security-common.TestAllocWE (Failed)
         77 - code_api|security-common.TestMemProtChg_FLAKY (Failed)
         78 - code_api|client.call-retarget (Failed)
         81 - code_api|client.alloc (Failed)
         82 - code_api|client.cleancall (Failed)
         83 - code_api|client.count-ctis (Failed)
         85 - code_api|client.syscall (Failed)
         86 - code_api|client.count-bbs (Failed)
         87 - code_api|client.app_inscount (Failed)
         89 - code_api|client.partial_module_map (Failed)
         90 - code_api|client.segfault (Failed)
         91 - code_api|client.execfault (Failed)
         92 - code_api|client.vg-annot (Failed)
         93 - code_api|client.vg-annot.full-decode (Failed)
         94 - code_api|client.vg-annot.full-decode.tiny-bb (Failed)
         95 - code_api|client.vg-annot.bb-truncate-1 (Failed)
         96 - code_api|client.vg-annot.bb-truncate-2 (Failed)
         97 - code_api|client.vg-annot-opt (Failed)
         98 - code_api|client.vg-annot-opt.full-decode (Failed)
         99 - code_api|client.vg-annot-opt.full-decode.tiny-bb (Failed)
        100 - code_api|client.vg-annot-opt.bb-truncate (Failed)
        101 - code_api|client.events (Failed)
        102 - code_api|client.events_cpp (Failed)
        103 - code_api|client.nudge_test (Failed)
        106 - code_api|client.signal (Failed)
        108 - code_api|client.file_io (Failed)
        110 - code_api|client.thread (Failed)
        111 - code_api|client.strace (Failed)
        115 - code_api|client.inline (Failed)
        117 - code_api|client.large_options (Failed)
        120 - code_api|client.drmgr-test (Failed)
        121 - code_api|client.drreg-test (Failed)
        122 - code_api|client.drx-test (Failed)
        123 - code_api|client.drwrap-test (Failed)
        124 - code_api|client.drwrap-test-callconv (Failed)
        125 - code_api|client.drsyms-test (Failed)
        126 - code_api|client.drutil-test (Failed)
        127 - code_api|client.pcache (Failed)
        128 - code_api|client.pcache-use (Failed)
        130 - code_api|client.nudge_ex (Failed)
        131 - code_api|client.retaddr (Failed)
        133 - code_api|sample.bbbuf (Failed)
        134 - code_api|sample.bbsize (Failed)
        135 - code_api|sample.empty (Failed)
        136 - code_api|sample.memtrace_simple (Failed)
        137 - code_api|sample.bbcount (Failed)
        138 - code_api|sample.cbr (Failed)
        139 - code_api|sample.countcalls (Failed)
        140 - code_api|sample.div (Failed)
        141 - code_api|sample.inc2add (Failed)
        142 - code_api|sample.memtrace_x86 (Failed)
        143 - code_api|sample.prefetch (Failed)
        144 - code_api|sample.ssljack (Failed)
        145 - code_api|sample.modxfer (Failed)
        146 - code_api|sample.modxfer_app2lib (Failed)
        147 - code_api|sample.instrcalls (Failed)
        148 - code_api|sample.cbrtrace (Failed)
        149 - code_api|sample.wrap (Failed)
        151 - code_api|sample.syscall (Failed)
        152 - code_api|sample.inline (Failed)
        153 - code_api|sample.inscount (Failed)
        154 - code_api|sample.inscount.cleancall (Failed)
        155 - code_api|sample.opcodes (Failed)
        156 - code_api|sample.stl_test (Failed)
        157 - code_api|tool.drltrace (Failed)
        158 - code_api|tool.drcov.fib (Failed)
        159 - code_api|tool.drcachesim.simple (Failed)
        160 - code_api|tool.drcachesim.TLB-simple (Failed)
        161 - code_api|tool.drcachesim.phys (Failed)
        162 - code_api|tool.drcachesim.multiproc (Failed)
        164 - code_api|tool.drcpusim.cpuid-Klamath (Failed)
        165 - code_api|tool.drcpusim.cpuid-Deschutes (Failed)
        166 - code_api|tool.drcpusim.cpuid-Pentium3 (Failed)
        176 - code_api|api.startstop_FLAKY (Failed)
        179 - code_api,opt_memory|client.events (Failed)
        180 - code_api,opt_speed|client.events (Failed)
        181 - code_api,thread_private|client.events (Failed)
        182 - code_api,disable_traces|client.events (Failed)
        183 - code_api,thread_private,disable_traces|client.events (Failed)
        184 - code_api,no_early_inject|client.events (Failed)
Errors while running CTest
derekbruening commented 4 years ago

These are now failing on 64-bit:

debug-internal-64: 45 tests passed, **** 4 tests failed, but ignoring 0 for i: ****
    code_api|common.decode-bad 
    code_api|client.abort 
    code_api|client.timer 
    code_api|tool.drcpusim.simple 

Privileged instructions about to happen
<Application /Users/travis/build/DynamoRIO/dynamorio/build_debug-internal-64/suite/tests/bin/common.decode-bad (33461).  Internal Error: DynamoRIO debug check failure: /Users/travis/build/DynamoRIO/dynamorio/core/unix/signal.c:2461 mc != NULL && sc != NULL
Error occurred @1408 frags)

  /Users/travis/build/DynamoRIO/dynamorio/build_debug-internal-64/bin64/drrun;-s;90;-quiet;-debug;-killpg;-stderr_mask;0xC;-dumpcore_mask;0;-code_api;-c;/Users/travis/build/DynamoRIO/dynamorio/build_debug-internal-64/suite/tests/bin/libclient.abort.dll.dylib;--;/Users/travis/build/DynamoRIO/dynamorio/build_debug-internal-64/suite/tests/bin/client.abort
  failed: 255 != 8 

<Application /Users/travis/build/DynamoRIO/dynamorio/build_debug-internal-64/suite/tests/bin/client.timer (33895).  Internal Error: DynamoRIO debug check failure: /Users/travis/build/DynamoRIO/dynamorio/core/unix/signal.c:2461 mc != NULL && sc != NULL
(Error occurred @1407 frags)

38/49 Test #216: code_api|tool.drcpusim.simple ................***Failed  Required regular expression not found.Regex=[^Hello, world!
$
]  0.36 sec
<Invalid Westmere instruction "xsave" @ libdyld.dylib+0x1281.  Aborting.>

As part of #1979 I'm removing them from the list for now.

derekbruening commented 4 years ago

On OSX 10.14 (10.13 was the Travis default):

https://travis-ci.com/DynamoRIO/dynamorio/jobs/287117155

49913/45 Test  #52: code_api|linux.sigaction .....................***Failed  Required regular expression not found. Regex=[^Sending SIGTERM first time
500Sending SIGTERM second time
501$
502]  0.41 sec
503Assertion failed: (rc == 0 && old_act.sa_sigaction == first_act.sa_sigaction && *(long *)&old_act.sa_mask == *(long *)&first_act.sa_mask), function test_query, file /Users/travis/build/DynamoRIO/dynamorio/suite/tests/linux/sigaction.c, line 83.
504<Application /Users/travis/build/DynamoRIO/dynamorio/build_debug-internal-64/suite/tests/bin/linux.sigaction (23090).  Internal Error: DynamoRIO debug check failure: /Users/travis/build/DynamoRIO/dynamorio/core/unix/os.c:1801 false
505(Error occurred @1426 frags)