llvmbot
commented
9 years ago Sorry, when I say "If I run this test manually", I'm referring to TestProcessLaunch.py. There may be others that are being misreported as well, this is just one easy repro.
Open llvmbot opened 9 years ago
llvmbot
commented
9 years ago Sorry, when I say "If I run this test manually", I'm referring to TestProcessLaunch.py. There may be others that are being misreported as well, this is just one easy repro.
Extended Description
If I run this test manually, two of the tests fails. Example:
d:\src\llvmbuild\ninja\lldb-test-traces>c:\python27_lldb\x86\python_d.exe D:\src\llvm\tools\lldb\test\dotest.py -q --arch=i686 --executable D:/src/llvmbuild/ninja/bin/lldb.exe -s D:/src/llvmbuild/ninja/lldb-test-traces -u CXXFLAGS -u CFLAGS --enable-crash-dialog -C d:\src\llvmbuild\ninja_release\bin\clang.exe D:\src\llvm\tools\lldb\packages\Python\lldbsuite\test\functionalities\process_launch --no-multiprocess The 'lldb-mi' executable cannot be located. The lldb-mi tests can not be run as a result. UNSUPPORTED: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_environment_with_special_char_dsym (TestProcessLaunch.ProcessLaunchTestCase) (dsym tests) FAIL: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_environment_with_special_char_dwarf (TestProcessLaunch.ProcessLaunchTestCase) FAIL: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_environment_with_special_char_dwo (TestProcessLaunch.ProcessLaunchTestCase) UNSUPPORTED: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_io_dsym (TestProcessLaunch.ProcessLaunchTestCase) (dsym tests) PASS: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_io_dwarf (TestProcessLaunch.ProcessLaunchTestCase) PASS: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_io_dwo (TestProcessLaunch.ProcessLaunchTestCase) UNSUPPORTED: LLDB (d:\src\llvmbuild\ninja_release\bin\clang.exe-i686) :: test_set_working_dir_dsym (TestProcessLaunch.ProcessLaunchTestCase) (dsym tests)
When I run the multiprocess test runner via dosep, it's being reported as a success.
410 out of 410 test suites processed - TestAddDsymCommand.py Ran 410 test suites (1 failed) (0.243902%) Ran 318 test cases (4 failed) (1.257862%) Failing Tests (1) FAIL: LLDB (suite) :: TestBreakpointLanguage.py (Windows zturner-win81 8 6.2.9200 AMD64 Intel64 Family 6 Model 45 Stepping 7, GenuineIntel) [79964 refs] ninja: build stopped: subcommand failed.
Stepping through the code, in dosep.py when it goes to print the summary list of failed tests, 'TestProcessLaunch.py' is in the list of passed tests. So something is wrong here, and a potentially unknown number of tests are currently being misreported.