Unittests Widget isn't reporting all passed tests

[th3coop]

Short Summary

While reporting https://github.com/Komodo/KomodoEdit/issues/2642 I realized that a LOT of the tests aren't being reported as passed where the summary numbers are displayed beside the test class that was run.

For example, running TestJavaScriptLegacyCompletions alone in the Komodo UI looks like this:

============================= test session starts =============================
platform win32 -- Python 2.7.13, pytest-3.1.1, py-1.4.33, pluggy-0.4.0 -- C:\Users\cgcho\Projects\komodo\codeintel\env\Scripts\python.exe
cachedir: .cache
rootdir: C:\Users\cgcho\Projects\komodo\codeintel, inifile:
collecting ... collected 41 items

tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_anon_class_properties PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_builtin_types PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_builtin_vars <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_closure PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_cpln_with_unknown_parent PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_ctor_scope_cheat PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_es6_rest_parameter PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_event_heuristic PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_exclude_dirs <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_fat_arrow_functions PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_files_in_same_dir <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_find_scope_from_line PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_function_alias_no_parens PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_function_completions PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_function_extra_completions PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_function_return_chaining PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_function_return_types PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_global_accessor PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_hash_completions PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_html_style_attribute <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_implied_globals_not_exported <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_instance_defined_in_class_function PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_instance_name_same_as_class PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_intermixed_class_definitions PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_jsdoc_constructor PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_keyword_completions <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_local PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_local2 <- ..\..\..\..\..\Python27\Lib\unittest\case.py xfail
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_local3 PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_local_variable_completions PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_namespace_mapping <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_obj_var_with_method_assignment PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_object_assignment PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_private_variables <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_prototype_class PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_prototype_vars PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_scoped_variable_constructor PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_string_literals PASSED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_variable_call <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_window_history <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_window_location <- ..\..\..\..\..\Python27\Lib\unittest\case.py SKIPPED
=========================== short test summary info ===========================
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: document does not exist in the main scope
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: local variables not implemented yet
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: window and history are not in the global namespace
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: imports from all files in current directory not implemented yet
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: keyword completions not supported yet
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: exclude directories not supported yet
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: namespace mapping not supported yet.
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: document is not in the global namespace
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: type-chain resolution is too overzealous
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: window is not in the global namespace
SKIP [1] ..\..\..\..\..\Python27\Lib\unittest\case.py:59: auto-globals from JS files in same directory not supported
XFAIL tests/language_tests/legacy/javascript/test_completer.py::TestJavaScriptLegacyCompletions::test_local2
  reason: 

============== 29 passed, 11 skipped, 1 xfailed in 0.96 seconds ===============

But Komodo says none passed. See the screenshot below.

Some times tests DO show that things passed though:

============================= test session starts =============================
platform win32 -- Python 2.7.13, pytest-3.1.1, py-1.4.33, pluggy-0.4.0 -- C:\Users\cgcho\Projects\komodo\codeintel\env\Scripts\python.exe
cachedir: .cache
rootdir: C:\Users\cgcho\Projects\komodo\codeintel, inifile:
collecting ... collected 11 items

tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyGotoDefinition::test_basic_goto_definition PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyGotoDefinition::test_file_goto_definition FAILED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyGotoDefinition::test_scoped_goto_definition PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_argument_defn_line PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_external_defns PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_inline_defns PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_inline_variables PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_inst_var PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_peer_module_defns FAILED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_scope_bounds PASSED
tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_scope_bounds_02 PASSED
=========================== short test summary info ===========================
FAIL tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyGotoDefinition::test_file_goto_definition
FAIL tests/language_tests/legacy/ruby/test_goto_definition.py::TestRubyLegacyGotoDefinition::test_peer_module_defns

================================== FAILURES ===================================
______________ TestRubyGotoDefinition.test_file_goto_definition _______________
tests\language_tests\legacy\ruby\test_goto_definition.py:50: in test_file_goto_definition
    }, env={"RUBYLIB": "."}, file="foo.rb", line=1)
tests\language_tests\common.py:421: in assertDefinitionIs
    self.assertIsNotNone(definition)
E   AssertionError: unexpectedly None
_____________ TestRubyLegacyGotoDefinition.test_peer_module_defns _____________
tests\language_tests\legacy\ruby\test_goto_definition.py:119: in test_peer_module_defns
    self.assertDefinitionIs(files, env={"RUBYLIB": "."}, type=AbstractNamespace, name="Dolphin", file=os.path.join("dolphin","fin.rb"), line=1)
tests\language_tests\common.py:421: in assertDefinitionIs
    self.assertIsNotNone(definition)
E   AssertionError: unexpectedly None
===================== 2 failed, 9 passed in 0.68 seconds ======================

[Naatan]

Please share an stdout of a test you ran that showed this problem. The shorter the stdout the better.

[th3coop]

I did. It's in the original report.

[Naatan]

I need the complete stdout, not part of it. When I say shorter the better I mean ideally from a reproduction case with as little output as possible, do not truncate the output.

[th3coop]

Do you not see this when you run the code intel unit tests in Komodo? I don't think setting up a new test suite of custom tests is a good use of time right now to be frank. I can paste the stdout from Komodo after running the test suite if that would help.

[th3coop]

@Naatan, I updated the output to show what Komodo outputs in stdout. I ran the tests by class to get a little output as possible. Unfortunately this doesn't reproduce the issue. It looks like only running the entire test suite causes this issue.

Running the test suite again to confirm.