jeffwright13
commented
1 year ago Improved with upcoming relrease, but still needs tweaking, because not all outcomes match.
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓
┃ Test NodeId ┃ Duration ┃ Outcome ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩
│ demo_tests/test_0.py::test0_pass_1 │ 50.4 ms │ failed │
│ demo_tests/test_0.py::test0_pass_2_logs │ 2.0039 ms │ passed │
│ demo_tests/test_0.py::test0_pass_3_error_in_fixture │ 2.0983 ms │ passed │
│ demo_tests/test_0.py::test0_fail_1 │ 6.227 ms │ failed │
│ demo_tests/test_0.py::test0_skip │ 18.271 ms │ passed │
│ demo_tests/test_0.py::test0_xfail │ 6.0532 ms │ XFailed │
│ demo_tests/test_0.py::test0_xpass │ 2.8989 ms │ XPassed │
│ demo_tests/test_0.py::test0_warning │ 2.5411 ms │ passed │
│ demo_tests/test_0.py::test_flaky_3 │ 3.1979 ms │ passed │
│ demo_tests/test_1.py::test_a_ok │ 992.4 ms │ passed │
│ demo_tests/test_1.py::test_b_fail │ 1.051 s │ failed │
│ demo_tests/test_1.py::test_c_error │ 1.641 ms │ failed │
│ demo_tests/test_1.py::test_d1_skip_inline │ 1.0406 s │ skipped │
│ demo_tests/test_1.py::test_d2_skip │ 1.0613 s │ passed │
│ demo_tests/test_1.py::test_d3_skip_decorator │ 1.0233 s │ skipped │
│ demo_tests/test_1.py::test_e1_xfail_by_inline_and_has_reason │ 1.0664 s │ XFailed │
│ demo_tests/test_1.py::test_e2_xfail_by_decorator_and_has_reason │ 1.0059 s │ XFailed │
│ demo_tests/test_1.py::test_f1_xfails_by_inline_even_though_assertTrue_happens_before_pytestDotXfail │ 1.0997 s │ XFailed │
│ demo_tests/test_1.py::test_f2_xpass_by_xfail_decorator_and_has_reason │ 1.0323 s │ XPassed │
│ demo_tests/test_1.py::test_g_eval_parameterized[3+5-8] │ 988.06 ms │ passed │
│ demo_tests/test_1.py::test_g_eval_parameterized[2+4-6] │ 999.88 ms │ passed │
│ demo_tests/test_1.py::test_g_eval_parameterized[6*9-42] │ 979.67 ms │ failed │
│ demo_tests/test_1.py::test_1_passes_and_has_logging_output │ 1.0405 s │ passed │
│ demo_tests/test_1.py::test_2_fails_and_has_logging_output │ 1.0608 s │ failed │
│ demo_tests/test_1.py::test_3_fails │ 1.0174 s │ failed │
│ demo_tests/test_1.py::test_4_passes │ 1.0101 s │ passed │
│ demo_tests/test_1.py::test_5_marked_SKIP │ 17.155 ms │ passed │
│ demo_tests/test_1.py::test_6_marked_xfail_by_decorator_but_passes_and_has_no_reason │ 1.0191 s │ XPassed │
│ demo_tests/test_1.py::test_7_marked_xfail_by_decorator_and_fails_and_has_no_reason │ 994.77 ms │ XFailed │
│ demo_tests/test_1.py::test_8_causes_a_warning │ 1.0593 s │ failed │
│ demo_tests/test_1.py::test_9_lorem_fails │ 10.888 ms │ failed │
│ demo_tests/test_1.py::test_10_fail_capturing │ 995.12 ms │ failed │
│ demo_tests/test_1.py::test_10b_failed_capturing │ 1.0882 s │ failed │
│ demo_tests/test_1.py::test_11_pass_capturing │ 44.739 ms │ failed │
│ demo_tests/test_1.py::test_12_fails_and_has_stdout │ 1.0144 s │ failed │
│ demo_tests/test_1.py::test_13_passes_and_has_stdout │ 946.47 ms │ passed │
│ demo_tests/test_1.py::test_14_causes_error_pass_stderr_stdout_stdlog │ 1.637 ms │ passed │
│ demo_tests/test_1.py::test_15_causes_error_fail_stderr_stdout_stdlog │ 2.3451 ms │ passed │
│ demo_tests/test_1.py::test_16_fail_compare_dicts_for_pytest_icdiff │ 12.027 ms │ failed │
│ demo_tests/test_random_results.py::test_1 │ 12.407 ms │ failed │
│ demo_tests/test_random_results.py::test_0 │ 18.41 ms │ failed │
│ demo_tests/test_random_results.py::test_10 │ 4.5941 ms │ passed │
│ demo_tests/test_warnings.py::test_1_fails_with_warnings │ 2.4029 s │ failed │
└─────────────────────────────────────────────────────────────────────────────────────────────────────┴───────────┴─────────┘versus
======================================= test session starts ========================================
platform darwin -- Python 3.10.5, pytest-7.2.2, pluggy-1.0.0 -- /Users/jwr003/coding/pytest-tally/venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/jwr003/coding/pytest-tally
plugins: Faker-17.6.0, tally-0.1.0
collected 82 items / 38 deselected / 44 selected
demo_tests/test_0.py::test0_pass_1 FAILED [ 2%]
demo_tests/test_0.py::test0_pass_2_logs PASSED [ 4%]
demo_tests/test_0.py::test0_pass_3_error_in_fixture ERROR [ 6%]
demo_tests/test_0.py::test0_fail_1 FAILED [ 9%]
demo_tests/test_0.py::test0_skip PASSED [ 11%]
demo_tests/test_0.py::test0_xfail XFAIL [ 13%]
demo_tests/test_0.py::test0_xpass XPASS [ 15%]
demo_tests/test_0.py::test0_warning PASSED [ 18%]
demo_tests/test_0.py::test_flaky_3 PASSED [ 20%]
demo_tests/test_1.py::test_a_ok PASSED [ 22%]
demo_tests/test_1.py::test_b_fail FAILED [ 25%]
demo_tests/test_1.py::test_c_error ERROR [ 27%]
demo_tests/test_1.py::test_d1_skip_inline SKIPPED (Skipping this test with inline call t...) [ 29%]
demo_tests/test_1.py::test_d2_skip PASSED [ 31%]
demo_tests/test_1.py::test_d3_skip_decorator SKIPPED (Skipping this test with inline cal...) [ 34%]
demo_tests/test_1.py::test_e1_xfail_by_inline_and_has_reason XFAIL (Marked as Xfail with...) [ 36%]
demo_tests/test_1.py::test_e2_xfail_by_decorator_and_has_reason XFAIL (Marked as Xfail w...) [ 38%]
demo_tests/test_1.py::test_f1_xfails_by_inline_even_though_assertTrue_happens_before_pytestDotXfail XFAIL [ 40%]
demo_tests/test_1.py::test_f2_xpass_by_xfail_decorator_and_has_reason XPASS (Marked as X...) [ 43%]
demo_tests/test_1.py::test_g_eval_parameterized[3+5-8] PASSED [ 45%]
demo_tests/test_1.py::test_g_eval_parameterized[2+4-6] PASSED [ 47%]
demo_tests/test_1.py::test_g_eval_parameterized[6*9-42] FAILED [ 50%]
demo_tests/test_1.py::test_1_passes_and_has_logging_output PASSED [ 52%]
demo_tests/test_1.py::test_2_fails_and_has_logging_output FAILED [ 54%]
demo_tests/test_1.py::test_3_fails FAILED [ 56%]
demo_tests/test_1.py::test_4_passes PASSED [ 59%]
demo_tests/test_1.py::test_5_marked_SKIP SKIPPED (unconditional skip) [ 61%]
demo_tests/test_1.py::test_6_marked_xfail_by_decorator_but_passes_and_has_no_reason XPASS [ 63%]
demo_tests/test_1.py::test_7_marked_xfail_by_decorator_and_fails_and_has_no_reason XFAIL [ 65%]
demo_tests/test_1.py::test_8_causes_a_warning FAILED [ 68%]
demo_tests/test_1.py::test_9_lorem_fails FAILED [ 70%]
demo_tests/test_1.py::test_10_fail_capturing FAIL stdout not captured, going directly to sys.stdout
FAIL stderr not captured, going directly to sys.stderr
FAILED [ 72%]
demo_tests/test_1.py::test_10b_failed_capturing FAILED stdout not captured, going directly to sys.stdout
FAILED stderr not captured, going directly to sys.stderr
FAILED [ 75%]
demo_tests/test_1.py::test_11_pass_capturing FAILED [ 77%]
demo_tests/test_1.py::test_12_fails_and_has_stdout FAILED [ 79%]
demo_tests/test_1.py::test_13_passes_and_has_stdout PASSED [ 81%]
demo_tests/test_1.py::test_14_causes_error_pass_stderr_stdout_stdlog ERROR [ 84%]
demo_tests/test_1.py::test_15_causes_error_fail_stderr_stdout_stdlog ERROR [ 86%]
demo_tests/test_1.py::test_16_fail_compare_dicts_for_pytest_icdiff FAILED [ 88%]
demo_tests/test_hoefling.py::test_1 FAILED [ 90%]
demo_tests/test_random_results.py::test_0 PASSED [ 93%]
demo_tests/test_random_results.py::test_1 SKIPPED (Skipped: Serious inside else memory i...) [ 95%]
demo_tests/test_random_results.py::test_10 FAILED [ 97%]
demo_tests/test_warnings.py::test_1_fails_with_warnings FAILED [100%]
Need to fix internal outcome classification so they match what Pytest shows in the
== short test summary ==section.