box / flaky

Plugin for nose or pytest that automatically reruns flaky tests.
Apache License 2.0
377 stars 60 forks source link

Support to capture "===Flaky Test Report===" content in order to dump it in multiple log streams #174

Open tarunmudgal opened 3 years ago

tarunmudgal commented 3 years ago

Currently there is no way to read ===Flaky Test Report=== details through any hook or fixture. There is a need to read this test report and write it to different loggers e.g. log files, error files

===Flaky Test Report===
test_example1 passed 1 out of the required 1 times. Success!
test_example2 failed (1 runs remaining out of 2).
        <class 'AssertionError'>
        assert 0 == 1
  +0
  -1
        [<TracebackEntry /Users/mtarun/tests/test_dummy.py:51>]
test_example2 failed; it passed 0 out of the required 1 times.
        <class 'AssertionError'>
        assert 0 == 1
  +0
  -1
        [<TracebackEntry /Users/mtarun/tests/test_dummy.py:51>]

===End Flaky Test Report===
adiroiban commented 3 years ago

It looks like nose is loading the tests before loading the plugins... so you can monkey patch the code


def patch_add_flaky_report(self, stream):
    """
    Patch flaky reporting to also write the output to a file.
    """
    original_add_flaky_report(self, stream)
    with open('flaky-report.txt', 'w') as output:
        original_add_flaky_report(self, output)

original_add_flaky_report = FlakyPlugin._add_flaky_report
FlakyPlugin._add_flaky_report = patch_add_flaky_report

Maybe add a `--flaky-report-file=path/to/file configuration