Merged report with multiple suites

Codeception / robo-paracept

Robo tasks for Codeception tests parallel execution

MIT License

57 stars 59 forks source link

Merged report with multiple suites #69

Open ccsuperstar opened 3 years ago

ccsuperstar commented 3 years ago

When I run my test runs in parallel, the merged html report does not contain all the suite titles. I think this is because the distribution of tests is done in such a way that each /_data/parallel_* file does not always contain tests from each suite.

Also, the execution time in the final report is not the maximum execution time.

Before v1, I had made a modification, with a report template file which match with our need, but I'm looking forward to your ideas!

vansari commented 3 years ago

Hi @ccsuperstar

This is the class where I did not have enough time for a UnitTest. But as you can see it is very useful to have a test which tests the expected result.

If you can provide some html test results it would be very helpful.

If not it's also okay 😉

ccsuperstar commented 3 years ago

If you use for example these 3 html reports (report_1.html, report_2.html, report_3.html), and you run the merge report command, as it is the first html report that is used to define the suite titles, you will have only the title Acceptance in report.html file reports.zip

While in the other reports, you have Api, Bdd, etc

So, either you have to use a template, or you have to parse all the reports to have all the suites

vansari commented 3 years ago

hi @ccsuperstar

is this the expected report what you want? (Its manually changed for a preview)

ccsuperstar commented 3 years ago

Yes it's the expected report!

reinholdfuereder commented 2 years ago

I think I might suffer under the same (or very similar) problem:

Background:
- First there is a "serial" Codeception test execution with (a) all (fast) PhpBrowser tests and (b) some (non-parallel-working) WebDriver tests (because they temporarily manipulate something in the DB -- yes, very ugly, but it is as it is; pragmatic colleagues...)
- Then there are a few "parallel" sets with parallel exeuction of slow WebDriver tests
Result:
- Due to starting with the "serial" HTML report all the "parallel" HTML report results are merged to the first and thus wrong section/suite (but the second suite is there too, because it is part of the first "serial" HTML report too)
- Screenshot:
- Files: merged-html-original-with-bug.zip
Expected result:
- Screenshot (sorry for unfortunately having used a slightly different test set with 2 skipped tests and obviously a new run with different timings):

reinholdfuereder commented 2 years ago

@vansari My PR https://github.com/Codeception/robo-paracept/pull/105 fixes my problem and I think this issue here => thanks in advance for reviewing and merging

vansari commented 1 year ago

@reinholdfuereder Is it possible that you add a test for your changes?