search

dutchiexl / BehatHtmlFormatterPlugin

This is a behat plugin to generate HTML reports
MIT License
112 stars 116 forks source link

Does not applicable when running a Parallel Test #76

Open magagan opened 7 years ago

magagan commented 7 years ago

This plugin only works for single run or session of the test. When I tried to run in parallel, the report is not consolidated.

Evgenas commented 7 years ago

same problem here, any plan to extend it for parallel run?

jboffel commented 7 years ago

I modified this plugin because I had the need to consolidate report after running tests in parallel on multiple servers. This is not trivial at all due to the way everything is implemented...

My approach has been to implement Serializable on all object used by BehatHTMLFormatter and to save their serialization in obj file so that I can merge them back using a tool I created before I call the render function for Twig as the initial code would have done. (You can't serialize right away due to some nested object from Behat framework that include Closure and are referred by this plugin)

This plugin make the assumption that all the test are run on one machine and in sequence. If you run some scenario of a feature on more than 1 server, you'll get the same feature reported more than one time in the report. Each of them only including the subset of scenario executed in this time.

Now let's assume also that like me you run the test not by suite or feature but by scenario, so I make a file containing direct scenario information file:line and I execute them in a shuffled order. Then in that case even in one test report generation from one server you can already get the same feature more than once.

This happen because the implementation tend to be very generic and does not try to identify a feature or a scenario with a unique id but rather pile them as they show up in arrays which are blindly served to the renderer after.

So you have to merge within a report and across reports. Then I also significantly modified the BehatHtmlFormatter class to make that easier. However my code is not generic at all and only fit my need so that I won't post it as a PR here, it would take extra effort to make it worth it.

Even you do all of that you will also get the scenarios in the wrong order per feature so you would also have sort them after the merge to get back your report as you expect it would be.

But just to let you know that yeah it's possible but the plugin architecture from the base is not really thought to make the parallel execution easy to support. So maybe it would be better to rewrite completely the BehatHtmlFormatter class to make that easier.

PurHur commented 5 years ago

@jboffel did you make any progess since then? Since im doing the same now i would do it the hacky way but just wanna know if you find something better?

donkidd commented 5 years ago

@PurHur Looks like @jhenya might have this fixed. (https://github.com/jhenya/BehatHtmlFormatterPlugin) He mentioned fixing something in #47 and his fork appears to handle Parallelization, maybe we should look at seeing if we can merge in his changes.

PurHur commented 5 years ago

Oh he made it public. Probably i will try to merge it after the first 2015 -> 2019 merges done. It isnt the best solution but to get it started. The printer itself has to be reworked no matter how this is going.