Specflow Living Doc is not updating the existing results as expected.

[HarikrishnanVK]

SpecFlow Version

Specflow : 3.9.58, Specflow Living DOC : 3.9.57

Which test runner are you using?

MSTest

Test Runner Version Number

3.9.58

.NET Implementation

equal or greater .NET Framework 4.6.1

Project Format of the SpecFlow project

Classic project format using packages.config

.feature.cs files are generated using

SpecFlow.Tools.MsBuild.Generation NuGet package

Test Execution Method

Visual Studio Test Explorer

SpecFlow Section in app.config or content of specflow.json

{ "$schema": "https://specflow.org/specflow-config.json",

"<None": "Update=\"specflow.json\">", "Always": "",

"stepAssemblies": [ { "assembly": "Allure.SpecFlowPlugin" } ],

"livingDocGenerator": { "enabled": true, "filePath": "TestExecution{ProcessId}{ThreadId}_{Now}.json" } }

Issue Description

We usually use Allure report to publish the results. Since the allure report is confined to the local machine where the json files are present, it is tedious process in case of sharing the reports to others. Hence we decided to adopt Specflow Living Doc which is official reporting tool of Spec flow.

While using the Spec flow Living Doc, we noticed that it is not updating the result of the feature properly.

i.e, For instance, if we run 10 features in a batch, 2 or 3 features may fail at 1st run. After the end of 1st run, we re-run the failed feature either as batch or through VS test runner individually. Once the re-run for all failures completed where all flaky test gets passed eventually, we run the below command with wildcard expression to get the consolidated reports:

livingdoc test-assembly "%project path%\bin\Debug\projectname.dll" -t "bin\Debug\TestExecution_*.json"

To our surprise, specflow living did not update our existing failed features as passed and it still remains as failed. But if we run the report by specifying particular json file (one of the file generated for passed feature in bin/debug folder), report is generated correctly for that only feature as passed.

livingdoc test-assembly "%projectpath%\bin\Debug\projectname.dll" -"bin\Debug\TestExecution_18212_8_20220425051714.json"

This seems to be a headache for us to generate the report every time by specifying the TestExecution.json after every re-run, as the results are not updating for existing features when we use wildcard expression.

so currently we switched back to Allure report where the results are getting updated as expected.

Specflow living doc looks promising and simple. Hence we like to use that going forward instead of allure in case of fixing the above said issue which is really annoying for us.

Steps to Reproduce

1. Run 10 Test cases as group. (Let 3 or 4 fail in first run).
2. Re-run the failed cases (which will get pass, as it failed first time due to flakiness).
3. Generate the consolidated report in CLI using wildcard expression (TestExecution_*.json).
4. Now the report is not updating the initially failed tests as passed even though they passed in re-run.

Link to Repro Project

Its a private project.

[SabotageAndi]

I think there is a misunderstanding how LivingDoc works. You don't update it with an livingdoc call and give it all ever generated TestExecution-json-files. The ability to pass more than one TestExecution.json as parameter is there if you split your test execution across multiple machines/agents for parallel execution reasons.

We interpret all passed TestExecution.json files as from the same logical test run. And in this case if one of the execution of a scenario fails, we display the scenario as failed, regardless how many successful executions there are. This is the reason why your rerun of flaky scenarios doesn't change the living documentation.

[HarikrishnanVK]

@SabotageAndi Thanks for your clarification. Is there any other way I can update my existing results with livingdoc. This is crucial for us to update the existing results as we don't want to generate multiple reports for re-run.

[SabotageAndi]

You would need to do the merging logic yourself and create a TestExecution.json with the execution result you want. The schema is not that complicated.

[HarikrishnanVK]

I suppose that is the purpose of wildcard here as per below screenshot from official site. But its not displaying the latest results for existing features after merging .

I understand that currently using wildcard we consolidate all the results and not checking for any recent updates. I believe it is ideal to update the recent status of already executed features and just add the current status of the feature in case of 1st time execution (similar to behavior of allure).

[infinitejest69]

Just to Echo OP's request i would like to see this as a config option in the generation. i am am using azure pipelines with a vstest task that reruns failed tests. as a result the pipeline results dont marry up with the living doc and makes for misunderstanding when non technical users see the living doc but dont immediately understand. it passed on rerun. Having the little icon to show is not immediately obvious.

the desired behaviour would be to have a test result show passed maybe with an on rerun similar to azure results.

i know that's a lot of rerun failures it's due to ongoing bugs when doing parallel user testing. i want to highlight scenarios that are failing every time than from the flake

[ClarksonAdam]

I second @infinitejest69 's comment. I have tests showing as failed in the SpecFlow LivingDocs that passed on the re-run due to transient errors. However, to the wider team, it looks like a bigger issue than just a slightly flakey test. It would be better to see these as 'passed' but with a note of the runs that failed

[infinitejest69]

@ClarksonAdam not sure if its useful i wrote a very quick and dirty command line app that will remove these failures from the json and output a single file. its seems to work for my purposes but take it with a pinch of salt and hopefully someone smarter makes it faster and better or a plugin for specflow living doc. if its useful chuck me a star on github

https://github.com/infinitejest69/SpecFlowLivingDocFailureRemover

[ClarksonAdam]

Thank you, @infinitejest69. I'll take a look at this

[HarikrishnanVK]

@infinitejest69 Thanks for the efforts.

Can you plz tell how to use this -> https://github.com/infinitejest69/SpecFlowLivingDocFailureRemover

Could not find any instructions in repo.

[infinitejest69]

@HarikrishnanVK Added a basic readme. the code has comments on it with what i was trying to achieve. you can run if from visual studio and replace the inputs with some variables