Reports are being accumulated when pipeline is rerun

[dkvaltech]

When hitting rerun the cucumber reports are beeing accumulated. For example when your pipeline contains 10 tests and 1 fails you get a result like "Passed: 9 Failed: 1". If you hit rerun and. the test works again you get a result like "Passed: 19 Failed: 1". The pipeline is green of course, but the cucumber report still reflects the failueres.

I tried putting new reports in a directory containing the job id which is run specific but that doesn't change anything. What it would require is an option to purge previous reports from the cucumber report. When having multiple jobs that would mean of course that successful test jobs are not displayed in reruns, but still better than the current situation. Of course the perfect solution would be to divide the reports by job id, for example on reparate tabs.

[maciejmaciejewski]

I have released a new version of the plugin, so this might no longer be the case. Please check if the issue still persist and get back with your observations.

[dkvaltech]

Thanks for the new version, subsequent reruns are now split by separate tabs. What's still a bit confusing is the naming - not that clear which is the most current one by the name alone (I think the one on the right is always the current one), but this issue here is definately resolved 👍

[maciejmaciejewski]

@dkvaltech There is an optional name parameter you can set for each of the runs

    - task: PublishCucumberReport@1
      displayName: 'Publish Cucumber Result'
      inputs:
        jsonDir: subdirectories/java
        outputPath: cucumber
        metadata: |
          {
            "ApplicationUrl": "$(App.Url)"
          }
        name: 'Functional Tests 1.1'
        title: 'Playwright Web'

This will show as a name of the tab