serenity-bdd / serenity-core

Serenity BDD is a test automation library designed to make writing automated acceptance tests easier, and more fun.
http://serenity-bdd.info
Other
722 stars 518 forks source link

Regression in TestOutcome.addTag(TestTag tag) method: test are not included in final report when running them using jUnit4 and -Dtags parameter #3531

Open EgorIu opened 2 months ago

EgorIu commented 2 months ago

What happened?

In versions prior to 3.9.x , the TestOutcome.addTag(TestTag tag) method exhibited a specific behavior where it invoked the getTags() method when the tags field was uninitialized.

    public void addTag(TestTag tag) {
        Set<TestTag> updatedTags = new HashSet<>(getTags());
        updatedTags.add(tag);
        this.tags = updatedTags;
        this.allTags = addFeatureTagTo(this.tags);
    }

This behavior was crucial as getTags() was used to enrich the TestOutcome object with tags annotated by @WithTags. This enrichment process was essential for ensuring that tests tagged through the -Dtags parameter during the test execution were correctly included in the final test report.

However, starting from version 3.9.x addTag() internal logic's been changed. The addTag(TestTag tag) method no longer triggers getTags() internally.

   public void addTag(TestTag tag) {
        Set<TestTag> updatedTags = (tags == null) ? new HashSet<>() : new HashSet<>(tags);
        updatedTags.add(tag);
        this.tags = updatedTags;
        //this.allTags = addFeatureTagTo(this.tags);
    }

Consequently, this change leads to the omission of tests in the report generation process as captured in the FreemarkerContext.getBuildContext() method, where "scenarios" and "allTestOutcomes" keys may not correctly reflect the intended tagged outcomes.

    public Map<String, Object> getBuildContext(TestOutcomes completeTestOutcomes,
                                               ReportNameProvider reportName,
                                               boolean useFiltering) {
        Map<String, Object> context = new HashMap<>();
        TestOutcomes testOutcomes = completeTestOutcomes.filteredByEnvironmentTags();
         ....
        List<ScenarioOutcome> scenarios = outcomeFilter.scenariosFilteredByTagIn(ScenarioOutcomes.from(testOutcomes, requirements));

This seems to be a regression from the prior behavior, potentially affecting any test executions relying on dynamically passed tags for reports.

What did you expect to happen?

The addTag(TestTag tag) should trigger getTags() when the tags field is null, ensuring all tagged tests are included in the report.

Serenity BDD version

All versions above 3.8.1

JDK version

17

Execution environment

Operation system: Windows Browsers: Edge, Chrome

How to reproduce the bug.

How can we make it happen?

Work on this myself and propose a PR (with Serenity BDD team guidance)