Before/After methods are not running when groups "include" in suite

chetand24 commented 6 years ago

TestNG Version : 6.12.0, 6.11.0

Note: only the latest version is supported

Expected behavior

The Before and After methods should run when "include" tag is present in the testng suite

Actual behavior

The Before and After methods are not running when "include" tag present in the testng suite. But is working fine when "exclude" is present

Is the issue reproductible on runner?

[ ] Shell
[ ] Maven
[* ] Gradle
[ ] Ant
[ *] Eclipse
[ *] IntelliJ
[ ] NetBeans

Test case sample

Please, share the test case (as small as possible) which shows the issue

The test class:

public class AnotherTest {

    @BeforeSuite
    public void setupSuite(){
        System.out.println("Before Suite Setup ...... ");
    }

    @BeforeTest
    public void setUpTest() throws Exception {
        System.out.println("Before Test Setup ...... ");
    }

    @BeforeMethod
    public void setUpMethod() throws URISyntaxException {
        System.out.println("Before Method Setup ...... ");
    }

    @AfterMethod
    public void tearDownMethod(){
        System.out.println("After Method Setup ...... ");
    }

    @AfterTest
    public void tearDownSuite() throws Exception {
        System.out.println("After test Setup ...... ");
    }

    @Test(groups = {"sometest"})
    public void someTest(){
        System.out.println("Some test ...... ");
    }

    @Test(groups = {"anothertest"})
    public void anothertest(){
        System.out.println("Another test ...... ");
    }
}

Example XML Suite

<suite name="Mobile Test" verbose="10">
    <test name="Some Tests">
        <groups>
            <run>
                <!--<exclude name="sometest"/>-->
                <include name="sometest"/>
            </run>
        </groups>
        <classes>
            <class name="com.practice.AnotherTest"/>
        </classes>
    </test>
</suite>

Output when included the group:

...
... TestNG 6.11 by Cédric Beust (cedric@beust.com)
...
Some test ...... 
===== Invoked methods
    AnotherTest.someTest()[pri:0, instance:com.practice.AnotherTest@7cf10a6f] 2096171631
=====
===============================================
Mobile Automated Test for Engage Android
Total tests run: 1, Failures: 0, Skips: 0
===============================================

Output when excluded the group:

...
... TestNG 6.11 by Cédric Beust (cedric@beust.com)
...
Before Suite Setup ...... 
Before Test Setup ...... 
Before Method Setup ...... 
Another test ...... 
After Method Setup ...... 
After test Setup ...... 
===== Invoked methods
  AnotherTest.setupSuite()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.setUpTest()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.setUpMethod()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
    AnotherTest.anothertest()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.tearDownMethod()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
  AnotherTest.tearDownSuite()[pri:0, instance:com.practice.AnotherTest@5ba23b66] 1537358694
=====

===============================================
Mobile Automated Test for Engage Android
Total tests run: 1, Failures: 0, Skips: 0
===============================================

krmahadevan commented 6 years ago

@chetand24 - There are two parts to your issue.

The Before and After methods are not running when "include" tag present in the testng suite.

By default if you would like your BeforeXXX and AfterXXX to be executed all the time, irrespective of what group is being executed, you should be using setting : alwaysRun=true

But is working fine when "exclude" is present

This part is a bug. I have fixed this, by raising a pull request ( https://github.com/cbeust/testng/pull/1575 )

juherr commented 6 years ago

@krmahadevan I'm not sure if it is a bug. The documentation doesn't specify it, but we can imagine that no group means the "default group".

So, include "x" won't add "default group" but exclude "x" won't remove "default group".

@cbeust Could you clarify the expected behavior of "exclude" for methods without a group?

juherr commented 6 years ago

In fact, if we remove "no group" methods with "exclude", I don't know how to run "no group" and some other groups methods.

krmahadevan commented 6 years ago

@juherr - You are right in terms of the behavior not defined properly.

Here's my understanding :

When a user specifies run by groups, then one explicitly likes to run one or more groups.
A test/config method without any groups, means its explicitly meant NOT to be run via the groups selection mechanism.
There is no concept of default/default group groups (similar to how we have a default package when we don't specify the package name for a class) in TestNG.
Currently no group (or) default group with include doesn't work any way because a user cannot combine methods with groups specified and methods with no groups specified together in the single execution via groups mechanism. Not at-least with the @Test methods. So the behavior should be consistent for exclude as well.

kool79 commented 6 years ago

@krmahadevan, your PR will broke backward compatibility. In my projects I use groups feature only to exclude tests (flacky, app-bug, time-consumind, etc). So I mark only those tests which I want to exclude. After your PR all my tests without groups will be excluded from run. I'll be forced to assign some groups to ALL methods when I want to exclude just 1 test-case. This is not obvious (I need to assign ANY group to method if I want to run it with exclusions activated) I'll try to rectify your understanding:

When you need to run only certain tests/methods, define group(s) for them
When you need to exclude certain tests/methods, define group(s) for them

About

Currently no group (or) default group with include doesn't work any way because a user cannot combine methods with groups specified and methods with no groups specified together in the single execution via groups mechanism.

Test with no-group belongs to virtual 'all-tests' group (as same as tests with groups). No need to include something into this group because it already contains all stuffs. But if you still want to combine (no-group) and some-group then probably you just need one more group like common-tests for such tests. P.S. Of cause, you always can write custom filter to override default behavior of XmlMethodSelector

krmahadevan commented 6 years ago

@kool79 - Thanks for your inputs. The only reason why this PR is pending merge and will perhaps never be merged is because of backward compatibility. The current behavior IMHO is not intuitive. Methods which dont belong to a group, mean they dont belong to a group. There cannot be a virtual all-tests group to which they get automagically added.

But debates apart, backward compatibility is more important than anything else. So this PR will not be merged.

javydreamercsw commented 4 years ago

It almost begs the question if alwaysRun should default to true and be disabled if the method has a group.

testng-team / testng