How come TestNG listeners are global?

[danielgil82]

Expected behavior

Whenever one doesn't add listener on class level, a listener still gets triggered.

Test case sample

Please, share the test case (as small as possible) which shows the issue

I'm adding this as following talk with @krmahadevan. https://stackoverflow.com/questions/78287583/why-custom-listener-is-called-when-i-dont-attach-it-to-testclass-not-via-annot

In concise I have a logging listener, that I added to one of my test classes. And didn't conclude this listener to the second test class. My LoggerListener implements ITestListener, IClassListener. The thing is that the second test class enters onBeforeClass and its onAfterClass, now It's surprising because, i'm asking my self what's the purpose of @Listeners then, In ITClassTestB there is no LoggerListener on the class level.. and it still enters where I said .

I'll add snippets of code, so you could see how to reproduce..

public class LoggerExtensionListener implements ITestListener, IClassListener {                                       
    private final Map<String, List<TestResultStatus>> testResultsStatusPerClass = new ConcurrentHashMap<>();          

    private enum TestResultStatus {                                                                                   
        SUCCESSFUL, FAILED, TIMED_OUT, SKIPPED;                                                                       
    }                                                                                                                 

    @Override                                                                                                         
    public void onBeforeClass(ITestClass testClass) {                                                                 
        Class<?> testRealClass = testClass.getRealClass();                                                            
        String testClassName = testRealClass.getSimpleName();                                                         

        initLogFile(testClassName);                                                                                   
    }                                                                                                                 

    @Override                                                                                                         
    public void onAfterClass(ITestClass testClass) {                                                                  
        String className = testClass.getRealClass().getSimpleName();                                                  
        .
        .
     }
   }

@Listeners({LoggerListener.class})
public class ITClassTestA extends BaseTest {

    @Test
    public void test1InClassTestA() throws InterruptedException {
        logger.info("Started test 1 in ClassTest A: " + TimeUtils.nowUTC());
        Thread.sleep(TimeUnit.SECONDS.toMillis(4));
        logTheName("SDK");
        logger.info("Ended test 1 in ClassTest A: " + TimeUtils.nowUTC());
    }

    @Test
    public void test2InClassTestA() throws InterruptedException {
        logger.info("Started test 2 in ClassTest A: " + TimeUtils.nowUTC());

        Thread.sleep(TimeUnit.SECONDS.toMillis(4));

        logger.info("Ended test 2 in ClassTest A: " + TimeUtils.nowUTC());
    }
}

public class ITClassTestB extends BaseTest {

    @Test
    public void test1InClassTestB() throws InterruptedException {
        logger.info("Started test 1 in ClassTest B at: " + TimeUtils.nowUTC());
        Thread.sleep(TimeUnit.SECONDS.toMillis(4));
        logTheName("testNG");
        logger.info("Ended test 1 In ClassTest B at: " + TimeUtils.nowUTC());
    }

    @Test
    public void test2InClassTestB() throws InterruptedException {
        logger.info("Started test 2 In ClassTest B at: " + TimeUtils.nowUTC());

        Thread.sleep(TimeUnit.SECONDS.toMillis(4));

        logger.info("Ended test 2 In ClassTest B at: " + TimeUtils.nowUTC());
    }
}

Xml file:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd">
<suite name="Suite A">
    <test name="FirstSuiteFirstClassTest" >
        <classes>
            <class name="ITClassTestA"/>
        </classes>
    </test>
    <test name="FirstSuiteSecondClassTest" >
        <classes>
            <class name="ITClassTestB"/>
        </classes>
    </test>
</suite>

Contribution guidelines

[krmahadevan]

@danielgil82 - Thanks for adding up this issue here.

The current behaviour of TestNG is that, all listeners are treated as "global" and we don't have the notion of "scoped" listeners in TestNG yet.

When I say "scoped" I mean something like below:

• class-level - Listener should be run only for the given class in question (or) its child classes.
• test-level - Listener should be run only if a particular test class belongs to a specified <test> tag in question.
• suite-level - Listener should be run only if a particular test class belongs to a specified <suite> tag in question (This becomes relevant when you create a suite of suites and then run them.

The other thing we need to also consider is how do I allow a user to specify this scope for me (Without breaking backward compatibility of-course). Some points to ponder would include the following

• On listeners, we can perhaps add it as an optional attribute to the @Listeners annotation.
• How to do this, when a listener is being wired in via the command line argument (or) via the <listeners> tag (or) via service loaders.
• Should scope be honoured for listeners that are NOT wired in via the @Listeners annotation.

Please share your expectations around these as comments so that the scope of this issue can be made clear.

[juherr]

i'm asking my self what's the purpose of @listeners then

@Listeners is currently the only way to register listeners by code when you use an external runner like maven or gradle. In the same way, it is not possible at all to declare a suite by code and drop the XML suite file. I agree the behavior is not the best but it is currently kept for historical reasons.

[danielgil82]

I think it's better suppose to be called per TestClass, I mean if a class wants to trigger a listener, then let it trigger the listener, now if a derived class wants to trigger the same listener, well he should too add the listener to the @Listeners section.

That sounds good, and also, it could be better if you could add a disable flag on the listener level, For example like DisableListener for a specific listener, actually I've noticed that flag inside the docs though it doesn't work. e.g :

@DisableListener
@Listeners({ LoggerListener.class})
public class ITClassTestB extends BaseTest {
    @Test
    public void test1InClassTestB() throws InterruptedException {
        logger.info("Started test 1 in ClassTest B at: " + TimeUtils.nowUTC());
        logTheName("testNG");
        logger.info("Ended test 1 In ClassTest B at: " + TimeUtils.nowUTC());
    }

    @Test
    public void test2InClassTestB() throws InterruptedException {
        logger.info("Started test 2 In ClassTest B at: " + TimeUtils.nowUTC())
        logger.info("Ended test 2 In ClassTest B at: " + TimeUtils.nowUTC());
    }