thread-count as it relates to background threads spawned in beforeGroups and elsewhere

[borisivan]

TestNG Version

6.12

Expected behavior

I want to define the number of tests that run in parallel, and have that number of tests run. Would also like the thread name to follow a consistent format.

Actual behavior

I'm seeing a smaller number run. Also, it seems that one test runs under a threadname of "testNG-method=", but the other tests running in parallel run under a different format of a threadname -- not the same format as described above (will update this report with copy & paste once I can get a chance to run again).

Is the issue reproductible on runner?

• [X] Maven
• [X] Eclipse

My suite is complex and involves background threads I spawn via executor from within the beforeGroups method. That background task lives throughout that entire duration of tests, and only gets stopped in the afterGroups method.

In addition, there are also background threads spawned by a shell reader as it executes some commands and reads from stdin/stderr. That might occur in the beforeSuite, beforeGroup, or individual test methods, and those threads come and go.

My question is: does any of that relate to the number of threads I specify in the suite xml? I'd like to run 6 tests in parallel, and I will not be able to know ahead of time how many background threads my suite might spawn at the various times described above. My suite file is based on parallel=tests, in which I have 6 test clauses for the parallel groups in the xml that each have a number of classes that all have multiple test methods, i.e:

    <suite name="theSuite" verbose="1" parallel="tests" thread-count="??">
        <test name="Parallel Group 1">
             classes....
        </test>
        <test name="Parallel Group 2">
             different classes....
        </test>
        ...
    </suite>

With what I described above re: long-living background threads being spawned in the beforeSuite, and beforeGroups (and some short lived ones inside the test methods) -- should my 'thread-count" be "6"? Or do I need to account for any of those background threads I create when making the count?

[borisivan]

Not sure if this is related, but here's something I observe:

If I have parallel=classes, thread-count 6, and all the classes listed in one <test> in the suite.xml, my tests get sleep interrupted all over the place (6 attempt to run in parallel, but fail re: sleep interrupted).

If I have the suite.xml defined as the original report shows, re: 6 <test> sections each with 5-6 classes, and at the suite definition I define parallel=tests, with a thread-count of 50 (I don't know why I chose that number), I do not get any sleep interrupted in the 6 parallel tests.

The 6 parallel individual running tests do not spawn any background threads, but the @beforeGroup and @beforeSuite code does.

I suspect if I define parallel = tests (as originally described above) and thread-count = 6, I'll probably see the same thing re: sleep interrupted.

This is why I'm asking re: if the executor background threads spawned by the @beforeGroups or @beforeSuite code count towards the threadcount limit I specify.

In the long run, all I want to do is run no more than 6 tests in parallel, but don't want testNG to worry / enforce whether or not I've spawned a background thread at some point in the @beforeSuite or @beforeGroups.

[borisivan]

Hoping someone can explain the correct behavior here. Having a bear of a time getting tests to run in parallel. Our background threads spawned in the @beforeGroup code really seem to count to the thread count re: parallel tests. But I don't want to limit threads, I want to limit parallel tests, which aren't necessarily the same thing. And I can't predict the exact qty of background threads I might spawn with the complex object that I instantiate in the @beforeGroup code.

[krmahadevan]

@borisivan - Would it be possible for you to create a standalone project (either a maven based or gradle based) which we can use to see the behavior that you are talking about ? That would add a bit more clarity to your question and help us suggest what could be done.

[krmahadevan]

does any of that relate to the number of threads I specify in the suite xml

AFAIK, TestNG keeps track of only those threads that it spawns. That thread pool size is governed by the below two attributes :

• thread-count=""
• data-provider-thread-count=""

Any other threads that you are spinning off via any of the TestNG listeners or TestNG configuration methods or TestNG test methods are basically user governed. That's my take on the behavior. @juherr please correct me if I am wrong here.

[borisivan]

Hi @krmahadevan -- that's an interesting point. When I run with parallel=tests, and have 6 <test> in the XML and multiple classes in each test running in sequence, the thread utilized per test is unique and separated from the group threads that processed the @beforeGroup or @beforeMethod work, (as long as I have timeOut=someAmountOfTime on each @Test method).

But when I run parallel=classes threadcount=6 in one large <test> in the xml, the tests run in the 'group' thread (i.e. group 1-6), instead of a unique thread per test that is different from the group thread. And therefore, since the background threads I spawned in the @beforeGroup run in the group thread, that's why I'm getting sleep interrupted in my individual @Tests...

Thinking about that for a bit. If I can verify that the only tests that are interrupted are the ones in the same group thread that spawned a background thread, at @beforeGroup time, then I think we're on to something. Some of that is made more complex in that I've had to semaphore protect the @beforeGroup code to ensure it doesn't run in each group thread, due to the issue you have a PR pending for... ;)

[krmahadevan]

Using the attribute timeout basically causes TestNG to use a separate executor service itself. That perhaps explains why you see a different group name.

Without this attribute, TestNG would re-use any of the existing threads that just now was freed up after running a configuration method.

[krmahadevan]

@borisivan - Oh btw, my fixes have been merged to master. So you should be able to consume TestNG 6.15.0-SNAPSHOT to validate it.

I re-iterate again. It would be really good if you could please help create a simple standalone project that can be used to recreate the issue. Its easier to visualize the problem when there's code that demonstrates it.

[borisivan]

Thanks @krmahadevan, I'll give it a shot, and I'll see if I can obfuscate enough to make a standalone project to host here for examination..

[borisivan]

Hi @krmahadevan, I added the sonatype snapshot repo, and added 6.15.0-SNAPSHOT as a dependency.

Unfortunately I'm getting compile errors by making that change only:

public class TestBaseClass extends AbstractTestNGSpringContextTests {

AbstractTestNGSpringContextTests shows an error re:

The type org.testng.IHookable cannot be resolved. It is indirectly referenced from required .class files

But with 6.14.2 and earlier, it's OK.

Should I log a separate issue for that? Not sure what the team would prefer, given that 6.15.0-SNAPSHOT isn't really released.

[krmahadevan]

@borisivan - TestNG went through with a release recently. So I think you should be able to work with 6.14.3.

Oh btw, org.testng.IHookable still exists in TestNG and hasn't been altered. So there's no reason why you should see a compilation error for that. But I would suggest that you file a separate issue for that so that it can be investigated independently.

[borisivan]

Thanks @krmahadevan. Regarding #1694, I just ran with 6.14.3, and see the same problem, re: @beforeGroups being executed multiple times instead of just once. I made a sample project here on github to demonstrate.

In this project, if you run the suite file src/main/testSuites/suite.xml, you should receive the following output:

[RemoteTestNG] detected TestNG version 6.14.3
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-1] -----@beforeClass for test group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-4] -----@beforeClass for test group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-4] -----@beforeClass for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-3] -----@beforeClass for test group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-3] -----@beforeClass for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-5] -----@beforeClass for test group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-5] -----@beforeClass for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-6] -----@beforeClass for test group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-6] -----@beforeClass for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-2] -----@beforeClass for test group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-2] -----@beforeClass for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-1] -----@beforeClass for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-1] -----@beforeGroups for group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-1] -----@beforeGroups for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-4] -----@beforeGroups for group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-4] -----@beforeGroups for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-5] -----@beforeGroups for group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-5] -----@beforeGroups for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-6] -----@beforeGroups for group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-6] -----@beforeGroups for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-2] -----@beforeGroups for group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-2] -----@beforeGroups for test group group1 end-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-3] -----@beforeGroups for group group1 begin-----
[22:30:59 INFO  GroupOne                      ] [TestNG-tests-3] -----@beforeGroups for test group group1 end-----
(truncated).

The .xml file has 6 tests re: parallel=tests, which each have one class (in my real project each of the 6 tests have 10-11 classes), they are all in the same group. But still the @beforeGroup method is run 6 times, not once (as shown in the output above).

I know that's only indirectly related to this issue here. Wasn't sure what the right process was to follow re: 1694, re: reopening it or logging a new issue, etc.

[krmahadevan]

The .xml file has 6 tests re: parallel=tests, which each have one class (in my real project each of the 6 tests have 10-11 classes), they are all in the same group. But still the @beforeGroup method is run 6 times, not once (as shown in the output above).

AFAIK, @BeforeGroups is supposed to be running once per <test> because <test> represents the smallest unit of execution. That perhaps explains why you are seeing that many number of occurrences. If you would like a setup to be executed once per <suite> then you should be adding it as part of @BeforeSuite and include the method in the group for which a one time execution is needed.

ping @cbeust @juherr - Please correct me if I am wrong

[borisivan]

Hi @krmahadevan, if I modify that suite file to have all the classes in one <test>, it is true that the @beforeGroup method only runs once total. Still seems odd -- why does your explanation not apply to @beforeSuite -- i.e. why does that only run once (not that I'd want it to run more), but yet @beforeGroup executes multiple times? I think the natural assumption (certainly for me) would be that the @beforeSuite code runs once before the suite regardless how many groups or tests exist, and the @beforeGroup code executes once before any of the tests in that group, regardless how many classes say they're part of that group (or how they're organized in the suite file).

My goal is to simply have 6 parallel executions, but each of those 6 have classes that run in sequence. If in one <test>, I can't organize into 6 parallel execution groups, but in 6 parallel execution groups via 6 <test>, the @beforeGroup behavior changes such that I have to semaphore protect it, and force the other 5 groups to wait until it's really complete, then return before executing. I can't execute that code multiple times, nor can I have the other groups not executing that code begin their tests until it was processed once.

re: the @beforeSuite being specific per-group.. hadn't thought of that. Honestly, I've never known what the supported behavior is for having multiple methods annotated with something like@beforeSuite -- I had assumed that it was only supported to exist once. Is it possible to have more than one method with @beforeSuite annotations, one global to be run once total regardless which group the test(s) may or may not be in, and one intended just for a particular group, and they are executed in that sequence?

[borisivan]

So close!

In my group class, (which extends baseClass), I changed the method with the @beforeGroup annotation to @beforeSuite annotation with a groups=param. It runs, and only runs once... but no Spring services initialize. Looks like testNG / Spring integration somehow initializes autowired spring objects only after it has gone through the @beforeSuite methods.

Specifically, below shows the spring autowired initialization. These are objects at the class level in my group class, and are successful when I have a @beforeGroup annotation on a method in that class: (successful output below)

[12:51:33 INFO  TestBaseClass                 ] [main] -----@beforeSuite end-----
[12:51:33 INFO  TestBaseClass                 ] [main] This is the beforeTEST method
[12:51:33 INFO  RemoteTestNG                  ] [main] Starting RemoteTestNG on Windows10 with PID 7744 (C:\Users\Administrator\eclipse\java-oxygen\eclipse\configuration\org.eclipse.osgi\859\0\.cp\lib\testng-remote.jar started by Administrator in C:\Users\Administrator\git\obfuscated)
[12:51:33 DEBUG RemoteTestNG                  ] [main] Running with Spring Boot v1.3.1.RELEASE, Spring v4.2.4.RELEASE
[12:51:33 INFO  RemoteTestNG                  ] [main] No active profile set, falling back to default profiles: default
[12:51:34 INFO  Version                       ] [pool-2-thread-1] HV000001: Hibernate Validator 5.2.2.Final
[12:51:34 INFO  RemoteTestNG                  ] [main] Started RemoteTestNG in 1.126 seconds (JVM running for 48.097)

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::        (v1.3.1.RELEASE)
various beforeGroup processing here. The Spring initialization banner above is simply due to the presence of those autowired spring objects at the class level in this class.

But when I change the annotation from @beforeSuite to @beforeGroup in that method in the group class (that's the only change I make...)... the autowired objects to not instantiate. The method with the annotation runs (and runs only once this time, perfect...), but spring objects are not accessible.

[krmahadevan]

@borisivan - @BeforeSuite is built to execute only once per <suite>

@BeforeGroups now correctly runs once per <test> when you are filtering tests based on groups. And since <test> is the smallest unit of execution it is going to be invoked once per <test>. So I guess that should clarify the behavior.

Is it possible to have more than one method with @beforeSuite annotations, one global to be run once total regardless which group the test(s) may or may not be in, and one intended just for a particular group, and they are executed in that sequence?

Yes you should be able to have more than one @BeforeSuite methods in your code and have them run. To get one running globally irrespective of what group was chosen to run dont forget to enable the attribute alwaysRun=true

and they are executed in that sequence

That I am not sure. I dont think there's any ordering amidst @BeforeSuite methods.

[krmahadevan]

@borisivan - I think we are discussing too many things in this issue. All discussions in this issue now are basically hijacking the original issue that you logged in this thread.

[borisivan]

@krmahadevan, agreed.

For this issue here, I was looking to confirm that testNG's thread-count limit has no relation to the background threads spawned by my @beforeGroup (or now, @beforeSuite) code. In other words, if I want 6 @Test run in parallel, whether I do this via parallel=classes, or parallel=tests, I don't want my background thread counting against that limit, or interrupting the 6 threads dedicated to the @Tests.

The background threads spawned are only in the @beforeGroups (or now, @beforeSuite code), but they do live after those methods are completed.

[krmahadevan]

@borisivan - As i mentioned before, I don't think TestNG counts the background threads that your TestNG methods spun off. Those are user spun off threads and so it shouldn't count in the TestNG thread pool