Is parallel tests execution under consideration?

[TripleG]

Hi @adrianth ,

Are you considering to implement parallel tests execution? By "parallel execution" I mean real parallel thread execution - the way TestNG supports it for example. Splitting tests into multiple templates and then running multiple sessions is not very convenient.

[galegor]

@TripleG I belive your request is a duplication of https://github.com/mcdcorp/opentest/issues/58 @adrianth that test session orchestration logic change is any near on the horizon?

[adrianth]

I'm afraid this is not going to be available in the near future. TestNG is just a simple and straightforward testing library so parallelization is very easy to implement. Of course that comes at the cost of leaving the difficult aspects of test automation and parallelization in the responsibility of the test author: test data management, data-driven testing, distributed testing involving multiple machines and/or networks, etc. The tester also has to deal with avoiding conflicts resulting from parallelization (e.g. one test creates product 123 in order to use it in subsequent flows; a different parallel test tries to create this product again with different attributes; both tests will most likely fail). There are many little questions and concerns that arise when trying to properly implement a parallelization feature in a test automation framework that are not that easy to resolve.

In order to do parallelization with OpenTest, please create two different test session templates that include the two (or more) sets of tests you want to run in parallel. Then you can create tests sessions using all those templates. As long as you have enough test actors to deal with all those test sessions simultaneously, all sessions will run in parallel. In case you don't have enough test actors, the OpenTest server will start all the sessions that it can, then wait for more actors to become available. As soon as more test actors free up, the rest of the test sessions will also start executing. Let me know if this approach is not clear an I will elaborate.

[TripleG]

Hi Adrian,

Thanks for your clarification.

Here are my 2 cents:

1. Current "parallelism" in OpenTest is clear, but it is far from convenient. If I have 20 actors and I want to keep them busy, I would need to split all tests into 20 templates, which is hard for tracking and after that for analysing. Then if I upgrade infrastructure and I can run 50 actors I must go and again redistribute all tests into 50 templates. And then each time I would be forced to go look into 50 sessions for results analysis.

2. I don't see why you are concerned about the authors' problems If they want to implement parallelism in their tests. Of course it is their responsibility to implement their test data/test logic, so the tests could be executed in parallel without affecting each other. How do you expect to protect the authors from the possible logical mistakes from within the framework itself?

If current OpenTest's architecture allows inner parallelism - for example 1 session to create sub-sessions and occupy actors, but it would still appear as 1 session for easier analysis, then in my opinion we are good to go. You don't need to worry about potential authors incapability of managing parallel testing - this is their problem, not framework's, right? Also, if they cannot manage it, then they would just not use it - it is their choice.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[AdarshBarbate]

Hi @adrianth,

According to your this comment "In order to do parallelization with OpenTest, please create two different test session templates that include the two (or more) sets of tests you want to run in parallel. Then you can create tests sessions using all those templates....".

I can execute test sessions parallelly on my local machine. But is it possible to execute multiple sessions parallelly on continuous delivery(CICD) pipeline as well ?

[adrianth]

There is not much difference between running on the local machine and running in CI. To run test session in parallel on a CI server you can use the server's parallelization support to kick off two different test sessions. Of course, since this will run unattended you'll use the OpenTest command line interface instead of the web UI for starting the sessions. Jenkins supports this through the parallel stages feature. I'm sure all mainstream CI servers also have support for running parallel steps in a CI pipeline.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.