qase-tms / qase-java

Qase TMS Java SDK
https://developers.qase.io
Apache License 2.0
17 stars 6 forks source link

Not able to run multiple Qase Test Cases in parallel #140

Open mdiep-asp opened 3 weeks ago

mdiep-asp commented 3 weeks ago

Currently, if we try to run multiple Qase Test Cases in parallel (using maven surefire plugin + testng) by splitting all test cases into 3 test suites, we will see this error:

[main] INFO com.app.appium.provider.DeviceProvider - The test device is: cloud_ios
[main] INFO com.app.appium.utilities.AppiumDriverUtils - Activating app
[main] INFO com.app.appium.core.BaseTest - Test: testABC , Result:failed
[main] INFO com.app.appium.utilities.AppiumDriverUtils - Terminating app
[main] INFO com.app.appium.provider.DeviceProvider - The test device is: cloud_ios
[main] INFO com.app.appium.utilities.AppiumDriverUtils - Activating app
[main] ERROR io.qase.api.CasesStorage - Previous case is still in progress.
[main] INFO com.app.appium.core.BaseTest - Test: testDEF , Result:failed
[main] INFO com.app.appium.utilities.AppiumDriverUtils - Terminating app
[main] INFO com.app.appium.provider.DeviceProvider - The test device is: cloud_ios
[main] INFO com.app.appium.utilities.AppiumDriverUtils - Activating app
[main] ERROR io.qase.api.CasesStorage - Previous case is still in progress.
[main] INFO com.app.appium.core.BaseTest - Test: testGHI , Result:failed
[main] INFO com.app.appium.utilities.AppiumDriverUtils - Terminating app

Upon looking into Qase api, we saw this:

Screenshot 2024-07-02 at 2 30 13 PM

Seems like each test case run is associated with a thread. Even though we run the other test suites with a different maven instance, somehow the existing running test case thread from one suite was being called by others and caused this issue.

Wonder Qase actually supports running in parallel? If not, will Qase soon support it? Please help. Thanks

savkk commented 2 weeks ago

@mdiep-asp Hi! Could you give an example of the parallel running settings that you use?

mdiep-asp commented 2 weeks ago

@savkk we ran our mobile automation tests on CircleCI with these configs:

We believe that even though there are 3 running distinct workflows. They are still on the same machine, and probably share the same threads pool.

savkk commented 1 week ago

@mdiep-asp, have you set up parallel run through testng? Could you give an example of testng configuration?