search

nightwatchjs / nightwatch

Integrated end-to-end testing framework written in Node.js and using W3C Webdriver API. Developed at @browserstack
https://nightwatchjs.org
MIT License
11.79k stars 1.31k forks source link

Limiting number of parallel threads based on number of cores #3928

Closed gravityvi closed 11 months ago

gravityvi commented 12 months ago

Our test automation framework has around 450 test suites and 5000+ test cases automated which we execute on Selenium Grid daily basis using parallel execution. For executing the scripts in parallel we pass workers = 95 and the total execution time it takes is around 2.5 hours

The current NightwatchJs version used by the framework is 2.6.19 and we were planning to upgrade it to 3.1

During the upgrade we are not able to execute those many test suites in parallel as it restricts to parallel execution only upto the no. of CPU cores irrespective of what no. of workers we are passing for parallel test execution

We just tried executing only smoke test suites consisting of 140 test suites out of the total automated test suites, which used to earlier take 35 mins for complete execution and now it is taking 2 hours

This is impacting the performance for our test execution and we are not able to achieve complete test automation execution in around 2.5 hours as earlier. The smoke test execution itself only is taking 2 hours now

We really need a solution for this as our framework has that huge number of test cases to be executed on daily basis.

Can you please share your inputs/suggestions on same.

Thanks

Originally posted by @aakanksha88 in https://github.com/nightwatchjs/nightwatch/discussions/3921#discussioncomment-7242362

gravityvi commented 12 months ago

As the description of the issue suggests that limiting the number of parallel threads might not be a good idea performance-wise. This problem is also discussed in detail in piscina issue over here: https://github.com/piscinajs/piscina/issues/334#issuecomment-1596866218

aakanksha88 commented 11 months ago

Hi @gravityvi Thank you for getting it fixed. I could see the changes are merged, so will it be now available in the nightwatchjs version 3.2.1 ?

vaibhavsingh97 commented 11 months ago

@aakanksha88, since we haven't released a new version. But you can try the change it with the following command:

npm install "https://github.com/nightwatchjs/nightwatch.git#main" --save
aakanksha88 commented 11 months ago

Sure @vaibhavsingh97 we will try with the command suggested by you. Thank you . Can you or @gravityvi please confirm in which version of nightwatchjs will this be available and by when, so that we can plan the version upgrade of our framework accordingly.

aakanksha88 commented 11 months ago

Hi @gravityvi @vaibhavsingh97 we tested the fix and are now able to see that the the number of test suites executed in parallel are equal to the number of test workers we are passing. Thank you for fixing this,

But we are seeing some issue in using the test setting 'parallel_process_delay'. The setting is not working while parallel execution, all the test suite execution starts at once without any delay irrespective of how many milli seconds we pass as a value for this test setting in nightwatch.json. We tried with 10000 and 20000 ms but none helped.

Could you please also provide some inputs on how we can achieve parallel process delay, it will be of great help. We need to access our application which has some restrictions on how many requests we can send once due to which we need to introduce some delay between the requests.

Have reported issue for same - [https://github.com/nightwatchjs/nightwatch/issues/3947] Thanks in advance.