puagarwa
commented
4 months ago HI @willbertSmillbert thanks for detailed trying out service in detail, Let me explain few things about service and how can we best utilize it.
Service working
- first thing, Service works on client service protocol where client runs at your machine which execute test script and send commands to browser server hosted at our side to perform browser operations.
- Given this working style, its expected that running a single test in local would always be faster than service as service will have some network latency.
- However, how service helps reduce the overall execution time is by allowing users to use more parallel workers which otherwise are restricted in local machine as per its resource.
- Test runner default number of worker logic uses N-1 core, which on a standard CI agent machines (2 Core) means worker=1, This default recommendation is for local mode due to browser launched in the machine.
- For Service, You can increase this worker count to reasonably high number depend upon multi factors like test endpoint, other load on client machine, number of parallel tests in yoru suite.
Coming back to your results for 6,10 and 20 worker,
- Since your results shows, linear decrease in execution time for local mode from 6 worker to 10, i assume you are doing this testing from machine which have more than 8 core, pls confirm it.
- If thats the case, then the result make sense as i mentioned local execution time for same worker would always be lower than service.
- However, You may not get these many cores agent machines if you CI, You can compare execution once in CI.
- I can see, you mentioned worker=20 took around 2min in service, you could even try with higher number of worker like 30 or 40 to see further improvement.
- However, there are few things which you need to check if you dont see further improvement in beyond 20 worker
- Slowest test file: You cant reduce overall time less than slowest test file, hence you need to figure and break it down to eliminate this bottleneck
- Consider fullParallel:True if you want to allow test within file to run in parallel but it may require you to check its working impact. if you can do this then look out for slowest test as next bottleneck
- Test Endpoint load, other thing to check is your Test Endpoint can take concurrent load for high parallelism
- I agree around cross region latency adding up here and as you are in AUS East, you are using our nearest region east asia, we do have plan to be in Australia East in sometime.
Coming to Flaky tests
- Most Likely, this doesnt directly caused by Service but rather it get highlighted more with service integration due to high parallelism and independent things.
- Sometime, flakiness is due to timeout in operations, navigation: we should understand those flaky failures and accordingly, we could adjust some of the timeouts in those tests.
- Sometime, flakiness is also due to dependency in tests which get highlighted when ran in more parallelism.
- Yes, flakiness does increase execution time because it need to run more retries and each retries actually launch a new browser.
- Overall, we should focus on identifying flakiness reasons and work on it to get best out of service.
- We checked our telemetry and we dont see any issue in terms of connection failure and browser allocation failure.
Few things you can confirm
- Your Current testing Client machine CPU cores
- How many test files you have and how many total tests.
- Slowest test file execution time
- Execution time of your test suite when ran from CI machine , if you have use
- Test Endpoint: is it publicly accessible or private or local host
willbertSmillbert
Hey - I've been trialing out this service for a week now, using the east asia server and getting mixed results. Some of this may be attributed to latency between me and the Azure service. So opening this up will allow me to make a more fair comparison. Ive also noticed more test flake using MPT. Hopefully this aspect will be resolved with opening it up to the Aus Region.
For reference - Im seeing about a 30% slowness through a single test ran through MPT compared to local.
Could the way Ive written my tests be causing a bottleneck on your resources, leading to slow test execution?
How my tests are set up
Below is some stats comparing local to MPT execution times for the same test suite, all times are derived from the HTML report - The activity log in the MPT dashboard doesnt seem to be accurate
Sorry I went off track a little with regards to the feature request, Just wanted to clarify my situation, to determine if the root cause can be the latency, some other issue on our end or MPT. Or if the outcome is what is expected when using MPT compared to local.
Also - I feel like the performance of MPT is rather inconsistent(seems to be improving though?); difficult to pinpoint though, as test flakiness could be the culprit in this regard
Azure Subscription Id: 15451e06-b6e5-4b64-ada2-1bf419245949
Thanks !
AB#1952247