search

vaadin / testbench

Vaadin TestBench is a tool for automated user interface testing of Vaadin applications.
https://vaadin.com/testbench
Other
20 stars 22 forks source link

Test performance of augmenting remote drivers and update manual accordingly #445

Closed vaadin-bot closed 12 years ago

vaadin-bot commented 12 years ago

Originally by @zch

When running tests on a remote machine, you need to augment the RemoteWebDriver instance in order to be able to e.g. grab screen shots.

Augmenting an instance of WebDriver is done as follows:

driver = new Augmenter().augment(driver);

Test whether always augmenting RemoteWebDrivers is an acceptable performance hit or not. If always augmenting is no big deal, modify the TestBenchTestCase.setDriver() to always augment RemoteWebDriver instances.

Update the manual (and javadoc) according to the findings.

vaadin-bot commented 12 years ago

Originally by mgrankvi

Changes in times would seem to be negligible even if Augmenter is run for all RemoteWebDriver instances.

For both solutions tried so far (adding the augmentation in 

   TestBench {
      public static WebDriver createDriver(WebDriver driver) {
         if(driver instanceof RemoteWebDriver) {
            driver = new Augmenter().augment((RemoteWebDriver) driver);
         }
         ...
      }
      ...
   }

OR

   TestBenchCommandExecutor {
      public TestBenchCommandExecutor(WebDriver actualDriver,
            ImageComparison imageComparison,
            ReferenceNameGenerator referenceNameGenerator) {
         if (actualDriver instanceof RemoteWebDriver) {
            this.actualDriver = new Augmenter().augment(actualDriver);
         } else {
            this.actualDriver = actualDriver;
         }
         ...
      } 
   }

The diver will begin opening multiple browser instances and loose the pointer to one of the first windows leaving it open after closing down the rest. This opening takes some time and will also fill the machine up with open instances when lots of tests are executed.

Augmenting in setDriver() did not work.

vaadin-bot commented 12 years ago

Originally by mgrankvi

Median times for running tests with (manual add to tests) and without augmentation of RemoteWebDriver

-Augmentation only on ScreenchotITCase ::*

com.vaadin.testbenchexample.AdvancedCommandsITCase :: 6.607 sec com.vaadin.testbenchexample.LoopingCalculatorITCase :: 3.672 sec com.vaadin.testbenchexample.ScreenshotITCase :: 6.491 sec com.vaadin.testbenchexample.SelectorExamplesITCase :: 4.285 sec com.vaadin.testbenchexample.SimpleCalculatorITCase :: 2.117 sec com.vaadin.testbenchexample.UsingHubITCase :: 0.002 sec com.vaadin.testbenchexample.VerifyExecutionTimeITCase :: 2.214 sec

-total* :: 31.057 sec

-Augmentation on all tests ::*

com.vaadin.testbenchexample.AdvancedCommandsITCase :: 6.754 sec com.vaadin.testbenchexample.LoopingCalculatorITCase :: 3.519 sec com.vaadin.testbenchexample.ScreenshotITCase :: 6.268 sec com.vaadin.testbenchexample.SelectorExamplesITCase :: 4.296 sec com.vaadin.testbenchexample.SimpleCalculatorITCase :: 2.109 sec com.vaadin.testbenchexample.UsingHubITCase :: 0.000333... sec com.vaadin.testbenchexample.VerifyExecutionTimeITCase :: 2.199 sec

-total* :: 31.002 sec

vaadin-bot commented 12 years ago

Originally by mgrankvi

Got automatic augmentation in createDriver to work as normal without leaving extra browser instances. Ran tests for comparable median times for the automatic augmentation.

-Automatic Augmentation on all tests ::*

com.vaadin.testbenchexample.AdvancedCommandsITCase :: 6.781 sec com.vaadin.testbenchexample.LoopingCalculatorITCase :: 3.630 sec com.vaadin.testbenchexample.ScreenshotITCase :: 6.482 sec com.vaadin.testbenchexample.SelectorExamplesITCase :: 4.526 sec com.vaadin.testbenchexample.SimpleCalculatorITCase :: 2.152 sec com.vaadin.testbenchexample.UsingHubITCase :: 0.001 sec com.vaadin.testbenchexample.VerifyExecutionTimeITCase :: 2.277 sec

total :: 31.497 sec

vaadin-bot commented 12 years ago

Originally by @zch

Introduced a small tweak to Mikael's patch: only augment if the driver does not implement TakesScreenshot. This takes care of windows that are left open in some cases.