Selenium IDE runs test in half the time as selenium-side-runner

[ThomasBPG]

🐛 Bug Report

The attached script runs in half the time in Firefox Selenium IDE vs. running on the CLI with selenium-side-runner.

To Reproduce

Running it in Firefox Selenium IDE I can run the script in approx. 5s:

If I run it on the commandline, even headless it takes approx. 10s:

Expected behavior

If anything, I would have assumed the test execution to be at least similar, but also faster with the headless approach in selenium-side-runner.

Project file reproducing this issue (highly encouraged)

{
  "id": "129e5f91-23be-43e1-8861-a1887e86f5f0",
  "version": "2.0",
  "name": "Administration-Almindelig",
  "url": "URL",
  "tests": [{
    "id": "34092ef5-ebd8-41cf-ae96-ecec4c651275",
    "name": "Administration-Almindelig",
    "commands": [{
      "id": "5a419fd2-54db-4223-ad4d-6bde4b380915",
      "comment": "",
      "command": "open",
      "target": "/sign-in",
      "targets": [],
      "value": ""
    }, {
      "id": "14f7a555-453e-4e8a-ad25-00995c830f85",
      "comment": "",
      "command": "setWindowSize",
      "target": "1241x875",
      "targets": [],
      "value": ""
    }, {
      "id": "27eeb20c-4f0b-48de-9267-c056b9165b62",
      "comment": "",
      "command": "click",
      "target": "name=email",
      "targets": [
        ["name=email", "name"],
        ["css=.Mui-focused > .MuiInputBase-input", "css:finder"],
        ["xpath=//input[@name='email']", "xpath:attributes"],
        ["xpath=//div[@id='root']/div/main/div/div/div[2]/div/div/form/div[2]/div/input", "xpath:idRelative"],
        ["xpath=//input", "xpath:position"]
      ],
      "value": ""
    }, {
      "id": "6ced2346-87dc-4830-a9ec-47c531c29b1d",
      "comment": "",
      "command": "type",
      "target": "name=email",
      "targets": [
        ["name=email", "name"],
        ["css=.Mui-focused > .MuiInputBase-input", "css:finder"],
        ["xpath=//input[@name='email']", "xpath:attributes"],
        ["xpath=//div[@id='root']/div/main/div/div/div[2]/div/div/form/div[2]/div/input", "xpath:idRelative"],
        ["xpath=//input", "xpath:position"]
      ],
      "value": "USERNAME"
    }, {
      "id": "b3bae40a-13f9-4dde-b1b3-7e752c54d4ac",
      "comment": "",
      "command": "type",
      "target": "name=password",
      "targets": [
        ["name=password", "name"],
        ["css=.Mui-focused > .MuiInputBase-input", "css:finder"],
        ["xpath=//input[@name='password']", "xpath:attributes"],
        ["xpath=//div[@id='root']/div/main/div/div/div[2]/div/div/form/div[3]/div/input", "xpath:idRelative"],
        ["xpath=//div[3]/div/input", "xpath:position"]
      ],
      "value": "PASSWORD"
    }, {
      "id": "c22ed8fc-d034-4ad5-b4c5-5e9421d38ba4",
      "comment": "",
      "command": "click",
      "target": "css=.MuiButton-label",
      "targets": [
        ["css=.MuiButton-label", "css:finder"],
        ["xpath=//div[@id='root']/div/main/div/div/div[2]/div/div/form/div[4]/button/span", "xpath:idRelative"],
        ["xpath=//button/span", "xpath:position"]
      ],
      "value": ""
    }, {
      "id": "8091620c-e4c2-4454-a23f-88f5d2a9943a",
      "comment": "",
      "command": "click",
      "target": "css=.MuiGrid-root:nth-child(1) path",
      "targets": [
        ["css=.MuiGrid-root:nth-child(1) path", "css:finder"]
      ],
      "value": ""
    }, {
      "id": "ccaf7de2-0811-40fe-8188-4c97715541d8",
      "comment": "",
      "command": "waitForElementVisible",
      "target": "css=.MuiTypography-caption",
      "targets": [
        ["css=.MuiTypography-caption", "css:finder"],
        ["xpath=//div[@id='root']/div/main/div[2]/span", "xpath:idRelative"],
        ["xpath=//main/div[2]/span", "xpath:position"],
        ["xpath=//span[contains(.,'Miljø: Produktion')]", "xpath:innerText"]
      ],
      "value": "30000"
    }]
  }],
  "suites": [{
    "id": "272ce1d8-8496-4908-99d1-de05e74fd501",
    "name": "Default Suite",
    "persistSession": false,
    "parallel": false,
    "timeout": 300,
    "tests": ["34092ef5-ebd8-41cf-ae96-ecec4c651275"]
  }],
  "urls": ["URL"],
  "plugins": []
}% 

Environment

OS: Mac OS X 10.15.5 Selenium IDE Version: 3.17.0 Selenium SIDE Runner Version: 3.17.0 Node version: v10.21.0 Browser: Mozilla Firefox Browser Version: 77.0.1

[corevo]

selenium-side-runner uses a completely different approach to automation, it uses Selenium WebDriver, while Selenium IDE uses Selenium Core.

This creates a lot of differences between the two, some of them are functional, for example in the runner you can't click an invisible element.

Another aspect of that is execution time, WebDriver is much faster than the core, and until we move to the electron app, this will not be fixed.

[ThomasBPG]

Thanks for the quick feedback. Can you give any hints to how I can update my side-script, so it runs faster with selenium-side-runner ? Appreciate there are differences between the two, but I actually see the complete opposite with selenium-side-runner being much slower than IDE.

[corevo]

It depends on the script, some things are faster in the IDE, while some are slower, it is not a straight up improvement, but rather tradeoffs.

For example assert element displayed is faster in the runner but slower in the IDE, but select window is faster in the IDE and slower in the runner.

[ThomasBPG]

Hm...okay, fair. I shall try to do some trial-and-error to see if I can optimize the execution time in the runner. Is it true, btw., that there's no way to output the timing for each step with the runner?

Thanks

[corevo]

Yes, the way the runner is coded it won't allow that, it is something we already took care of in the new runtime.

[corevo]

You could try building what we dubbed for now as side-cli which is the electron cli utility, it has playback capabilities, you can build it off of master, it might give you better cli results.

[ThomasBPG]

I've been doing a lot more debugging and the script is now essentially just doing open and assert text. Still have excellent run times in the IDE (approx 1s) and the following from runner:

thomas@Thomass-MacBook-Pro Availability Monitoring % selenium-side-runner -c "browserName=firefox" Administration-Almindelig2.side 
info:    Running Administration-Almindelig2.side
 PASS  ./DefaultSuite.test.js (8.243s)
  Default Suite
    ✓ Administration-Almindelig (5044ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        8.72s
Ran all test suites.

thomas@Thomass-MacBook-Pro Availability Monitoring % selenium-side-runner -c "browserName=chrome" Administration-Almindelig2.side
info:    Running Administration-Almindelig2.side
 PASS  ./DefaultSuite.test.js
  Default Suite
    ✓ Administration-Almindelig (2428ms)

Test Suites: 1 passed, 1 total
Tests:       1 passed, 1 total
Snapshots:   0 total
Time:        4.444s, estimated 9s
Ran all test suites.

Guess I can conclude my website contains something the runner really don't like! :-) It's a customer website we are developing on behalf of a client, so I shall refrain from posting the URL here, but it's a React based website with two simple login boxes. Still puzzles me what makes the runner slow so much down....

Anyway, thanks for your feedback. I shall close down the issue.

[corevo]

There is no need to close, it is a legitimate concern and an issue with the IDE, that will not be so easily fixed, once we will unify the runtimes we can safely close this issue.