search

SeleniumHQ / selenium-ide

Open Source record and playback test automation for the web.
https://selenium.dev/selenium-ide/
Apache License 2.0
2.81k stars 768 forks source link

selenium-side-driver --filter doesn't recognize regex anchors and can't find matching test names reliably #1550

Closed balimoon777 closed 1 year ago

balimoon777 commented 2 years ago

🐛 Bug Report

selenium-side-driver --filter doesn't recognize regex anchors and can't find matching test names reliably

To Reproduce

(1) --filter doesn't recognize regex anchor $ and ^

% selenium-side-runner --filter="credential$" demo_project_2.side info: Running demo_project_2.side No tests found, exiting with code 1 Run with --passWithNoTests to exit with code 0 In /Users/blah/demo/side-suite-demo_project_2 8 files checked. testMatch: {*/credential$/.test.js,*/credential$*.test.js} - 0 matches testPathIgnorePatterns: /node_modules/ - 8 matches testRegex: - 0 matches Pattern: - 0 matches

(2) --filter doesn't find matching test names reliably

The project file contains the test suite 'awesome_suite' which has a test 'good_credential'. The --filter=good fails to find the test 'good_credential'

% selenium-side-runner --filter="good" demo_project_2.side info: Running demo_project_2.side No tests found, exiting with code 1 Run with --passWithNoTests to exit with code 0 In /Users/blah/demo/side-suite-demo_project_2 8 files checked. testMatch: {*/good/.test.js,*/good*.test.js} - 0 matches testPathIgnorePatterns: /node_modules/ - 8 matches testRegex: - 0 matches Pattern: - 0 matches

Expected behavior

A clear and concise description of what you expected to happen.

Project file reproducing this issue (highly encouraged)

{ "id": "c4eb88d3-9c18-4b8e-821b-e6e0dc35925a", "version": "2.0", "name": "demo_project_2", "url": "https://myurl.com/", "tests": [{ "id": "4a7b0901-1431-4aa3-ab5c-c4bd694e5b3b", "name": "invalid_credential", "commands": [{ "id": "d1944b98-8c98-4557-90e5-1ca7caf4fe17", "comment": "", "command": "run", "target": "global_static_vars", "targets": [], "value": "" }, { "id": "203dc4be-6a16-445a-92fe-fc5b04073f42", "comment": "", "command": "store", "target": "${invalid_username}", "targets": [], "value": "username" }, { "id": "b7a77db9-95d4-499d-9f72-6a8e7435df6c", "comment": "", "command": "store", "target": "${invalid_password}", "targets": [], "value": "password" }, { "id": "f6bd6d86-a620-442f-94e8-9b88ec9cd97c", "comment": "", "command": "run", "target": "login", "targets": [], "value": "" }, { "id": "a049f4e2-d993-4c86-93d7-9de878947c51", "comment": "", "command": "assertText", "target": "css=.alert:nth-child(2) > span", "targets": [], "value": "Sorry, we were not able to find a user with that username and password." }] }, { "id": "80b92de5-e26d-4bde-8e3c-2818c9155f9b", "name": "invalid_credential_v2_blah", "commands": [{ "id": "4960df13-04d1-427b-a5dc-4abf303a5583", "comment": "", "command": "run", "target": "global_static_vars", "targets": [], "value": "" }, { "id": "df86218e-4cb8-4383-9c33-257908869790", "comment": "", "command": "store", "target": "${invalid_username}", "targets": [], "value": "username" }, { "id": "94186dfa-3dce-4ee2-acd5-f14aed59e5f5", "comment": "", "command": "store", "target": "${invalid_password}", "targets": [], "value": "password" }, { "id": "20e98354-a899-4557-ba3c-eacdea3bd928", "comment": "", "command": "run", "target": "login", "targets": [], "value": "" }, { "id": "ff591ecb-a6f5-431e-b8b5-16855f1fc126", "comment": "", "command": "assertText", "target": "css=.alert:nth-child(2) > span", "targets": [], "value": "Sorry, we were not able to find a user with that username and password." }] }, { "id": "e778bf8b-566a-4197-a62a-2e3505d8e350", "name": "global_static_vars", "commands": [{ "id": "adb7cb16-eb3a-4f67-90ee-32b0632a5a7f", "comment": "", "command": "store", "target": "user1", "targets": [], "value": "valid_username" }, { "id": "f413f97e-e17d-4248-83fe-d0199503080e", "comment": "", "command": "store", "target": "paswword1", "targets": [], "value": "valid_password" }, { "id": "81ffa638-b2cb-4547-a45e-2f1fb3172821", "comment": "", "command": "store", "target": "blah", "targets": [], "value": "invalid_username" }, { "id": "27f3f303-2d7f-474b-9d56-a3ca412d6d06", "comment": "", "command": "store", "target": "blah", "targets": [], "value": "invalid_password" }] }, { "id": "b5d3d691-f830-4e81-a86f-83175610fd64", "name": "login", "commands": [{ "id": "a5f1ac7a-8a1f-441e-ba7e-74378d94db07", "comment": "", "command": "open", "target": "login/auth", "targets": [], "value": "" }, { "id": "fd38906d-c250-4c78-a8e3-898ee1d179bb", "comment": "", "command": "setWindowSize", "target": "1588x1113", "targets": [], "value": "" }, { "id": "e670d3a0-28a4-41f8-a848-7817f68d7472", "comment": "", "command": "click", "target": "id=username", "targets": [ ["id=username", "id"], ["name=j_username", "name"], ["css=#username", "css:finder"], ["xpath=//input[@id='username']", "xpath:attributes"], ["xpath=//input", "xpath:position"] ], "value": "" }, { "id": "720efb69-383a-4d5b-8079-5bbdd16a6bad", "comment": "", "command": "type", "target": "id=username", "targets": [ ["id=username", "id"], ["name=j_username", "name"], ["css=#username", "css:finder"], ["xpath=//input[@id='username']", "xpath:attributes"], ["xpath=//input", "xpath:position"] ], "value": "${username}" }, { "id": "1cca6d0f-091f-48b8-938e-aaac5b1d7518", "comment": "", "command": "click", "target": "name=j_password", "targets": [ ["name=j_password", "name"], ["css=.form-label-group:nth-child(3) > .form-control", "css:finder"], ["xpath=//input[@name='j_password']", "xpath:attributes"], ["xpath=//div[3]/input", "xpath:position"] ], "value": "" }, { "id": "513f52df-405f-4ca0-a612-89cb36bb4372", "comment": "", "command": "type", "target": "name=j_password", "targets": [ ["name=j_password", "name"], ["css=.form-label-group:nth-child(3) > .form-control", "css:finder"], ["xpath=//input[@name='j_password']", "xpath:attributes"], ["xpath=//div[3]/input", "xpath:position"] ], "value": "${password}" }, { "id": "a23dd99e-ee6f-47ed-ab08-a7696d5274ae", "comment": "", "command": "click", "target": "css=.btn", "targets": [ ["css=.btn", "css:finder"], ["xpath=//button[@type='submit']", "xpath:attributes"], ["xpath=//div[4]/div/button", "xpath:position"], ["xpath=//button[contains(.,'Login')]", "xpath:innerText"] ], "value": "" }] }, { "id": "537d47c1-8c8e-42eb-95e9-c4f3a666d5d5", "name": "logout", "commands": [{ "id": "3e96ed6c-3ff3-40d1-9eeb-b273de45a944", "comment": "", "command": "click", "target": "css=.user-image", "targets": [ ["css=.user-image", "css:finder"], ["xpath=//div[@id='logoutDD']/div", "xpath:idRelative"], ["xpath=//div[5]/div/div", "xpath:position"] ], "value": "" }, { "id": "4e3f3b66-bb2f-466a-b1ca-24b9d48cd366", "comment": "", "command": "click", "target": "css=.dropdown-item:nth-child(4)", "targets": [ ["css=.dropdown-item:nth-child(4)", "css:finder"], ["xpath=//div[@id='root']/div/header/div/div/div[4]/div[5]/ul/li[4]", "xpath:idRelative"], ["xpath=//li[4]", "xpath:position"], ["xpath=//li[contains(.,'Log Out')]", "xpath:innerText"] ], "value": "" }] }, { "id": "320cc669-a912-4688-8fec-93690c40dffc", "name": "bad_credential", "commands": [{ "id": "5b78af93-eab8-4c35-8fe8-704dd313d942", "comment": "", "command": "run", "target": "global_static_vars", "targets": [], "value": "" }, { "id": "dd063b90-6b5e-4571-8896-73ad25af48a2", "comment": "", "command": "store", "target": "${invalid_username}", "targets": [], "value": "username" }, { "id": "c9bf47e7-6b5d-4d13-9894-4a8eba199c79", "comment": "", "command": "store", "target": "${invalid_password}", "targets": [], "value": "password" }, { "id": "6efea7f0-206e-49b6-8979-ad3c3f812a75", "comment": "", "command": "run", "target": "login", "targets": [], "value": "" }, { "id": "5e98f221-08fb-47e1-ba82-2569433575b4", "comment": "", "command": "assertText", "target": "css=.alert:nth-child(2) > span", "targets": [], "value": "Sorry, we were not able to find a user with that username and password." }] }, { "id": "9c85a005-3f52-4249-8030-5b7c17390c2b", "name": "good_credential", "commands": [{ "id": "cb19706f-899e-4cf6-8fcb-08a594dcb34c", "comment": "", "command": "run", "target": "global_static_vars", "targets": [], "value": "" }, { "id": "bd994f5d-b5f6-4b95-9fc0-f2d2a104b075", "comment": "", "command": "store", "target": "${valid_username}", "targets": [], "value": "username" }, { "id": "7eb2a654-91b3-44d2-bf16-c7d8389174cf", "comment": "", "command": "store", "target": "${valid_password}", "targets": [], "value": "password" }, { "id": "02ea0076-c003-47b1-9686-871d24d9b1ed", "comment": "", "command": "run", "target": "login", "targets": [], "value": "" }, { "id": "bbc36519-4c33-4543-b1bb-4df4d2319198", "comment": "", "command": "run", "target": "logout", "targets": [], "value": "" }] }], "suites": [{ "id": "55446e0b-1b88-4fe5-870d-e49e55d4f506", "name": "unhappy_suite", "persistSession": false, "parallel": true, "timeout": 300, "tests": ["320cc669-a912-4688-8fec-93690c40dffc", "4a7b0901-1431-4aa3-ab5c-c4bd694e5b3b", "80b92de5-e26d-4bde-8e3c-2818c9155f9b"] }, { "id": "0a3af420-9b07-48f2-9818-014444e1f417", "name": "awesome_suite", "persistSession": false, "parallel": false, "timeout": 300, "tests": ["9c85a005-3f52-4249-8030-5b7c17390c2b"] }, { "id": "ab66ac7f-b3df-4aa2-986e-d8ddbd4c54dc", "name": "common_suite", "persistSession": false, "parallel": false, "timeout": 300, "tests": ["b5d3d691-f830-4e81-a86f-83175610fd64", "537d47c1-8c8e-42eb-95e9-c4f3a666d5d5", "e778bf8b-566a-4197-a62a-2e3505d8e350"] }], "urls": ["https://myurl.com/", "plugins": [] }

Environment

OS: MacOS Monterey 12.6 Selenium IDE Version: v3.17.0 Selenium SIDE Runner Version: 4.0.0-alpha.15 Node version: v18.10.0 ChromeDriver version: 106.0.5249.61 Browser: Browser Version: 106.0.5249.61

toddtarsi commented 1 year ago

@balimoon777 - Hmm, I'm not sure. This might be like a shell character escaping issue. Can you try swapping the double quotes on your argument for single quotes?

toddtarsi commented 1 year ago

@balimoon777 - Yep, this appears to work fine. Just tried in a linux shell with this command:

> node ./packages/side-runner/dist/bin.js -t 15000 -f 'else$' ./tests/examples/control-flow.side

  console.info

          Project Control Flow doesn't have any suites matching filter else$,
          attempting to iterate all tests in project

      at src/main.test.ts:150:13
          at Array.forEach (<anonymous>)

info: Running test control flow else
info: Building driver for chrome
info: Driver has been built for chrome
info: Finished test control flow else Success
 PASS  packages/side-runner/dist/main.test.js
  Running project Control Flow
    ✓ Running test control flow else (1070 ms)

The project is located here:

https://github.com/SeleniumHQ/selenium-ide/blob/trunk/tests/examples/control-flow.side

The fact that the test called 'control-flow-else-if' was filtered tells me that the characters are working fine.

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.