Find Elements from shadow roots is using wrong end points to search

[AutomatedTester]

Description of the bug/issue

When searching from a shadow root Nightwatch should use the Find from shadow root endpoints

Steps to reproduce

Run the test below

Sample test

describe("shadow root example",
    it("can inspect a shadow root", async (browser) => {
        browser.navigateTo("https://mdn.github.io/web-components-examples/word-count-web-component/");

        const shadowRoot = await browser.getShadowRoot("body > article:nth-child(2) > p:nth-child(4)")
        const shadowRootLabel = await shadowRoot.find("span")
        await expect(shadowRootLabel.property('innerHTML')).to.include("212")
    })
)

Command to run

npx nightwatch test/e2e/shadow.js -e chrome

Verbose Output

davidburns in ~/development/testpad/nightwatch-todo on trunk ● ● λ npx nightwatch test/e2e/shadow.js -e chrome --verbose

[E2e/Shadow] Test Suite
──────────────────────────────────────────────
⠋ Starting ChromeDriver on port 9515...
 Starting ChromeDriver...
Selenium Manager binary found at /Users/davidburns/development/testpad/nightwatch-todo/node_modules/selenium-webdriver/bin/macos/selenium-manager
Driver path: /Users/davidburns/.cache/selenium/chromedriver/mac-arm64/119.0.6045.105/chromedriver
⠙ Starting ChromeDriver on port 9515...
   Request POST /session  
   {
     capabilities: {
       firstMatch: [ {} ],
       alwaysMatch: {
         browserName: 'chrome',
         'goog:chromeOptions': {
           w3c: true,
           args: [],
           binary: '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
         }
       }
     }
⠹ Starting ChromeDriver on port 9515...
Error: The following dependencies are imported but could not be resolved:

  @emotion/is-prop-valid (imported by /Users/davidburns/development/testpad/nightwatch-todo/tests_output/nightwatch-html-report/index.html?id=0)

Are they installed?
    at file:///Users/davidburns/development/testpad/nightwatch-todo/node_modules/vite/dist/node/chunks/dep-bb8a8339.js:45779:23
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
    at async file:///Users/davidburns/development/testpad/nightwatch-todo/node_modules/vite/dist/node/chunks/dep-bb8a8339.
⠴ Starting ChromeDriver on port 9515...
   Response 200 POST /session (1104ms)
   {
     value: {
       capabilities: {
         acceptInsecureCerts: false,
         browserName: 'chrome',
         browserVersion: '119.0.6045.159',
         chrome: {
           chromedriverVersion: '119.0.6045.105 (38c72552c5e15ba9b3117c0967a0fd105072d7c6-refs/branch-heads/6045@{#1103})',
           userDataDir: '/var/folders/y1/2d_l1hxj4bv29b4_j4v1qlkw0000gn/T/.org.chromium.Chromium.rpo5kS'
         },
         'fedcm:accounts': true,
         'goog:chromeOptions': { debuggerAddress: 'localhost:65160' },
         networkConnectionEnabled: false,
         pageLoadStrategy: 'normal',
         platformName: 'mac',
         proxy: {},
         setWindowRect: true,
         strictFileInteractability: false,
         timeouts: { implicit: 0, pageLoad: 300000, script: 30000 },
         unhandledPromptBehavior: 'dismiss and notify',
         'webauthn:extension:credBlob': true,
         'webauthn:extension:largeBlob': true,
         'webauthn:extension:minPinLength': true,
         'webauthn:extension:prf': true,
         'webauthn:virtualAuthenticators': true
       },
       sessionId: 'cb0ebb9a8d90a2e532ae0fdd70138eab'
     }
ℹ Connected to ChromeDriver on port 9515 (1269ms).
  Using: chrome (119.0.6045.159) on MAC.

 Received session with ID: cb0ebb9a8d90a2e532ae0fdd70138eab

 → Running [before]:
 → Completed [before].

  Running can inspect a shadow root:
───────────────────────────────────────────────────────────────────────────────────────────────────
 → Running [beforeEach]:
 → Completed [beforeEach].

 → Running command: navigateTo ('https://mdn.github.io/web-components-examples/word-count-web-component/')
   Request POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/url  
   {
     url: 'https://mdn.github.io/web-components-examples/word-count-web-component/'
  }
   Response 200 POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/url (334ms)
   { value: null }
 → Completed command: navigateTo ('https://mdn.github.io/web-components-examples/word-count-web-component/') (334ms)

 → Running command: getShadowRoot ('body > article:nth-child(2) > p:nth-child(4)')
   Request POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/elements  
   {
     using: 'css selector',
     value: 'body > article:nth-child(2) > p:nth-child(4)'
  }
   Response 200 POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/elements (15ms)
   {
     value: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '39D3DB7D8CA6E5F533596E773051FED0_element_3'
       }
     ]
  }
   Request POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/execute/sync  
   {
     script: 'return (function(element) {\n' +
       '            return element && element.shadowRoot;\n' +
       '          }).apply(null, arguments);... (114 characters)',
     args: [
       {
         'element-6066-11e4-a52e-4f735466cecf': '39D3DB7D8CA6E5F533596E773051FED0_element_3'
       }
     ]
  }
   Response 200 POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/execute/sync (4ms)
   {
     value: {
       'shadow-6066-11e4-a52e-4f735466cecf': '39D3DB7D8CA6E5F533596E773051FED0_element_4'
     }
  }
 → Completed command: getShadowRoot ('body > article:nth-child(2) > p:nth-child(4)') (22ms)

 → Running command: element().find ('span')
   Request POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/element/39D3DB7D8CA6E5F533596E773051FED0_element_4/element  
   { using: 'css selector', value: 'span' }
   Response 200 POST /session/cb0ebb9a8d90a2e532ae0fdd70138eab/element/39D3DB7D8CA6E5F533596E773051FED0_element_4/element (7ms)
   {
     value: {
       'element-6066-11e4-a52e-4f735466cecf': '39D3DB7D8CA6E5F533596E773051FED0_element_5'
     }
  }
 → Completed command: element().find ({value, using}) (7ms)
 → Running [afterEach]:

 → Running command: element().property ('innerHTML')
   Request GET /session/cb0ebb9a8d90a2e532ae0fdd70138eab/element/39D3DB7D8CA6E5F533596E773051FED0_element_5/property/innerHTML  

   Response 200 GET /session/cb0ebb9a8d90a2e532ae0fdd70138eab/element/39D3DB7D8CA6E5F533596E773051FED0_element_5/property/innerHTML (4ms)
   { value: 'Words: 212' }
 → Completed command: element().property ('innerHTML') (4ms)

 → Running command: expect(<value>)..to.include('212') ()
  ✔ Expected 'Words: 212'  to include('212'): 
 → Completed command: expect(<value>)..to.include('212') () (1ms)
 → Completed [afterEach].

  ✨ PASSED. 1 assertions. (378ms)
 → Running [after]:
 → Completed [after].

 → Running command: end (true)

 → Running command: session ('delete', [Function])
   Request DELETE /session/cb0ebb9a8d90a2e532ae0fdd70138eab  

   Response 200 DELETE /session/cb0ebb9a8d90a2e532ae0fdd70138eab (53ms)
   { value: null }
 → Completed command: session ('delete', [Function]) (54ms)
 Wrote log file to: /Users/davidburns/development/testpad/nightwatch-todo/logs/e2e/shadow_chromedriver.log
 → Completed command: end (true) (57ms)
 ChromeDriver process closed.
 Wrote HTML report file to: /Users/davidburns/development/testpad/nightwatch-todo/tests_output/nightwatch-html-report/index.html

 Wrote Rerun Json report file to: /Users/davidburns/development/testpad/nightwatch-todo/tests_output/minimal_report.json
 Wrote JSON report file to: /Users/davidburns/development/testpad/nightwatch-todo/tests_output/e2e/CHROME_119.0.6045.159__shadow.json
 Wrote XML report file to: /Users/davidburns/development/testpad/nightwatch-todo/tests_output/e2e/CHROME_119.0.6045.159__shadow.xml
 Analytics send event:
   {
     client_id: 'bb2df1c3-13a3-42b7-a1b2-00e1f98f8b44',
     non_personalized_ads: true,
     timestamp_micros: 1701293908431000,
     events: [
       {
         name: 'nw_test_run',
         params: {
           arg_parallel: 'undefined',
           browser_name: 'chrome',
           test_workers_enabled: false,
           use_xpath: false,
           is_bstack: false,
           test_runner: 'default',
           event_time: 1701293906196000,
           env_os: 'darwin/22.1.0/Apple M2',
           env_lang: 'en_GB.UTF-8',
           env_nw_version: '3.3.2',
           env_node_version: 'node v18.16.1',
           test_env: 'chrome',
           run_id: '76c1f1bc-4413-4469-bade-d9b3bdb7af98'
         }
       }
     ]
  }

Nightwatch Configuration

// Refer to the online docs for more details:
// https://nightwatchjs.org/gettingstarted/configuration/
//

//  _   _  _         _      _                     _          _
// | \ | |(_)       | |    | |                   | |        | |
// |  \| | _   __ _ | |__  | |_ __      __  __ _ | |_   ___ | |__
// | . ` || | / _` || '_ \ | __|\ \ /\ / / / _` || __| / __|| '_ \
// | |\  || || (_| || | | || |_  \ V  V / | (_| || |_ | (__ | | | |
// \_| \_/|_| \__, ||_| |_| \__|  \_/\_/   \__,_| \__| \___||_| |_|
//             __/ |
//            |___/

module.exports = {
  // An array of folders (excluding subfolders) where your tests are located;
  // if this is not specified, the test source must be passed as the second argument to the test runner.
  src_folders: ['test', 'nightwatch/examples'],

  // See https://nightwatchjs.org/guide/concepts/page-object-model.html
  page_objects_path: ['nightwatch/page-objects'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-commands.html
  custom_commands_path: ['nightwatch/custom-commands'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-custom-assertions.html
  custom_assertions_path: ['nightwatch/custom-assertions'],

  // See https://nightwatchjs.org/guide/extending-nightwatch/adding-plugins.html
  plugins: ['@nightwatch/react'],

  // See https://nightwatchjs.org/guide/concepts/test-globals.html
  globals_path: 'nightwatch/globals.js',

  vite_dev_server: {
    start_vite: true,
    port: 5173
  },

  webdriver: {},

  test_workers: {
    enabled: true
  },

  test_settings: {
    default: {
      disable_error_log: false,
      launch_url: 'http://localhost:5173',

      screenshots: {
        enabled: false,
        path: 'screens',
        on_failure: true
      },

      desiredCapabilities: {
        browserName: 'chrome'
      },

      webdriver: {
        start_process: true,
        server_path: ''
      },

    },

    firefox: {
      desiredCapabilities: {
        browserName: 'firefox',
        alwaysMatch: {
          acceptInsecureCerts: true,
          'moz:firefoxOptions': {
            args: [
              // '-headless',
              // '-verbose'
            ]
          }
        }
      },
      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // very verbose geckodriver logs
          // '-vv'
        ]
      }
    },

    chrome: {
      desiredCapabilities: {
        browserName: 'chrome',
        'goog:chromeOptions': {
          // More info on Chromedriver: https://sites.google.com/a/chromium.org/chromedriver/
          //
          // w3c:false tells Chromedriver to run using the legacy JSONWire protocol (not required in Chrome 78)
          w3c: true,
          args: [
            //'--no-sandbox',
            //'--ignore-certificate-errors',
            //'--allow-insecure-localhost',
            //'--headless'
          ]
        }
      },

      webdriver: {
        start_process: true,
        server_path: '',
        cli_args: [
          // --verbose
        ]
      }
    },

  },

  usage_analytics: {
    enabled: true,
    log_path: './logs/analytics',
    client_id: 'bb2df1c3-13a3-42b7-a1b2-00e1f98f8b44'
  }

};

Nightwatch.js Version

2.6.21, 3.3.0

Node Version

18

Browser

Firefox 120, Chrome 119

Operating System

No response

Additional Information

No response

[garg3133]

As I am investigating this, I found that the getShadowRoot command should return a ShadowRoot instance and officially, there are only two methods that are supposed to work on it, findElement and findElement (methods like find and findAll can be added as aliases for these methods for better syntax).

But the problem is that right now we treat the shadow root as an instance of WebElement instead of ShadowRoot, which provides a lot of other methods like property and when we use some of these methods directly on the shadow root (being treated as a WebElement right now), it works on them, even on Firefox.

Try this:

describe("shadow root example",
    it("can inspect a shadow root", async (browser) => {
        browser.navigateTo("https://mdn.github.io/web-components-examples/word-count-web-component/");

        const shadowRoot = await browser.getShadowRoot("body > article:nth-child(2) > p:nth-child(4)")
        // const shadowRootLabel = await shadowRoot.find("span")
        await expect(shadowRoot.property('innerHTML')).to.include("Words: 212")
    })
)

and it will pass on all browsers, while the property method is not officially supported on ShadowRoot.

So, while the correct way to solve this seems to be to treat ShadowRoot as the instance of ShadowRoot class instead of an instance of WebElement, but doing so might break the behaviour for some users as many methods which currently work on a ShadowRoot in Nightwatch will stop working.

So, what I propose it to solve the issue where getShadowRoot() is not working in Firefox (which can be solved by hitting /element/{element id)/shadow endpoint instead of executing a script) but still treat it as a WebElement for further chaining of operations.

[AutomatedTester]

I think it's fine to special case it here so we can maintain good chaining