Using this with cucumber doesn't seem to work

[dietergeerts]

Hi,

I'm trying to use this for our end-2-end testing with cucumber, but it doesn't seem to work.

  // This works, waits for it to be displayed, and continues
  await browser.$('p*=select your region').waitForDisplayed()
  // So now we are sure that this is in the document body
  // but the following now seems to fail, as I always get `null` returned:
  expect(await browser.queryByText(/select your region/)).not.toBe(null)

Is it a known issue that it doesn't work with cucumber, or perhaps it can be something else? Any pointers would be helpful. I would like to use this as we already use it for our unit tests, and it makes us write better code, by making sure our application is semantically correct.

[olivierwilkinson]

Hi there 👋

Thanks for raising this. I wasn't able to reproduce the issue, but I did have to change the example to await the selector call before calling waitForDisplayed:

await(await browser.$(`button*=${this.text}`)).waitForDisplayed()

I think support for async chaining has been added recently in WebdriverIO v7.9, is that the version you are using?

I've pushed my branch and run the tests in an action in case you wanted to have a look / compare the differences in setup and stuff. 😄 (the tests results are in the validate step under Node 12 or 14 in the action)

I'm hoping it's the potentially missing await that's the problem but if it's not let me know and we can dig deeper 👍

[dietergeerts]

Well, I have 2 end-2-end tests that work with just webdrive-io and cucumber combo. When I add this library, and thus get the following code, they fail. The following is a piece of "page object" that I updated to use the queryByText with. Maybe I'm using it wrong?

I get the error " Cannot read property 'waitForDisplayed' of null", so it seems this library can't find the elements, while webdriver-io does.

export const authLogin = {
  get welcomeMessage() {
    return browser.$('h1*=Welcome')
  },
  get gotToLoginButton() {
    return browser.$('button*=login')
  },
  navigate() {
    return browser.url('/')
  },
  async waitForLoaded() {
    const { queryByText } = setupBrowser(browser)
    const message = await queryByText(/Welcome/i)
    return message.waitForDisplayed()

    // The following just times out, so it also doesn't has anything to do with waiting:
    // return browser.waitUntil(async () => await queryByText(/Welcome/i))

    // This is the code that works, with just webdriver-io:
    // return this.welcomeMessage.waitForDisplayed()
  },

Using the following dependencies:

    "expect-webdriverio": "^3.1.0",
    "@testing-library/webdriverio": "^3.0.2",
    "@wdio/cli": "^7.9.0",
    "@wdio/cucumber-framework": "^7.9.0",
    "@wdio/local-runner": "^7.9.0",
    "@wdio/spec-reporter": "^7.9.0",
    "@cucumber/cucumber": "7.0.0",

    // used by unit tests in the same package, I guess these will not have any influence?
    "@testing-library/jest-dom": "^5.11.9",
    "@testing-library/react": "^11.2.5",
    "@testing-library/react-hooks": "^5.1.2",
    "@testing-library/user-event": "^12.8.3",

[olivierwilkinson]

Hi again, sorry for the slow response! I've been on holiday 😄

Thanks for the extra information, I've aligned our package versions but unfortunately I'm still unable to reproduce the problem. I am able to reproduce the error you are getting, but only when that the element is not currently in the DOM when you execute queryByText. Would you mind replacing queryByText with findByText to see if that resolves your problem? e.g.

async waitForLoaded() {
  const { findByText } = setupBrowser(browser)
  await findByText(/Welcome/i, {}, { timeout: 5000 }) // where the timeout matches your wdio timeout
}

The reason I think that might help it is that the queryBy methods don't behave the same way as WebdriverIO selectors. queryByText will resolve to null if the DOM element cannot be found, whereas selectors will resolve with an WebdriverIO.Element even if nothing could not be found; this is to support calling methods such as element.waitForDisplayed after the fact. findByText(/Welcome/i) is a closer match to the behaviour of $('h1*=Welcome').waitForDisplayed() as it also waits for the element to be displayed.

Would you mind also trying the following to verify what the WebdriverIO selector is finding in your case?

async waitForLoaded() {
  const message = await browser.$('h1*=Welcome')
  console.log(message)
  console.log('wait for displayed...')
  await message.waitForDisplayed();
  console.log('elementId': message.elementId)
}

If the element does not exist when await browser.$('h1*=Welcome') is called then you should find a 'no such element' error in the resolved Element and there should not be an elementId property. Then after waitForDisplayed resolves the the elementId property should exist:

Element {
  error: {
    error: 'no such element',
    message: 'Unable to locate element: .//h1[contains(., "Welcome")]',
  },
  selector: 'h1*=Welcome',
  // etc... there should be no elementId property
 }
 wait for displayed...
 elementId: 9fb9676a-1157-544d-bb9a-9d214403cf8e

If using a findBy query doesn't help and the selector is indeed finding the element without calling waitForDisplayed then I'm plum out of ideas on how to fix this. If you are able/permitted to share a minimal example of the app you are testing that would probably help but I appreciate that is not always possible.

Anyways let me know how you get on 😁

[dietergeerts]

Ok, I'm back from vacation and trying this out.

If I just replace the queryBy with findBy, it still doesn't work, though I was hopeful after your explanation of the difference between the two that this was the mistake I was making.

When I then went on and executed your logs, I see that there is an element found, so the following:

    const message = await browser.$('h1*=Welcome')
    console.log('#####', message)
    console.log('#####', 'wait for displayed...')
    await message.waitForDisplayed()
    console.log('#####', `elementId: ${message.elementId}`)

Gives me these logs:

 ##### Element {
[0-0]   sessionId: 'd075a7c1-06b4-4843-ada9-5d49300d4910',
[0-0]   elementId: 'ELEMENT-1',
[0-0]   'element-6066-11e4-a52e-4f735466cecf': 'ELEMENT-1',
[0-0]   selector: 'h1*=Welcome',
[0-0]   parent: Browser {
[0-0]     sessionId: 'd075a7c1-06b4-4843-ada9-5d49300d4910',
[0-0]     capabilities: {
[0-0]       browserName: 'Chrome Headless',
[0-0]       browserVersion: '93.0.4577.82',
[0-0]       platformName: 'win32',
[0-0]       platformVersion: '10.0.18363',
[0-0]       'goog:chromeOptions': [Object]
[0-0]     },
[0-0]     addCommand: [Function (anonymous)],
[0-0]     overwriteCommand: [Function (anonymous)],
[0-0]     addLocatorStrategy: [Function (anonymous)],
[0-0]     config: {
[0-0]       specs: [Array],
[0-0]       suites: {},
[0-0]       exclude: [],
[0-0]       outputDir: undefined,
[0-0]       logLevel: 'info',
[0-0]       logLevels: {},
[0-0]       excludeDriverLogs: [],
[0-0]       bail: 0,
[0-0]       waitforInterval: 500,
[0-0]       waitforTimeout: 5000,
[0-0]       framework: 'cucumber',
[0-0]       reporters: [Array],
[0-0]       services: [],
[0-0]       maxInstances: 100,
[0-0]       maxInstancesPerCapability: 100,
[0-0]       filesToWatch: [],
[0-0]       connectionRetryTimeout: 120000,
[0-0]       connectionRetryCount: 3,
[0-0]       execArgv: [],
[0-0]       runnerEnv: {},
[0-0]       runner: 'local',
[0-0]       specFileRetries: 0,
[0-0]       specFileRetriesDelay: 0,
[0-0]       specFileRetriesDeferred: false,
[0-0]       reporterSyncInterval: 100,
[0-0]       reporterSyncTimeout: 5000,
[0-0]       cucumberFeaturesWithLineNumbers: [],
[0-0]       autoCompileOpts: [Object],
[0-0]       mochaOpts: [Object],
[0-0]       jasmineOpts: [Object],
[0-0]       cucumberOpts: [Object],
[0-0]       onPrepare: [],
[0-0]       onWorkerStart: [],
[0-0]       before: [],
[0-0]       beforeSession: [],
[0-0]       beforeSuite: [],
[0-0]       beforeHook: [],
[0-0]       beforeTest: [],
[0-0]       beforeCommand: [],
[0-0]       afterCommand: [],
[0-0]       afterTest: [],
[0-0]       afterHook: [],
[0-0]       afterSuite: [],
[0-0]       afterSession: [],
[0-0]       after: [],
[0-0]       onComplete: [],
[0-0]       onReload: [],
[0-0]       beforeFeature: [],
[0-0]       beforeScenario: [],
[0-0]       beforeStep: [],
[0-0]       afterStep: [],
[0-0]       afterScenario: [],
[0-0]       afterFeature: [],
[0-0]       baseUrl: 'https://company-qa-url/',
[0-0]       _: [Array],
[0-0]       '$0': 'node_modules\\@wdio\\cli\\bin\\wdio.js',
[0-0]       'config-path': 'wdio.conf.js',
[0-0]       ignoredWorkerServices: []
[0-0]     }
[0-0]   },
[0-0]   emit: [Function: bound ],
[0-0]   isReactElement: false,
[0-0]   addCommand: [Function (anonymous)],
[0-0]   overwriteCommand: [Function (anonymous)]
[0-0] }
...
[0-0] ##### wait for displayed...
...
[0-0] ##### elementId: ELEMENT-1

[dietergeerts]

So let me state my config and test, maybe you can see something missing or different about it.

package.json script

    "test:e2e": "wdio run wdio.conf.js",

package.json dependencies (listing more dependencies which are used for unit testing but not for e2e testing, because maybe this library does some auto-loading that conflicts?)

    "@wdio/cli": "^7.9.0",
    "@wdio/cucumber-framework": "^7.9.0",
    "@wdio/local-runner": "^7.9.0",
    "@wdio/spec-reporter": "^7.9.0",
    "@testing-library/jest-dom": "^5.11.9",
    "@testing-library/react": "^11.2.5",
    "@testing-library/react-hooks": "^5.1.2",
    "@testing-library/user-event": "^12.8.3",
    "@testing-library/webdriverio": "^3.0.4",
    "@cucumber/cucumber": "7.0.0",
    "expect-webdriverio": "^3.1.0",

wdio.conf.js

const webdriverDefaultConfig = require('./src/@company/tools-test-webdriver-io/lib/config');
const chromeDefaultConfig = require('./src/@company/tools-test-webdriver-io/lib/config-chrome');

/**
 * @see {@link https://webdriver.io/docs/configurationfile/}
 * @type {WebdriverIO.Config}
 */
module.exports.config = {
  ...webdriverDefaultConfig,
  capabilities: [{ ...chromeDefaultConfig }],
  baseUrl: 'https://company-qa-url/',
};

webdriverDefaultConfig

/**
 * @see {node_modules/@cucumber/cucumber/lib/cli/argv_parser.d.ts}
 * @see {@link https://github.com/cucumber/cucumber-js/blob/main/docs/cli.md}
 * @see {@link https://github.com/cucumber/cucumber-js/blob/main/docs/profiles.md}
 * @type {Partial<Omit<WebdriverIO.Config, 'capabilities'>>}
 */
module.exports = {
  specs: ['./src/**/*.feature'],
  framework: 'cucumber',
  cucumberOpts: {
    require: [require.resolve('../../tools-code-babel/lib/register'), './src/**/*.steps.ts'],
  },
  autoCompileOpts: { autoCompile: false },
  reporters: ['spec'],
};

chromeDefaultConfig

/**
 * @see {@link https://developers.google.com/web/updates/2017/04/headless-chrome}
 * @type {WebDriver.DesiredCapabilities}
 */
module.exports = {
  browserName: 'chrome',
  'goog:chromeOptions': {
    args: ['--headless', '--disable-gpu'],
  },
};

our-feature.feature

Feature: Auth login region select

  Scenario: The login landing page has a region select available
    Given I navigate to the company application
    Then I see instructions for selecting my region

auth-login.steps.ts

import { When } from '@cucumber/cucumber'
import { authLogin } from './auth-login.fragment'

When(/^I navigate to the company application$/, async () => {
  await authLogin.navigate()
  await authLogin.waitForLoaded()
})

auth-login.fragment.ts

export const authLogin = {
  get welcomeMessage() {
    return browser.$('h1*=Welcome')
  },
  get gotToLoginButton() {
    return browser.$('button*=login')
  },
  navigate() {
    return browser.url('/')
  },
  async waitForLoaded() {
    // This is the code that works, with just webdriver-io:
    // return this.welcomeMessage.waitForDisplayed()

    // See logs in previous comment
    const message = await browser.$('h1*=Welcome')
    console.log('#####', message)
    console.log('#####', 'wait for displayed...')
    await message.waitForDisplayed()
    console.log('#####', `elementId: ${message.elementId}`)

    // This is the code I want to have working, so we can create simpler selectors for selecting on text,
    //  and not be limited by the text selection of webdriverio, which is very specific.
    // const { findByText } = setupBrowser(browser)
    // const message = await findByText(/Welcome/i, {}, { timeout: 5000 })
    // return message.waitForDisplayed()
  },

[dietergeerts]

Giving this another try, without any luck. I see strange logs when I run the tests:

Could this be because cucumber is driven by @wdio directly?

[dietergeerts]

Another thing I notice is that when the element can't actually be found, I do get an error thrown that this is the case. When I use a selector that finds the element, I just get null returned every time.

import 'expect-webdriverio'
import { within } from '@testing-library/webdriverio'

// section is an actual WebdriverIO.Element, as I can see during debugging.
const modelName = await within(section).findByLabelText('Model name')
await expect(modelName).toBeDisplayed()

So it appears that the label can indeed be found, because otherwise I see an error being thrown, but the function gives back null every time, even if I use a timeout. So "modelName" here is null instead of a WebdriverIO.Element.

[olivierwilkinson]

Another thing I notice is that when the element can't actually be found, I do get an error thrown that this is the case. When I use a selector that finds the element, I just get null returned every time.

import 'expect-webdriverio'
import { within } from '@testing-library/webdriverio'

// section is an actual WebdriverIO.Element, as I can see during debugging.
const modelName = await within(section).findByLabelText('Model name')
await expect(modelName).toBeDisplayed()

So it appears that the label can indeed be found, because otherwise I see an error being thrown, but the function gives back null every time, even if I use a timeout. So "modelName" here is null instead of a WebdriverIO.Element.

Hmm that is very interesting! That shouldn't be happening for sure. Possibly I'm executing the wrong underlying dom-testing-library query then because findBy should never return null... I'll have to investigate.

Sorry for the radio-silence, I left this alone in the hope I'd have some fresh ideas when I got back to it. I'm going to have another go at reproducing the issues you are finding. I'll let you know if I find anything!

[olivierwilkinson]

Giving this another try, without any luck. I see strange logs when I run the tests:

Could this be because cucumber is driven by @wdio directly?

Oh these logs are because this library needs to inject some dependencies into the browser env, specifically dom-testing-library and simmer. The first execute call is to check whether dom-testing-library or simmer need to be injected, and the second execute call (and the gnarly looking log) are dom-testing-library being injected by executing it's umd distribution.

[olivierwilkinson]

@dietergeerts I unfortunately still haven't been able to reproduce what you are finding. Would you be interested in connecting over discord sometime to debug it together? Otherwise I think a repro would help, I'm struggling to find the problem as it stands 😭

[markusmeyer]

I have the same problem and after digging into it, I found out that the problem is the serialization of the returned elements in the .executeAsync call to the browser. If done manually with JSON.stringify, it complains about circular references, which are introduced by reactFiber* and reactProps* fields. If I replace the HTML element in the result by something else, it works fine (but only for queries that return a selector). So that would point in the direction of React Fibre causing these issues. Uncaught TypeError: Converting circular structure to JSON --> starting at object with constructor 'HTMLHeadingElement' | property '__reactFiber$9wgbdr4e8rq' -> object with constructor 'FiberNode' --- property 'stateNode' closes the circle at JSON.stringify (<anonymous>) at <anonymous>:1:18 (anonymous) @ VM1005:1

[olivierwilkinson]

Hi @markusmeyer,

Thank you for letting us know what you found! I have just checked and trying to serialize elements with circular references does cause the result to be null or an empty array so this could be the underlying issue. It looks like the issue is similar to this one, except in our case we need to serialize the element to send it back from the browser rather than a worker.

Anyways I think I have a fix which I will push up tomorrow 🥳

[olivierwilkinson]

I've realised I made a mistake when I reproduced this yesterday, I had crashed the toy react app I was using which is why the element wasn't found. It does look like Webdriver deals with circular references on HTML elements in some way when sending them back to from executeAsync. I'm going to dig deeper however as the issues in jest with React fibre seem to be difficult to reproduce. It could be that the underlying Webdriver version could have an impact too so I'll try exploring that. @markusmeyer Would you be able to provide a repo / snippet with the code that failed for you? I do think this is a good area to keep exploring!

[dietergeerts]

@olivierwilkinson , is there any further progress on this? We're starting to write more and more unit tests, so it would be easier to write them with these helpers, and more maintainable.

[olivierwilkinson]

@olivierwilkinson , is there any further progress on this? We're starting to write more and more unit tests, so it would be easier to write them with these helpers, and more maintainable.

Hi @dietergeerts, sadly not. I tried going through multiple underlying versions and examples but wasn't able to pin down the culprit. I would really appreciate a repro if you are able to provide me with one, I've been trying various things but it's more or less guess work at the moment. I know that providing a repro might be a decent amount of work but I promise I'll dedicate myself to getting it fixed as soon as possible afterwards! I really want to get it fixed also 🙏

[dietergeerts]

@olivierwilkinson , is there any template I can use to quickly create a repro?

[olivierwilkinson]

@olivierwilkinson , is there any template I can use to quickly create a repro?

Not for this project no, but I do have a branch that I was using as a test bed: test-cucumber-support. I wasn't able to reproduce the issue on that branch though and it's likely to be something we haven't thought of that is specific to your project, so probably the easiest way is to clone your project into a new repo and remove everything unrelated and/or sensitive.

[segovia]

By default, wdio uses chromedriver. When using that, findByRole works as expected, but if I switch out chromedriver and use selenium-standalone, findByRole only returns null. Maybe this is the difference between your configurations and a clue to why it is not working.

[olivierwilkinson]

By default, wdio uses chromedriver. When using that, findByRole works as expected, but if I switch out chromedriver and use selenium-standalone, findByRole only returns null. Maybe this is the difference between your configurations and a clue to why it is not working.

Thank you for this! 🙌 I'll look into that as soon as I can, probably Monday as I am away this weekend.

[segovia]

I did a bit more digging. I believe the following combination doesn't work well: selenium-standalone + shadow DOM + React (ReactFiber). So, I saw the code uses simmerjs to determine a selector for the element, however that doesn't work properly for shadow DOMs. I was able to modify the code to get a selector based off the shadow DOM root, but then I ran into the next issue where webdriverIO tries to serialize the response of the executeAsync and fails. Possibly related to the fact that there is a circular structure in the element (the reactFibre property mentioned in a previous comment).

In the end, I decided to make some modifications to the lib so it would work for my project. What I did is, I completely avoid sending the HTML element as part of the executeAsync response and now I just send a selector and I also mark the elements with a unique data attribute, so they can easily be found later. Since simmerjs is no longer used, I dropped the lib.

Here is the change: https://github.com/segovia/webdriverio-testing-library/commit/bd1e5088f41e589a9d3d9c2b9395033a11525a5b

@olivierwilkinson Do you think it makes sense to integrate some or all of these changes to this code base?

[olivierwilkinson]

I did a bit more digging. I believe the following combination doesn't work well:

selenium-standalone + shadow DOM + React (ReactFiber).

So, I saw the code uses simmerjs to determine a selector for the element, however that doesn't work properly for shadow DOMs. I was able to modify the code to get a selector based off the shadow DOM root, but then I ran into the next issue where webdriverIO tries to serialize the response of the executeAsync and fails. Possibly related to the fact that there is a circular structure in the element (the reactFibre property mentioned in a previous comment).

In the end, I decided to make some modifications to the lib so it would work for my project. What I did is, I completely avoid sending the HTML element as part of the executeAsync response and now I just send a selector and I also mark the elements with a unique data attribute, so they can easily be found later. Since simmerjs is no longer used, I dropped the lib.

Here is the change: https://github.com/segovia/webdriverio-testing-library/commit/bd1e5088f41e589a9d3d9c2b9395033a11525a5b

@olivierwilkinson Do you think it makes sense to integrate some or all of these changes to this code base?

Oh interesting! I think the strategy of adding a custom id to create a selector from might break some wdio behaviours with retrying stale selectors but I might be wrong, there should be a failing test if my suspicions are correct. Definitely raise a PR though because we can figure that out and iterate on what you have here 😁. I'll also try to set up a React app in the tests so we can check the Fiber stuff.

Thanks for digging further on this! Really appreciate it 🙌

[olivierwilkinson]

Just to keep everyone in the loop I have a reproduction locally, once I have time I'll update the tests to include a case for this scenario 😄

It is a problem when testing production React apps with the default WebdriverIO config, which uses Puppeteer under the hood to drive the browser.

Using the selenium-standalone or chromedriver service seems to fix it for me. @dietergeerts if that doesn't fix the issue for you please let me know. 🤞

Once I've updated the test setup with a failing test I'll move on to look at the PR @segovia has raised more closely 😄

Thanks everyone for adding info as they found it and being patient! 🙌

[olivierwilkinson]

@dietergeerts I have just published v3.0.7 which I think will fix the issues you have been finding! Big thanks to @segovia for the PR 🙌

Let me know if it works for you and then I'll close this 😄

[dietergeerts]

Apologies for not responding. We are currently in a merger of 2 companies, and thus there is no time/effort we can do on checking this out. When we can move further with this, I'll sure will check it out and let you know if it works or not in our use-case. Thx for all the work done!