Test doesn't wait long enough for search page results to load

[antoniym]

We observed a test instability while developing a Demokit test journeys in wdi5.

wdio-ui5-service version: 0.9.0-rc4

Manual test steps:

1. Navigate to https://openui5nightly.hana.ondemand.com/
2. In the "Search" field (upper right-hand side) type "OPA" and click Enter to submit the search.

Result: The test should verify the titles in the search result page ('All (1 - 25 of 132)' and 'Search Results for "OPA"'). However, if the results page loading takes a little longer, the test does not wait for it to load and fails. If we add an explicit sleep of 5 seconds (await browser.pause(5000)), the page loads completely and the test passes successfully.

Please find the test step as follows:

//Assertions searchResultsArePresent: async () => { var searchResultsPageTitle = await browser.asControl({ selector: { id: "sdk---searchPage--searchTitle", } });

    var searchResultsListTitle = await browser.asControl({
        selector: {
            controlType: "sap.m.Title",
            viewId: "sdk---searchPage",
            bindingPath: {
                propertyPath: "/visibleAllLength",
                modelName: "searchView"
            }
        }
    });

    await browser.pause(5000);
    await expect(await searchResultsListTitle.getText()).toMatch(/All \(1 - /);
    await expect(await (await searchResultsPageTitle.getWebElement()).$('span[id=sdk---searchPage--searchTitle-innerTitle]').getText()).toBe('Search Results for "OPA"');
}

[dominikfeininger]

Can you please provide your config

Like the wdio documentation explains (https://webdriver.io/docs/api/browser/pause/), please use (https://webdriver.io/docs/api/element/waitForExist) waitForExist to wait for an element to be loaded.

[antoniym]

const path = require('path');

exports.config = { runner: 'local', // // ==================== // Runner Configuration // ==================== // // // ================== // Specify Test Files // ================== // Define which test specs should run. The pattern is relative to the directory // from which wdio was called. // // The specs are defined as an array of spec files (optionally using wildcards // that will be expanded). The test for each spec file will be run in a separate // worker process. In order to have a group of spec files run in the same worker // process simply enclose them in an array within the specs array. // // If you are calling wdio from an NPM script (see https://docs.npmjs.com/cli/run-script), // then the current working directory is where your package.json resides, so wdio // will be called from there. // specs: [ './*.spec.js' ], // Patterns to exclude. exclude: [ // 'path/to/excluded/files' ], // // ============ // Capabilities // ============ // Define your capabilities here. WebdriverIO can run multiple capabilities at the same // time. Depending on the number of capabilities, WebdriverIO launches several test // sessions. Within your capabilities you can overwrite the spec and exclude options in // order to group specific specs to a specific capability. // // First, you can define how many instances should be started at the same time. Let's // say you have 3 different capabilities (Chrome, Firefox, and Safari) and you have // set maxInstances to 1; wdio will spawn 3 processes. Therefore, if you have 10 spec // files and you set maxInstances to 10, all spec files will get tested at the same time // and 30 processes will get spawned. The property handles how many capabilities // from the same test should run tests. // // maxInstances: 2, // // If you have trouble getting all important capabilities together, check out the // Sauce Labs platform configurator - a great tool to configure your capabilities: // https://saucelabs.com/platform/platform-configurator // capabilities: [{

    // maxInstances can get overwritten per capability. So if you have an in-house Selenium
    // grid with only 5 firefox instances available you can make sure that not more than
    // 5 instances get started at a time.
    maxInstances: 2,
    //
    browserName: 'chrome',
    'goog:chromeOptions': {
        args: ['--headless', '--disable-gpu', '--window-size=1600,1200', `--disable-dev-shm-usage`]
    },

    acceptInsecureCerts: true,

    // If outputDir is provided WebdriverIO can capture driver session logs
    // it is possible to configure which logTypes to include/exclude.
    // excludeDriverLogs: ['*'], // pass '*' to exclude all driver session logs
    // excludeDriverLogs: ['bugreport', 'server'],
}],

wdi5: {
    screenshotPath: path.join('wdio-ui5-service', 'test', 'report', 'screenshots'),
    logLevel: 'verbose',
    platform: 'browser',
    url: 'index.html',
    deviceType: 'web',
},

//
// ===================
// Test Configurations
// ===================
// Define all options that are relevant for the WebdriverIO instance here
//
// Level of logging verbosity: trace | debug | info | warn | error | silent
logLevel: 'info',
//
// Set specific log levels per logger
// loggers:
// - webdriver, webdriverio
// - @wdio/browserstack-service, @wdio/devtools-service, @wdio/sauce-service
// - @wdio/mocha-framework, @wdio/jasmine-framework
// - @wdio/local-runner
// - @wdio/sumologic-reporter
// - @wdio/cli, @wdio/config, @wdio/utils
// Level of logging verbosity: trace | debug | info | warn | error | silent
logLevels: {
    webdriver: 'warn',
},
//
// If you only want to run your tests until a specific amount of tests have failed use
// bail (default is 0 - don't bail, run all tests).
bail: 0,
//
// Set a base URL in order to shorten url command calls. If your `url` parameter starts
// with `/`, the base url gets prepended, not including the path portion of your baseUrl.
// If your `url` parameter starts without a scheme or `/` (like `some/path`), the base url
// gets prepended directly.
baseUrl: 'https://sapui5untested.int.sap.eu2.hana.ondemand.com/demokit/',
//
// Default timeout for all waitFor* commands.
waitforTimeout: 10000,
//
// Default timeout in milliseconds for request
// if browser driver or grid doesn't send response
connectionRetryTimeout: 120000,
//
// Default request retries count
connectionRetryCount: 3,
//
// Test runner services
// Services take over a specific job you don't want to take care of. They enhance
// your test setup with almost no effort. Unlike plugins, they don't add new
// commands. Instead, they hook themselves up into the test process.
services: ['chromedriver','ui5'],
before: function (capabilities, specs) {
    browser.setWindowSize(1600, 1200);
},

// Framework you want to run your specs with.
// The following are supported: Mocha, Jasmine, and Cucumber
// see also: https://webdriver.io/docs/frameworks
//
// Make sure you have the wdio adapter package for the specific framework installed
// before running any tests.
framework: 'mocha',
//
// The number of times to retry the entire specfile when it fails as a whole
// specFileRetries: 1,
//
// Delay in seconds between the spec file retry attempts
// specFileRetriesDelay: 0,
//
// Whether or not retried specfiles should be retried immediately or deferred to the end of the queue
// specFileRetriesDeferred: false,
//
// Test reporter for stdout.
// The only one supported by default is 'dot'
// see also: https://webdriver.io/docs/dot-reporter
reporters: ['spec'],

//
// Options to be passed to Mocha.
// See the full list at http://mochajs.org/
mochaOpts: {
    ui: 'bdd',
    timeout: 60000
}

}

[antoniym]

Config file provided above.

Like the wdio documentation explains (https://webdriver.io/docs/api/browser/pause/), please use (https://webdriver.io/docs/api/element/waitForExist) waitForExist to wait for an element to be loaded.

Thank you for the advise about waitForExist. Unfortunately, I tried to substitute await browser.pause(5000) in my code snippet with the following: await searchResultsListTitle.$().waitForExist({ timeout: 9000 }); await searchResultsPageTitle.$().waitForExist({ timeout: 9000 });

However, it doesn't wait for the search results to load.

[vobu]

Hi, could you please update to the most recent version of wdi5, namely v0.9.0-rc4.2 and check whether the issue still persists? Background: we're relying on the UI5-internal (async) event queue to wait until all async operations in the app have resolved. It might be worth investigating how the search on the UI5 site is done/implemented and check from there how to best test it. Hth, V.

[antoniym]

Hi,

Thank you very much for the feedback!

We have tested with the latest version v0.9.0-rc4.2. However, the instability is still observed. We have also noticed one more instability in this test suite, which is also observed in both versions:

Manual test steps:

1. Navigate to https://openui5nightly.hana.ondemand.com/tools
2. Verify that the title of the page is "Tools"

Result: The test should verify that the title of the page is "Tools" However, if the title loads slower than the sections below it, the test does not wait for it and fails.

Code:

toolsPageTitleIsDisplayed: async () => { var expectedToolsPageTitle = "Tools"; var actualToolsPageTitle = await browser.asControl({ selector: { id: "sdk---tools--landingImageHeadline" } });

    expect(await actualToolsPageTitle.getText()).toBe(expectedToolsPageTitle);
}

[vobu]

well, that might actually be more an issue with the UI5 site being served from the CDN and less an issue of wdi5 - because we had the same experience when connecting to the UI5 site in our tests. (Plus, you'll need to click "OK" on the cookie banner in your test prior to getting access to the site itself). That's why we've refactored those tests and now only connect to "our" servers in the tests. Long story short: please try not running your tests against the UI5 SDK site :)

[hmanchev]

Hi Volker, I don't understand your suggestion. UI5 usually consumed from the CDN - this is the more often case of UI5 users. This means that most of the wdi5 tests will be executed against apps, which are loading UI5 from the CDN.

In our case - the cookie dialog can be easily handled, but the issue here is that the title of the "Tools" page is loading slowly and wdi5 is not synchronizing with the page. I think that such cases can happen in different applications and there should be general solution, but not loading from CDN is not variant. Our test is supposed to be executed against "UI5 SDK site" and I don't understand why wdi5 could require specific app or hostting to synchronize fine.

In my opinion, the test framework shouldn't be dependant of the app or hosting - whether is loading fast or slow.

[vobu]

@hmanchev that's a misunderstanding here - I was talking about the www site itself, so https://(open|sap)ui5.hana.ondemand.com, not the bootstrapping of UI5 via the CDN. And the UI5 site itself behaved flaky in our tests.

[dominikfeininger]

@hmanchev still ongoing work here?

[github-actions[bot]]

closed 📴 because silencio 🤫 since an additional 14 days after staleness 📠