Protractor E2E test scripts are passing inconsistently under Mocha framework, classic chain promissing

[bharathssubramanian]

Hi there!

Protractor E2E test scripts using Mocha BDD framework does not pass every time (pass rate is very less, inconsistent) when we run both in local and Sauce Labs.

We are using classic promise chaining concept instead of promise manager in our E2E test scripts. We reffered here https://github.com/SeleniumHQ/selenium/wiki/WebDriverJs

We used two different types of approach using MochaDone(done) callback and with try/catch block returning a promise.

Bug report

• Node Version: 8.9.4 ``

• Protractor Version: 5.3.0 ``

• Mocha Version: 3.5.0 ``

• Angular Version: 1.6``

• Browser(s): Google Chrome 62,63,64``

• Operating System and Version: Windows 7, 10 ``

• Your protractor configuration file

  return [
      {
          browserName: 'chrome',
          chromeOptions: {
              args: [
                  'use-fake-device-for-media-stream',
                  'use-fake-ui-for-media-stream'
              ]
          },
          idleTimeout: 2000,
          name,
          platform: 'Windows 10'
      }
  ];

  } else { return [ { browserName: 'chrome', chromeOptions: { args: [ 'use-fake-device-for-media-stream', 'use-fake-ui-for-media-stream' ] } } ]; }

}

export let config: IBasePtorConfig = { allScriptsTimeout: 60000, debug: false,

framework: 'mocha',

getMultiCapabilities: getBaseRequiredMultiCapabilities,
mochaOpts: {
    reporter: getMochaReporter(),
    timeout: 60000
},

noGlobals: true,

onPrepare: () => {
    return browser
        .getProcessedConfig()
        .then((conf: IBasePtorConfig) => {
            // EC 09-01-17:  Verifies required params are set on the config object.
            let requiredParamsProperties: Array<string> = ['automationEnv', 'dlapAdminPassword'];
            let message: string = `

SELENIUM_PROMISE_MANAGER: false, //E.C. 07-14-17 - if tests work, we know it is migrated
specs: ['./**/*.e2e.spec.js']

};

• A relevant example test

  beforeEach((done: MochaDone) => { dashboard = new Dashboard(); login = new AutoCommon.page.Login(); nav = new AutoCommon.page.Navigation();

  AutoCommon.util.log('Navigating to Log In page and attempting to log in.');
  
  login.login(domain.userspace, teacher.username, teacher.password)
      .then(() => AutoCommon.util.log('Logged in.'))
      .then(() => nav.navigate(NavigationArea.Dashboard))
      .then(() => AutoCommon.util.log('Navigated to Dashboard.'))
      .then(() => done())
      .catch((err) => done(err));

  });

  describe('When viewing the \'On Schedule\' donut table', () => { it('should show the correct students in the \'On Schedule\' donut table.', (done: MochaDone) => { // arrange

      // act
      dashboard.pace.tab.click()
          .then(() => AutoCommon.util.log('Selected the Pace tab.'))
          .then(() => browser.waitForAngularEnabled(false))
          .then(() => AutoCommon.util.log('Desynchronized from angular.'))
          .then(() => browser.wait(ExpectedConditions.visibilityOf(dashboard.pace.onSchedule), 30000, 'Waiting for On Schedule donut to render.'))
          .then(() => dashboard.pace.onSchedule.click())
          .then(() => AutoCommon.util.log('Selected On Schedule Donut.'))
          .then(() => browser.wait(ExpectedConditions.visibilityOf(dashboard.performance.grid.tbody), 30000, 'Waiting for the student grid to render.'))
          .then(() => dashboard.pace.grid.studentLinks.count())
  
          // assert
          .then((studentCount: number) => expect(studentCount).to.equal(global['dashboardData'].users.length - 1))
          .then(() => done());
  });

  }); });

Sample2:

describe('Dashboard Pace Tab', () => { let course: DlapInterfaces.ICourse, domain: DlapInterfaces.IDomain, teacher: DlapInterfaces.IUser, student: DlapInterfaces.IUser;

beforeEach(done => {
    dashboard = new Dashboard();
    login = new AutoCommon.page.Login();
    nav = new AutoCommon.page.Navigation();

    AutoCommon.util.log('Navigating to Log In page and attempting to log in.');
    try {
        return login.login(domain.userspace, teacher.username, teacher.password)
            .then(() => AutoCommon.util.log('Logged in.'))
            .then(() => nav.navigate(NavigationArea.Dashboard))
            .then(() => browser.ignoreSynchronization = true)
            .then(() => AutoCommon.util.log('Navigated to Dashboard.'))
            .then(() => {
                done();
            });
    } catch (e) {
        return done(e);
    }
});

describe('Teacher: When viewing the \'On Schedule\' donut table', () => {
    // Test case id - 34513
    it('Should show the correct students in the \'On Schedule\' donut table.', done => {
        // act    
        try {
            return browser.wait(ExpectedConditions.invisibilityOf(dashboard.pace.loadingBar), 60000, 'Waiting for On Schedule donut to render.')
                .then(() => browser.wait(ExpectedConditions.elementToBeClickable(dashboard.pace.tab), 60000, 'Waiting for On Schedule donut to render.'))
                //Implemented this time out to solve the sync issues. Will follow up and implement a better solution in future.
                //.then(() => browser.sleep(10000))
                .then(() => dashboard.pace.tab.click())
                .then(() => browser.wait(ExpectedConditions.invisibilityOf(dashboard.pace.loadingBar), 60000, 'Waiting for On Schedule donut to render.'))
                .then(() => browser.wait(ExpectedConditions.elementToBeClickable(dashboard.pace.tab), 60000, 'Waiting for On Schedule donut to render.'))
                .then(() => AutoCommon.util.log('Selected the Pace tab.'))
                .then(() => browser.wait(ExpectedConditions.elementToBeClickable(dashboard.pace.onSchedule), 60000, 'Waiting for On Schedule donut to render.'))
                //Implemented this time out to solve the sync issues. Will follow up and implement a better solution in future.
                // .then(() => browser.sleep(10000))
                .then(() => dashboard.pace.onSchedule.click())
                .then(() => AutoCommon.util.log('Selected On Schedule Donut.'))
                .then(() => browser.wait(ExpectedConditions.visibilityOf(dashboard.performance.grid.tbody), 30000, 'Waiting for the student grid to render.'))
                .then(() => dashboard.pace.grid.studentLinks.count())

                // assert
                .then((studentCount: number) => expect(studentCount).to.equal(global['dashboardData'].users.length - 2))
                .then(() => {
                    done();
                });
        } catch (e) {
            return done(e);
        }

    });
});

});

• Output from running the test

1) Dashboard Newly Enrolled Tab - Admin "before each" hook: ret for "Should verify dashboard newly enrolled tab show special status(IEP,504,ELL,NCAA) for admin": Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

2) When viewing the grid on the Pace or Performance tabs should take the user to the student's grade book upon selecting the student's name: Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

3) When viewing the grid on the Pace or Performance tabs should take the user to the course's grade book selecting the course's name: Error: Timeout of 60000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves.

4) Dashboard Newly Enrolled Tab "before each" hook: ret for "Should verify dashboard newly enrolled tab show special status(IEP,504,ELL,NCAA) for teacher": Error: Error while waiting for Protractor to sync with the page: "both angularJS testability and angular testability are undefined. This could be either because this is a non-angular page or because your test involves client-side navigation, which can interfere with Protractor's bootstrapping. See http://git.io/v4gXM for details" at runWaitForAngularScript.then (node_modules/protractor/built/browser.js:463:23) at at process._tickCallback (internal/process/next_tick.js:188:7)

5) Dashboard Pace Tab Teacher: When viewing the 'On Schedule' donut table Should show the correct students in the 'On Schedule' donut table.: WebDriverError: unknown error: unhandled inspector error: {"code":-32000,"message":"Cannot find context with specified id"} (Session info: chrome=63.0.3239.132) (Driver info: chromedriver=2.35.528157 (4429ca2590d6988c0745c24c8858745aaaec01ef),platform=Mac OS X 10.13.2 x86_64) at Object.checkLegacyResponse (node_modules/selenium-webdriver/lib/error.js:546:15) at parseHttpResponse (node_modules/selenium-webdriver/lib/http.js:509:13) at doSend.then.response (node_modules/selenium-webdriver/lib/http.js:441:30) at at process._tickCallback (internal/process/next_tick.js:188:7)

• Steps to reproduce the bug

• The URL you are running your tests against (if relevant) It is angular app integrated inside another angular app but it will miss the app directive

  So Protractor is not identifying angular load when we use Mocha. However it is working well with Jasmine 2.0 (ignoreSynchronization, waitForAngular etc)

[bharathssubramanian]

Problem Statement: Test execution results are inconsistent when we use “MochaDone” for resolving promises and “waitForAngularEnabled ()” to handle protractor sync with AngulaJS application. Kindly let us know if you have faced these challenges in the past (or) in the current and how did you overcome this issue for smooth UI automation.

Have tried with four different options to resolve promise issues across features (angularJS SPA applications) inside Buzz angularJS application (parent).

1. PromiseAll()
2. “Try” “Catch” with single return statement for every “It” function (Expected wait & waitForAngularEnabled())
3. Async/await (https://github.com/SeleniumHQ/selenium/wiki/WebDriverJs )
4. Promise chaining with waitForAngularEnabled()

Note: We are automating angularJS application where many angularJS SPA integrated inside the parent angularJS app and inside each SPA, every page has “ng-app” as root element.