Some tests flake only if test runner's browser loses focus (or run headlessly)

[blunderdome]

Current behavior:

We have a number of tests which:

• Always pass when run in Chrome 76 with the browser in focus the whole time
• Always pass when run in Electron 61 with the browser in focus the whole time
• Often fail when run in Chrome 76 without the browser in focus
• Often fail when run in Electron 61 without the browser in focus
• Often fail when run headless

Desired behavior:

Browser does not need to stay in focus; tests also pass headlessly.

Steps to reproduce: (app code and test code)

it('is just an example', function () {
  cy.visit('https://noredink.com/logout')
  cy.contains('Log In').click()
  cy.findField('Email or username').type('example_user')
  cy.findField('Password').type('easypassword')
  cy.contains('button', 'Log in').click()
})

Here is a simple case - logging into our production app. This passes if you keep the browser in focus but sometimes fails if it loses focus (or is run headlessly):

Video: in focus and passing

Video: out of focus and failing

We have other examples but they involve the admin side of our site and we probably shouldn't share them here. If you need more examples, let me know and we'll try to find something else that's appropriate to share.

Versions

Cypress: 3.4.1 OS: macOS Mojave 10.14.5 Browser(s): Chrome 76 and Electron 61

Other

https://github.com/cypress-io/cypress/issues/1892 may be relevant.

[jennifer-shehane]

This has been something @Bkucera has been working on. It is not an easy issue to solve.

Thanks for providing a reproducible issue. I wasn't able to reproduce it in the few short times I ran this - could you explain what 'often fails' looks like? Do we have to run the test 5 times to see the error? 100 times?

Test code:

it('is just an example', function () {
  cy.visit('https://noredink.com/logout')
  cy.contains('Log In').click()
  cy.get('#Nri-Ui-TextInput-Email-or-username').type('example_user')
  cy.get("#Nri-Ui-TextInput-Password").type('easypassword')
  cy.contains('button', 'Log in').click()
})

[blunderdome]

Hmm, I'm not sure. For me it fails almost every time, but a colleague said it was working for him, so there may be something about our setup or what we're doing that isn't exactly the same.

The tiny example I gave runs pretty quickly, so if you don't switch focus away from the browser right away, it will already have passed. You could have it wait for a few seconds if you want to make sure you've switched focus away before it gets to that login menu.

Let me know if there's anything else about my browser, computer, or config that would be helpful to share ... can also jump on a call with someone if it would be helpful to see it live.

[vesper8]

not sure if related but I am also noticing that tests seem to randomly fail when running headless but run perfectly and consistently when run with the --headed flag. And it's not the same tests that randomly fail, it's quite random, but only happens when running headless.

not to mention that tests complete slower (sometimes MUCH slower) when running headless

[kuceb]

@blunderdome I think your cy.findField command must be sensitive to window focus, and that's the reason your test fails. We also can't run your code for that reason. Is that command from https://github.com/testing-library/cypress-testing-library ?

try using the code @jennifer-shehane wrote, and see if the issue goes away.

[fr0]

I am experiencing something like this related to the focusout event. If the browser window has focus while the test is running, it passes. Otherwise, it fails.

Specifically, I have a popup that is hidden when it loses focus (when it receives focusout). Is there a known workaround for this?

[kuceb]

@fr0 this should have been fixed in 3.4.1, could you give a reproducible?

[fr0]

@fr0 this should have been fixed in 3.4.1, could you give a reproducible?

Sure, here you go: https://github.com/fr0/cypress-focus-test

[ginna-baker]

This is a big issue for us testing our accessibility features, particular the "Skip to Main Content" feature in which the button is only visible on focus.

[kuceb]

I have this in my backlog, or PRs welcome. thanks for the reproducible @fr0

[jennifer-shehane]

Another example, this UI also seems to have some logic listening to some sort of 'focus' event.

failing spec:

it("Navigates to help article with mobile header menu", () => {
  cy.viewport("iphone-x");
  cy.visit("https://help.doterra.com/");
  cy.get('[data-aura-class="cHC_StartingCategories"]', { timeout: 25000 });
  cy.get('[data-aura-class="cDT_HeaderMobileMenu"]')
    .click();
  cy.get('.dtds-chevron-arrow-right[data-value="section-Help Mobile"]')
    .click();
  // this assertion fails if the window is not focused
  cy.get('.mobile-menu-carousel').should('be.visible')
});

[boda-sh]

Same issue here with the .click() method, only the error message is different and this problem doesn't exist with Electron 78

Behaviour

• Always pass when run in Chrome 80 with the browser in focus the whole time
• Always pass when run in Electron 78 with the browser in focus the whole time
• Always pass when run in Electron 78 without the browser in focus
• Often fail when run in Chrome 80 without the browser in focus
• Often fail when run in CI/headless mode

Steps to reproduce

Simplified code with problem

• Codesandbox link

Cypress test code

describe("Cypress fails when not in focus", () => {
  it("Try to click on the modal text button", () => {
    // deployed from codesandbox
    cy.visit("https://csb-lu86t-40qtfxquh.now.sh/");

    // gives us time to minimize (de-focus) the Cypress window
    cy.wait(5000);

    cy.get("[data-cy=show-modal]").click();

    // error here!
    cy.get("[data-cy=modal-text]").click();
  });
});

Error message (only with Chrome 80 not in focus or CI/headless)

CypressError: Timed out retrying: cy.click() failed because this element is not
visible:

<button data-cy="modal-text">i used ...</button>

This element '<button>' is not visible because its ancestor has 'position: fixed'
CSS property and it is overflowed by other elements. How about scrolling to the
element with cy.scrollIntoView()?

Fix this problem, or use {force: true} to disable error checking.

https://on.cypress.io/element-cannot-be-interacted-with

Versions Cypress: 3.8.2 OS: macOS Catalina 10.15.4 Browser(s): Chrome 80 and Electron 78

[einfallstoll]

@bodazhao I have one test case which often fails in CI/headless with the same setup as you do:

• Select element, click it to show modal
• Select element within modal, click it
• Error, because it's invisible

[juanvillegas]

Is there any way to approach this? We are having issues with native checkboxes, if the window doesn't have focus the checkboxes aren't found by the cy.get() command.

[cmraible]

I've been experiencing similar "flakiness" in my tests, only when the window is in the background or minimized. The tests that fail all assert on visibility or click elements like Modals or Dropdowns, and they fail with an error stating that the element has opacity set to 0 (and thus is not visible). This seems to jive with the other examples in the thread, which all seem to be Modals/Dropdowns or similar.

From Mozilla:

requestAnimationFrame() calls are paused in most browsers when running in background tabs or hidden