Test runs differently when Test Runner window does not have focus

JasonFairchild commented 4 years ago

Current behavior:

The Cypress 'watch files' rerun appears to behave unexpectedly different compared to the rerun using the cypress UI button.

Here is what I observe:

When I press ctrl s and it reruns, the click on "Help" menu item appears to close the menu unexpectedly
When I press the UI rerun button, the menu always pans to the next menu that includes "order status" as I expect

This video is when running headless chrome. It shows the test getting further (requires that 500ms wait), the same way it does when I press the rerun UI button. It isn't really important, unless the code I give doesn't reproduce the issue for anyone else. Then maybe it will just help show that I am not crazy :). Its only the very end of the video that shows the action, the footage before is primarily from a very unfortunately slow page load.

reproduce.spec.js-compressed.mp4.zip

Desired behavior:

This isn't super important for me to have fixed, but I asked if someone might want to investigate and Gleb gave the go-ahead. Its very reproducible for me and may well highlight an unintended difference between the two types of reruns. I would expect this test to run the same way, regardless of how the rerun is initiated.

In general, I do notice differences between when cypress runs headless/completely newly pulling up the browser vs when it reruns with the browser already up. Mostly speed differences. I do wonder about all the reasons for why the speed difference can be so great. Is it mainly the proxy cache behavior or something else? I think this is a question for another issue or time though :) This is the first time I've ever noticed a significant difference between the watch rerun and UI rerun, however.

Test code to reproduce

Hopefully this works for others like it does for me 100% of the time. This is a public site, so should be fine to just run this test by itself:

describe("This odd issue", function() {
    beforeEach(function() {
        cy.viewport("iphone-x");
        cy.visit("https://help.doterra.com/");
        // This is because the initial page load often takes a long time
        cy.get('[data-aura-class="cHC_StartingCategories"]', { timeout: 25000 });
    });

    it.only("Navigates to help article with mobile header menu", function() {
        cy.get('[data-aura-class="cDT_HeaderMobileMenu"]')
          .click();
        cy
        // I've noticed that sometimes, after the menu pulls up, cypress will find the next element fast, and think it is not visible
        // Hence this wait, but it only seems to happen for me when first starting cypress after closing the browser entirely (or headless run)
        // In other words, this wait never matters when running with watch (ctrl s), or pressing the UI rerun button
        //   .wait(500)
          .get('.dtds-chevron-arrow-right[data-value="section-Help Mobile"]')
          .click();

        // Here is the problem: when I press ctrl s and it reruns, the click on "Help" menu item appears to close the menu
        // When I press the UI rerun button, the menu always pans to the next menu that includes "order status" as I expect
        // I've experimented putting waits all over the place, to see if it is a timing issue, but nothing seems to matter for this rerun distinction

        // This is the next part of my test I want to do but doesn't work for different reasons, though still related to .click() I think
        // cy.get('[data-target="Help_Order_Status"][data-aura-class="cDT_HeaderLink"]')
        //   .contains("Order Status")
        //   .click();
    });
});

For more context, this menu I'm trying to click on is made with flickity. It may be an implementation that is not super common. I can give more details if its thought it may be important.

Versions

Cypress 4.1.0 Chrome 80.0.3987.132 Windows 10 Pro 64-bit Build 18363

jennifer-shehane commented 4 years ago

This has to do with the current 'focus' of the window. Hitting ctrl+s in your IDE to 'rerun' the tests doesn't work because you are 'focused' on your IDE window instead of the Test Runner. Hitting ctrl+s will work if directly after you 'focus' in the Cypress window to run the tests.

This shouldn't happen, but it does seem like the application under tests's menu is specifically listening to focus events, so whenever you pull focus outside of the currently opened menu - it closes. I've attached a video of this below.

failing spec:

it("Navigates to help article with mobile header menu", () => {
  cy.viewport("iphone-x");
  cy.visit("https://help.doterra.com/");
  cy.get('[data-aura-class="cHC_StartingCategories"]', { timeout: 25000 });
  cy.get('[data-aura-class="cDT_HeaderMobileMenu"]')
    .click();
  cy.get('.dtds-chevron-arrow-right[data-value="section-Help Mobile"]')
    .click();
  // this assertion fails if the window is not focused
  cy.get('.mobile-menu-carousel').should('be.visible')
});

jennifer-shehane commented 4 years ago

Actually this is duplicate of https://github.com/cypress-io/cypress/issues/5023

JasonFairchild commented 4 years ago

Thanks, that makes more sense =). Always seemed fishy that the only difference I could tell was between rerunning different ways, haha. This information should help, though it did seem like this would have always been fine when running in CI.

cypress-io / cypress