EPIC: Improve Cypress testing

AliceR commented 4 years ago

Every now and then the tests fail when running on CI, triggering them again will let them pass. This shouldn't happen, false alarms reduce our sensitivity for failing tests.

How can we make our tests more stable? I need to learn more on Best Practices.

AliceR commented 4 years ago

Examples:

1 failing
  1) Homepage
       focus areas
         navigates to the campaign list with the focus area as filter applied:
     AssertionError: Timed out retrying: Expected to find element: `[data-cy=filter-chip]`, but never found it. Queried from element: <main>
  AssertionError: Expected to find element: `[data-cy=filter-chip]`, but never found it.
      at ensureExistence (http://localhost:8000/__cypress/runner/cypress_runner.js:137673:32)
      at Object.ensureElExistence (http://localhost:8000/__cypress/runner/cypress_runner.js:137691:12)
      at ensureExistence (http://localhost:8000/__cypress/runner/cypress_runner.js:127867:23)
      at onFailFn (http://localhost:8000/__cypress/runner/cypress_runner.js:127905:9)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
      at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:7000:31)
      at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:7057:18)
      at Promise._settlePromise0 (http://localhost:8000/__cypress/runner/cypress_runner.js:7102:10)
      at Promise._settlePromises (http://localhost:8000/__cypress/runner/cypress_runner.js:7178:18)
      at _drainQueueStep (http://localhost:8000/__cypress/runner/cypress_runner.js:3772:12)
      at _drainQueue (http://localhost:8000/__cypress/runner/cypress_runner.js:3765:9)
      at Async.../../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:3781:5)
      at Async.drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:3651:14)

or

1 failing
  1) Explore
       platforms
         displays the number of items to explore:
     AssertionError: Timed out retrying: expected 'Showing 10 campaigns' to match /Showing [0-9]+ platforms/i
      at applyChainer (http://localhost:8000/__cypress/runner/cypress_runner.js:131898:32)
      at http://localhost:8000/__cypress/runner/cypress_runner.js:131940:16
      at arrayReduce (http://localhost:8000/__cypress/runner/cypress_runner.js:18578:21)
      at Function.reduce (http://localhost:8000/__cypress/runner/cypress_runner.js:27576:14)
      at applyChainers (http://localhost:8000/__cypress/runner/cypress_runner.js:131929:22)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
      at Function.Promise.attempt.Promise.try (http://localhost:8000/__cypress/runner/cypress_runner.js:6339:29)
      at Context.shouldFn (http://localhost:8000/__cypress/runner/cypress_runner.js:131946:26)
      at Context.should (http://localhost:8000/__cypress/runner/cypress_runner.js:131964:23)
      at http://localhost:8000/__cypress/runner/cypress_runner.js:127819:39
      at assertions (http://localhost:8000/__cypress/runner/cypress_runner.js:128079:14)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
      at Object.gotValue (http://localhost:8000/__cypress/runner/cypress_runner.js:8209:18)
      at Object.gotAccum (http://localhost:8000/__cypress/runner/cypress_runner.js:8196:25)
      at Object.tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
      at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:7000:31)

or

1 failing
  1) Campaign
       the platforms section
         displays a platform carousel:
     CypressError: Timed out retrying: `cy.click()` failed because this element is `disabled`:
`<button aria-label="next" type="button" style="border: 0px; background: rgba(0, 0, 0, 0.4); color: white; padding: 10px; text-transform: uppercase; cursor: not-allowed; opacity: 0.3;" disabled="">></button>`
Fix this problem, or use `{force: true}` to disable error checking.
https://on.cypress.io/element-cannot-be-interacted-with
      at cypressErr (http://localhost:8000/__cypress/runner/cypress_runner.js:146621:16)
      at cypressErrByPath (http://localhost:8000/__cypress/runner/cypress_runner.js:146630:10)
      at Object.throwErrByPath (http://localhost:8000/__cypress/runner/cypress_runner.js:146593:11)
      at Object.ensureNotDisabled (http://localhost:8000/__cypress/runner/cypress_runner.js:137570:24)
      at runAllChecks (http://localhost:8000/__cypress/runner/cypress_runner.js:127486:14)
      at retryActionability (http://localhost:8000/__cypress/runner/cypress_runner.js:127542:16)
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
      at Function.Promise.attempt.Promise.try (http://localhost:8000/__cypress/runner/cypress_runner.js:6339:29)
      at tryFn (http://localhost:8000/__cypress/runner/cypress_runner.js:140680:21)
      at whenStable (http://localhost:8000/__cypress/runner/cypress_runner.js:140715:12)
      at http://localhost:8000/__cypress/runner/cypress_runner.js:140259:16
      at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
      at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:7000:31)
      at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:7057:18)
      at Promise._settlePromise0 (http://localhost:8000/__cypress/runner/cypress_runner.js:7102:10)
      at Promise._settlePromises (http://localhost:8000/__cypress/runner/cypress_runner.js:7182:18)

AliceR commented 4 years ago

Ideas:

save screenshots on ci to be able to inspect the cause of failure
increase timeout (see defaults here)
run e2e:ci tests on the built project instead of development server

AliceR commented 4 years ago

Possibly related issues:

Ideas to solve from issue comments:

In my experience, most of these cases (when tests fails only in headless mode) it's due the faster speed.

Example: in my app, after the login, we have a request to get more info about the user, and only then store some data in local storage and cookies.

While testing in Chrome, or even with Electron, it works. Headless, it was randomly failing (because the request was not completed, so user info was not saved, redirecting to login again).

To fix, I just include a check for some element that should be displayed only after the user request is finished, like one of item of the logged user menu, like cy.contains('Home').

(https://github.com/cypress-io/cypress/issues/1297#issuecomment-417385572)

... One of our engineers, @bahmutov , wrote a great blog post about this you can read here: https://www.cypress.io/blog/2019/01/22/when-can-the-test-click/

I would write tests using cy.route() and cy.wait() to verify that the XHR requests have responded completely before attempting to assert that the row elements are visible. ...

Recommended reading by comment author: https://docs.cypress.io/guides/guides/network-requests.html#Testing-Strategies

(https://github.com/cypress-io/cypress/issues/4639#issuecomment-507936428)

AliceR commented 4 years ago

[20329:0824/200854.737644:FATAL:memory.cc(22)] Out of memory. size=13447168
We detected that the Chromium Renderer process just crashed.

This is the equivalent to seeing the 'sad face' when Chrome dies.

This can happen for a number of different reasons:

- You wrote an endless loop and you must fix your own code
- There is a memory leak in Cypress (unlikely but possible)
- You are running Docker (there is an easy fix for this: see link below)
- You are running lots of tests on a memory intense application
- You are running in a memory starved VM environment
- There are problems with your GPU / GPU drivers
- There are browser bugs in Chromium

You can learn more including how to fix Docker here:

https://on.cypress.io/renderer-process-crashed

AliceR commented 4 years ago

Some thoughts for further conversation:

I played around with this exact change in the past. It seems to resolve all flakiness in the load/test timing, but comes at the cost of testing the navigation elements.

The question I had when working on this, is how do we balance thorough testing against consistent results? Is it worth removing flaky tests we can't stabilize?

https://github.com/developmentseed/airborne-metadata-catalog-internal/pull/105#discussion_r476513941

AliceR commented 4 years ago

Another example of a flaky test:

Running:  navigation.test.js                                                            (12 of 13)

  Navigation
    ✓ going back or forward in the browser's history is possible (969ms)
    1) reloading the page maintains the url
    ✓ visiting a remote url directly loads the respective page (494ms)

  2 passing (6s)
  1 failing

  1) Navigation
       reloading the page maintains the url:
     AssertionError: Timed out retrying: expected '/' to include 'contact'
      at Context.eval (http://localhost:8000/__cypress/tests?p=cypress/e2e/navigation.test.js:57:29)

This test is based off of the cypress kitchensink examples, https://github.com/cypress-io/cypress-example-kitchensink/blob/master/cypress/integration/examples/navigation.spec.js Why is this one flaky??

AliceR commented 3 years ago

After upgrading to Cypress 6 we could use intercept() to mock api requests. Useful here: https://github.com/developmentseed/admg-inventory/blob/develop/cypress/e2e/filter-and-search.test.js#L212

LanesGood commented 2 months ago

We've removed cypress from the project; now using playwright and jest. See #457

NASA-IMPACT / admg-casei

EPIC: Improve Cypress testing #33