Closed AliceR closed 2 months ago
Examples:
1 failing
1) Homepage
focus areas
navigates to the campaign list with the focus area as filter applied:
AssertionError: Timed out retrying: Expected to find element: `[data-cy=filter-chip]`, but never found it. Queried from element: <main>
AssertionError: Expected to find element: `[data-cy=filter-chip]`, but never found it.
at ensureExistence (http://localhost:8000/__cypress/runner/cypress_runner.js:137673:32)
at Object.ensureElExistence (http://localhost:8000/__cypress/runner/cypress_runner.js:137691:12)
at ensureExistence (http://localhost:8000/__cypress/runner/cypress_runner.js:127867:23)
at onFailFn (http://localhost:8000/__cypress/runner/cypress_runner.js:127905:9)
at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:7000:31)
at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (http://localhost:8000/__cypress/runner/cypress_runner.js:7102:10)
at Promise._settlePromises (http://localhost:8000/__cypress/runner/cypress_runner.js:7178:18)
at _drainQueueStep (http://localhost:8000/__cypress/runner/cypress_runner.js:3772:12)
at _drainQueue (http://localhost:8000/__cypress/runner/cypress_runner.js:3765:9)
at Async.../../node_modules/bluebird/js/release/async.js.Async._drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:3781:5)
at Async.drainQueues (http://localhost:8000/__cypress/runner/cypress_runner.js:3651:14)
or
1 failing
1) Explore
platforms
displays the number of items to explore:
AssertionError: Timed out retrying: expected 'Showing 10 campaigns' to match /Showing [0-9]+ platforms/i
at applyChainer (http://localhost:8000/__cypress/runner/cypress_runner.js:131898:32)
at http://localhost:8000/__cypress/runner/cypress_runner.js:131940:16
at arrayReduce (http://localhost:8000/__cypress/runner/cypress_runner.js:18578:21)
at Function.reduce (http://localhost:8000/__cypress/runner/cypress_runner.js:27576:14)
at applyChainers (http://localhost:8000/__cypress/runner/cypress_runner.js:131929:22)
at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
at Function.Promise.attempt.Promise.try (http://localhost:8000/__cypress/runner/cypress_runner.js:6339:29)
at Context.shouldFn (http://localhost:8000/__cypress/runner/cypress_runner.js:131946:26)
at Context.should (http://localhost:8000/__cypress/runner/cypress_runner.js:131964:23)
at http://localhost:8000/__cypress/runner/cypress_runner.js:127819:39
at assertions (http://localhost:8000/__cypress/runner/cypress_runner.js:128079:14)
at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
at Object.gotValue (http://localhost:8000/__cypress/runner/cypress_runner.js:8209:18)
at Object.gotAccum (http://localhost:8000/__cypress/runner/cypress_runner.js:8196:25)
at Object.tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:7000:31)
or
1 failing
1) Campaign
the platforms section
displays a platform carousel:
CypressError: Timed out retrying: `cy.click()` failed because this element is `disabled`:
`<button aria-label="next" type="button" style="border: 0px; background: rgba(0, 0, 0, 0.4); color: white; padding: 10px; text-transform: uppercase; cursor: not-allowed; opacity: 0.3;" disabled="">></button>`
Fix this problem, or use `{force: true}` to disable error checking.
https://on.cypress.io/element-cannot-be-interacted-with
at cypressErr (http://localhost:8000/__cypress/runner/cypress_runner.js:146621:16)
at cypressErrByPath (http://localhost:8000/__cypress/runner/cypress_runner.js:146630:10)
at Object.throwErrByPath (http://localhost:8000/__cypress/runner/cypress_runner.js:146593:11)
at Object.ensureNotDisabled (http://localhost:8000/__cypress/runner/cypress_runner.js:137570:24)
at runAllChecks (http://localhost:8000/__cypress/runner/cypress_runner.js:127486:14)
at retryActionability (http://localhost:8000/__cypress/runner/cypress_runner.js:127542:16)
at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
at Function.Promise.attempt.Promise.try (http://localhost:8000/__cypress/runner/cypress_runner.js:6339:29)
at tryFn (http://localhost:8000/__cypress/runner/cypress_runner.js:140680:21)
at whenStable (http://localhost:8000/__cypress/runner/cypress_runner.js:140715:12)
at http://localhost:8000/__cypress/runner/cypress_runner.js:140259:16
at tryCatcher (http://localhost:8000/__cypress/runner/cypress_runner.js:9065:23)
at Promise._settlePromiseFromHandler (http://localhost:8000/__cypress/runner/cypress_runner.js:7000:31)
at Promise._settlePromise (http://localhost:8000/__cypress/runner/cypress_runner.js:7057:18)
at Promise._settlePromise0 (http://localhost:8000/__cypress/runner/cypress_runner.js:7102:10)
at Promise._settlePromises (http://localhost:8000/__cypress/runner/cypress_runner.js:7182:18)
Ideas:
Possibly related issues:
Ideas to solve from issue comments:
In my experience, most of these cases (when tests fails only in headless mode) it's due the faster speed.
Example: in my app, after the login, we have a request to get more info about the user, and only then store some data in local storage and cookies.
While testing in Chrome, or even with Electron, it works. Headless, it was randomly failing (because the request was not completed, so user info was not saved, redirecting to login again).
To fix, I just include a check for some element that should be displayed only after the user request is finished, like one of item of the logged user menu, like
cy.contains('Home')
.
(https://github.com/cypress-io/cypress/issues/1297#issuecomment-417385572)
... One of our engineers, @bahmutov , wrote a great blog post about this you can read here: https://www.cypress.io/blog/2019/01/22/when-can-the-test-click/
I would write tests using
cy.route()
andcy.wait()
to verify that the XHR requests have responded completely before attempting to assert that therow
elements are visible. ...
Recommended reading by comment author: https://docs.cypress.io/guides/guides/network-requests.html#Testing-Strategies
(https://github.com/cypress-io/cypress/issues/4639#issuecomment-507936428)
[20329:0824/200854.737644:FATAL:memory.cc(22)] Out of memory. size=13447168
We detected that the Chromium Renderer process just crashed.
This is the equivalent to seeing the 'sad face' when Chrome dies.
This can happen for a number of different reasons:
- You wrote an endless loop and you must fix your own code
- There is a memory leak in Cypress (unlikely but possible)
- You are running Docker (there is an easy fix for this: see link below)
- You are running lots of tests on a memory intense application
- You are running in a memory starved VM environment
- There are problems with your GPU / GPU drivers
- There are browser bugs in Chromium
You can learn more including how to fix Docker here:
https://on.cypress.io/renderer-process-crashed
Some thoughts for further conversation:
I played around with this exact change in the past. It seems to resolve all flakiness in the load/test timing, but comes at the cost of testing the navigation elements.
The question I had when working on this, is how do we balance thorough testing against consistent results? Is it worth removing flaky tests we can't stabilize?
https://github.com/developmentseed/airborne-metadata-catalog-internal/pull/105#discussion_r476513941
Another example of a flaky test:
Running: navigation.test.js (12 of 13)
Navigation
✓ going back or forward in the browser's history is possible (969ms)
1) reloading the page maintains the url
✓ visiting a remote url directly loads the respective page (494ms)
2 passing (6s)
1 failing
1) Navigation
reloading the page maintains the url:
AssertionError: Timed out retrying: expected '/' to include 'contact'
at Context.eval (http://localhost:8000/__cypress/tests?p=cypress/e2e/navigation.test.js:57:29)
This test is based off of the cypress kitchensink examples, https://github.com/cypress-io/cypress-example-kitchensink/blob/master/cypress/integration/examples/navigation.spec.js Why is this one flaky??
After upgrading to Cypress 6 we could use intercept() to mock api requests. Useful here: https://github.com/developmentseed/admg-inventory/blob/develop/cypress/e2e/filter-and-search.test.js#L212
We've removed cypress from the project; now using playwright and jest. See #457
Every now and then the tests fail when running on CI, triggering them again will let them pass. This shouldn't happen, false alarms reduce our sensitivity for failing tests.
How can we make our tests more stable? I need to learn more on Best Practices.