Enable Lifecycle Events to be customizable

brian-mann commented 7 years ago

Why this is necessary

Cypress automatically executes code in "between" tests. The idea behind this is that Cypress will try to "reset the state" and enable each test to run in its own pure state without other tests affecting or leaking into one another.

Currently it does the following:

clears localStorage
clears sessionStorage
clears cookies
removes aliases you've created
resets routes you've defined
removes stubs, spies, clocks you've created
clears the cache once before the browser opens

Currently what it does not do:

clears the actual application under test

Why this needs to change

It really makes no sense the way we currently handle this. This is often a source of confusion we've seen from multiple users.

For instance it is:

not clear that this is happening
not clear how you can stop lifecycle events from applying
not clear why we reset everything above but not the application itself (?)

Because we don't forcibly clear the application state it means that it "appears" nothing has changed between the tests, but under the hood everything that built up the current session was reset.

This often results with users clicking links only to find out they've been magically "logged out".

In addition to sessions being cleared, it is actually technically possible for your application to do things in the tiniest briefest moment that Cypress finishes running one test and "switches" to the new test. Since your application is still active in the window, that means it will continue to run everything it's built to run.

This can cause all kinds of havoc.

Here's a sequence of events that demonstrates the problem:

before test 1
- cypress resets state asynchronously
test 1 begins
- your cypress spec code runs
- you visit your application
- you setup cy.route to route XHR's
- cypress finishes executing all commands
test 1 finishes
- cypress moves onto test 2
before test 2
- cypress resets state asynchronously
- while this happens, the application you visited in test 1 runs a poll / timer which creates an XHR
- the state from the routes has been reset, so instead of XHR being routed, it goes out to your server
- your server throws a 500, and the onerror handler of your XHR bubbles up and changes your application's state
- your application is now in an invalid state and you have no idea why
test 2 begins
- your cypress spec code runs
- you don't visit your application again (you already did in test 1)
- your commands fail because the state of your application in test 1 is now invalid
test 2 finishes, displays error

The reason we don't hear every user complain about this - is that for one its incredibly rare for this to happen and represents a "perfect storm" of events.

Additionally when we originally built these lifecycle events, they were with the assumption that most users would be visiting their applications before each test.

When visiting before each test (and thus always recreating / rebuilding the sessions and state), you insulate yourself from all of these problems.

What users really want to do

With that said many of our users really want to do one thing: login once in a before hook during the first test, and then have all subsequent tests run with this session.

While we don't really officially endorse this as the best pattern - it often makes sense in these situations:

your app loads slow (albeit you have a much bigger problem than writing tests)
you're not actually mutating session state in tests and therefore theres no reason to enforce needing to have it cleared each time

Another example users have suggested is when you apply a "unit testing" pattern to integration tests and write a bunch of "tiny" tests with a single assertion. However, we consider this an anti-pattern.

Irrespective of what we believe - the bottom line is that only logging in once before your tests will without a doubt enable all of your tests to run faster. If they have less to do in each test, they will simply complete faster.

Therefore it's imperative that we adequately solve this use case, and in the documentation and guides we provide resources / suggestions / hints / tips / tricks / philosophy to guide you to making these choices.

BTW in case you're curious we believe most of the need for a single login in a before hook is mitigated completely by using programatic API's like cy.request for things like setting state and logging in. By avoiding your UI you skip most of the reasons tests are slow.

~One thing we absolutely do not endorse and will never support is writing tests~ Edited by Jennifer Shehane: I have no idea what this was meant to originally say

Changes that need to be made

Clear the App

Enforce clearing the Application at the end of test if it is not the last test but before the next test runs any of its code.

In other words, unless this is the very last test, the moment test code finishes and before any async code runs we MUST blow the application out of the window so no async events, polls, timers, promises, etc, run.

This will 100% prevent application leakage in "between" the tests and ensure the application is torn down properly.

Update the GUI

Display a new "instrument section" that lists each lifecycle event that was applied (or skipped).

screen shot 2017-09-23 at 8 03 26 pm

Create new commnad

We'll need a new cy.clearApp() command which brings into parity the other cy.clear* commands.

This enables you to issue it within a test as a "one-off" command the same way the others work.

Remove old API's

Deprecate / remove the Cypress.Cookies.preserveOnce API's since that would be superseded by these Lifecycle events.

Other Considerations

Having first class lifecycle events with their own programatic API also means we could turn these completely off when writing unit tests. As it stands Cypress automatically runs the clearing code in between each test. These are asynchronous and involve talking to the automation servers (in the browser and server) and therefore are quite slow (in terms of clock cycles).

Examples of how we could do this

I don't like the idea of making this global configuration in cypress.json. Instead I think we should create API's that enable you to control this in code.

Since Cypress has to know how you intend to apply lifecycle events this code would have to go before any tests are defined or run.

Likely this is what you want - you'd likely want these changes to propagate to an entire spec file.

// users_spec.js

Cypress.lifecycle() // returns an object with the current configuration

Cypress.lifecycle(false) // disable all lifecycle events
Cypress.lifecycle(true) // enable all lifecycle events

Cypress.lifecycle({
  clearApp: true, // leave this on
  clearInternals: true // leave this on
  clearCookies: false // nope
  clearLocalStorage: false // nope
  clearSessionStorage: false // nope
})

describe('my suite', () => {
  it('test 1', () => {
    ...
  })

  it('test 2', () => {
    ...
  })
})

Users wanting to apply Lifecycle events globally to all specs would just add the code into a support file.

Use Cases

At first glance it seems as if lifecycle events would be binary: you'd either want to have them all enabled (by default) or turn them all off.

Not so fast Jack - let's take a deeper look and why there's a bit more complexity than this:

Clear Everything (the default)

Pros

Absolutely no state build up
Simple, clear, insulated
Promotes taking shortcuts and skipping the UI as much as possible

Cons

Realistically a bit slower to run all tests

Summary

Slowest but flake free. Easy to understand.

Clear Nothing (what many users want)

Pros

You visit once in a before hook, usually establishing the session there
All subsequent tests use this existing session and don't visit
The tests will run faster

Cons

All of the tests are loosely coupled to one another unless you're really careful
The state from one test will immediately apply to the subsequent test
When running a single test vs all you will likely see different behavior that is difficult to debug / understand
You're still susceptible to actions in your application leaking into other tests (as described above in detail)

Summary

Fastest but susceptible to edge cases, potential flake, and poorly written tests.

Clear Some Things (likely what you should do instead)

If you want the benefit of only establishing a session once, but don't want to problems of loosely coupled tests then you likely want this configuration:

Cypress.lifecycle(false) // disable everything
Cypress.lifecycle{
  clearApp: true // but always clear the app
})

describe('my suite', function () {
  before(function () {
    cy.login() // establish the session only once!
  })

  beforeEach(function () {
    cy.visit('/') // but continue to visit before each test
  })

  it('test 1')

  it('test 2')

  it('test 3')
})

Pros

The current state of the app is always blown away before each test
The session is preserved between tests avoiding this to be re-created

Cons

Takes more explanation and deeper understanding of Cypress to know about this
Requires visiting before each test since the app is blown away
Keeping the same session around can lead to state issues on the server (if the server isn't stubbed)
You're apt not to take shortcuts which means that the gains you receive from running all tests are lost when running and iterating on a single test

Summary

Faster than the first open, slower than 2nd. Likely what you want instead.

brian-mann commented 7 years ago

Related to:

BenoitAverty commented 6 years ago

One case where tests need to build up state between them is when each test is a step of a cucumber scenario. In that case, each test depends on the previous steps of the same scenario.

I'm using cypress-cucumber-preprocessor but I'm currently stuck because of this (I have opened TheBrainFamily/cypress-cucumber-preprocessor#43 about this).

With the feaure discussed in this issues, it would be doable for the preprocessor to manage the state clearing in such a way that state would be cleared in between scenarios but not between steps of the same scenario.

I'm guessing this is still a long way down the road, but do you have an estimate on when you may be tackling this ?

BenoitAverty commented 6 years ago

In the meantime, if I wanted to experiment, can anyone point me to the direction of the function(s) or pieces of code that do the clearing ? I'd like to see if this can't be solved in the preprocessor itself.

joshacheson commented 6 years ago

Is there any progress on this issue, or can anyone shed light on how product / engineering at Cypress are viewing this? Resolution (or knowing this is a no-op) on this issue would help me a great deal when it comes to evaluating whether or not Cypress is the best solution for problems I'm trying to solve.

jdhines commented 6 years ago

At the very least refer to this issue in the docs. I spent several hours pulling my hair out wondering why my second cy.route() (declared in a before() and used in test number 2) was not actually stubbing. If I had read in the before, route, stubs, and possibly other parts of the docs (which I thought I'd read pretty thoroughly) had this info, I could have easily refactored my test code and gotten my tests passing.

pacop commented 6 years ago

Is some progress or ETA on this proposal? This would be awesome!

ronlawrence3 commented 6 years ago

Wasted a lot of time trying to figure out why my authentication token was lost during tests seemingly randomly. I'm still not sure there is not an issue with the plugin I'm using in invoking this lifecycle during a test, but it would have helped to have known that this was there.

Also a thought about your "not that this is a good testing practice" wording I see all over your docs... While I tend to agree with most of them I've seen, there are many existing protractor / selenium tests out there where there may not be the best approaches, and if you intend to win over those folks, there needs to be a way to migrate from "bad patterns", not just a scolding from you about the wisdom of doing it that way and no other way of doing it!

Knaledge commented 6 years ago

I just want to express appreciation for the typical proposal structure I see from @brian-mann throughout the project.

In using Cypress, it became evident that the state of things was reset between each "it" (test) - which started as a concern at first, which resulted in loosely forming as an observation which then turned into tinkering, spending some time as a suspicion, and then finally becoming apparent/understood.

What's great though is that along that journey, I started to question why I initially reacted with "concern". It gave me something to think about.

And while thinking about it, I realized that all along, I had always believed in "clean test states" - I advocate for this in my own work routinely. Just for whatever reason, it didn't click right away while using Cypress.

What I'm saying is that I appreciate the thought that goes into these requests, the adherence to best practices - and walking the talk. I came around to this understanding of Cypress (and their prescription of "best practice" in this case) organically - and it's just reassuring to see the same thought-journey expressed here in a request from one of the primaries on the project.

All that said, I believe there is a need for this and hopefully maintains the proposed "invocable" / on-demand nature.

Keep up the good work!

jennifer-shehane commented 6 years ago

If an issue is labeled as "stage: proposal", then it is within our roadmap to do someday, but typically no work has been put directly into the feature aside from some discussion and determining it to be technically feasible to achieve. Sometimes there are tertiary tasks done to prepare for a proposal's delivery though.

Phillipe-Bojorquez commented 6 years ago

I use cypress as an e2e tool for my UI automation work. Because my companies app is so UI heavy building up state is actually desired. While most of my test suits are discrete units of logic, I also have to evaluate the performance of an app over time from a users built up state to simulate risk conditions. So I really appreciate this new feature request.

yannicklerestif commented 6 years ago

One of the benefits of having non-independent it() in a spec is that it helps a lot structuring a test report for a longer spec. For example:

describe('Adding a Student to a Class', () => {

    it('Should Login', () => { //... };

    it('Should Create Class', () => { //... };

    it('Should Create Student', () => { //... };

    it('Should Add Student to the Class', () => { //... };
}

Apparently non-independent it()s inside a describe() are considered a bad practice.

How do you suggest achieving the same result?

Floriferous commented 6 years ago

This is completely insane. After months dealing with seemingly obscure logouts, having a horrible cypress experience, and spending hours debugging this (to the best of our knowledge..), we finally kind-of figured this out.

These lifecycle functions should be mentioned within the first 2 minutes of getting started with Cypress in Huge, Bold, italic, underlined letters, like right here: https://docs.cypress.io/guides/getting-started/writing-your-first-test.html#Add-a-test-file.

If at the very least the clearing functions would actually work, and not just completely randomly, it'd be much easier to get around this.

Bobgy commented 5 years ago

I've been affected by not clearing the app. My specific use-case is that: I need to setup some cookies/local storage before visiting the page in each test, so I always put visit commands inside tests.

In one corner case, local storage setup triggered event listeners of the previous app and broke a test. As a workaround, I added a workaround clearApp command in beforeEach hook.

beforeEach(() => cy.visit('/some-blank-page-that-has-nothing'))

This has solved my issue perfectly as of now.

lukeapage commented 5 years ago

I would like to use this to increase performance - I would like to manage caching myself through a per run cache buster, I believe this will help since the longest time during our run is loading the page.

mariotsi commented 5 years ago

3.2.0 fixed it while on watch mode. Headless with electron is still giving the same issue.

toddgallimore commented 5 years ago

You mentioned above

One thing we absolutely do not endorse and will never support is writing tests

I might be missing something but I can't figure out the context in which you've said this. What is it that you're not going to endorse or support? It seems that writing tests is the one thing that you are defiantly going to support and endorse.

lvl99 commented 5 years ago

These lifecycle functions should be mentioned within the first 2 minutes of getting started with Cypress in Huge, Bold, italic, underlined letters, like right here: https://docs.cypress.io/guides/getting-started/writing-your-first-test.html#Add-a-test-file.

I agree with this point as well. We're using AWS Amplify/Cognito which stores logged in user data in localStorage and it wasn't entirely intuitive to find out that the localStorage was being cleared between tests. Having the ability to whitelist specific keys -- or even prevent this behaviour from happening -- would be nicer.

In the short term it makes sense to allow a user to opt in to clearing localStorage by putting in a cy.clearLocalStorage() in a beforeEach/beforeAll or something. At least when reading the test you can see where and why localStorage is being cleared.

pete-om commented 5 years ago

This epic would be very useful for us. In particular, I would dearly love the option to conditionally disable automatic cache clearing

It would speed up tests that don't require a completely clean slate (ie: most), dramatically so if tests are being run on external infrastructure with inherent latency and/or bandwidth issues
It would allow us to confirm correct functionality of our site when assets are loaded from cache, which we rely on to make our site performant
It would also allow us to confirm cache busting

Cheers

vesper8 commented 5 years ago

2 years later and still pulling my hair out and copy/pasting dirty workarounds to be able to log in a single time and not have to write huge tests that I'd much rather break up into many small tests.. is there a beta version somewhere we can use that supports this yet??

danatemple commented 5 years ago

@vesper8

This works for me - to just disable clearing of LocalStorage. https://github.com/cypress-io/cypress/issues/4582#issuecomment-507399740

And add an explicit clear of LS in any functions like "log in", so everything is clean from the start of each test sequence consisting of the small tests that don't want a logout between them.

sebinsua commented 5 years ago

Does anybody know a way of keeping JavaScript files cached between test suites? This is a big performance hit for us, since we're using this to try to test a gigantic Storybook with a massive JavaScript bundle.

WillsB3 commented 5 years ago

From the original post in this issue:

One thing we absolutely do not endorse and will never support is writing tests

😄

I assume something is missing here :)

pavelsmajda commented 4 years ago

2 years later and still pulling my hair out and copy/pasting dirty workarounds to be able to log in a single time and not have to write huge tests that I'd much rather break up into many small tests.. is there a beta version somewhere we can use that supports this yet??

Did something worked for you? After a painful setup and 99999x unsuccessful tries, we just ended with another automation tool. But I would like to give another chance to this tool, due to great potential . Anyway, "Silent Sign In" is still an issue for me.

danatemple commented 4 years ago

@pavelsmajda see https://github.com/cypress-io/cypress/issues/4582#issuecomment-507399740

This works for us, added to a file included by support/index.js. You can then choose exactly when you want to clear your local storage (like, at logout).

Cypress.LocalStorage.clear = function (keys, ls, rs) {
   return;
}

pavelsmajda commented 4 years ago

@pavelsmajda see #4582 (comment)

This works for us, added to a file included by support/index.js. You can then choose exactly when you want to clear your local storage (like, at logout).
Cypress.LocalStorage.clear = function (keys, ls, rs) {
   return;
}

Hi, @danatemple and thanks for link. I remember we already tried this also with extra steps and modifications later, but without success. We're using OpenID Connect (OIDC) and basically there is no way to force cypress to keep session open. I always get "Silend Sign In failed" error message in console and it's displayed, after we received a token. Or do you have different opinion? Can you share your test according to code you sent above, please?

danatemple commented 4 years ago

@pavelsmajda are you also preventing cookies from being cleared between tests as per https://docs.cypress.io/api/cypress-api/cookies.html#Syntax ? I am not familiar with OIDC, so I don't know what browser data storage it relies on. To debug, I suggest you use the developer tools to have a look what storage has been set.

it('log in and look at storage', function() {
    // log in here
   cy.pause();
   // look at localStorage and cookies in developer tools and see what has been set
});
it('new test, see how storage is doing', function() {
   cy.pause();
   // look at localStorage and see if anything has changed
});

jcuffe commented 4 years ago

One thing we absolutely do not endorse and will never support is writing tests

Any additional clarity on this? I feel the same way, but I'd like to know what was meant here!

adomyaty55foundry commented 4 years ago

What's the status on this? I just started extending my tests to automating front-end stuff and I'm running into this issue (My localstorage I would love for it to maintain) this is producing a big headache for me as I'm now forced to do A-Z on almost each test... not ideal at all

jennifer-shehane commented 4 years ago

This issue is still in the 'proposal' stage, which means no work has been done on this issue as of today, so we do not have an estimate on when this will be delivered.

rgehan commented 4 years ago

It has been a proposal for ~2 years now, and has been labelled priority: high for a year, you surely have some really rough idea of when/if it will be done.

What is your process with such proposals? Are you actively discussing this? Are there particular road blocks you need to solve before working on this? Is the proposal simply dead? Couldn't you leave implementation to the community if necessary?

Cypress is an awesome tool, I love what you've built, but you can't leave people hanging for two years, considering this is a major issue that cannot really be worked around for now.

bahmutov commented 4 years ago

@rgehan I understand your frustration and believe me, better lifecycle control would, in my opinion, solve a LOT of phantom issues with flaky tests people see in CI.

We do look at issues every week as a team, and our challenge is always to balance the impact a feature can have versus how much effort it requires to implement. As you can see we are up to 6k issues in this repository, with 1k issues still open. That's a lot of work! We like it though, the number of issues means the test runner is popular and is used across every platform, every architecture, every browser version, every CI system.

As a team, we currently concentrate on cross-browser support, because that is what 99% of our users are asking. But we gladly accept pull requests from anyone in the OSS community, provided the code solves the issue and has matching tests, etc. I would not expect an outside PR for Lifecycle events - this is a big change that would require deep knowledge of how Test Runner works, but if you want to help move it forward, please submit fixes and features for other open issues. It will alleviate our team's load allowing more time to implement the Lifecycle events.

Our roadmap: https://on.cypress.io/roadmap Our changelog for each version: https://on.cypress.io/changelog

eran-shlomo commented 4 years ago

Evaluating cypress and it took me 3 hours to encounter this issue and 2 more to root cause it to this bug.

Svish commented 4 years ago

One thing we absolutely do not endorse and will never support is writing tests

🤔

qmphan commented 4 years ago

Does anyone have a workarround to avoid cache clearing between tests? Our app is very heavy (more than 1M gzipped) and we have many tests. Disable cache clearing would speed up our tests a lot.

jkytomak commented 4 years ago

Does anyone have a workarround to avoid cache clearing between tests?

We do it like this in commands.ts:

const doClearLocalStorage = () => {
    localStorage.clear()
}
const doNotClearLocalStorage = (_keys?: string[]) => {}

// By default Cypress clears local storage between every spec. We disable Cypress local storage clearing function, so that we can test local storage usage
// TODO after Cypress adds support for lifecycle events use them instead to do it: https://github.com/cypress-io/cypress/issues/686
Cypress.LocalStorage.clear = doNotClearLocalStorage
// We need own version to manually clear local storage in tests, because above one disables also cy.clearLocalStorage
Cypress.Commands.add('doClearLocalStorage', doClearLocalStorage)
declare namespace Cypress {
    // tslint:disable-next-line interface-name
    interface Chainable {
        doClearLocalStorage: () => void
    }
}

So we clear the storage only in where needed manually with that custom doClearLocalStorage

qmphan commented 4 years ago

@jkytomak Thanks but your code is to clear local storage.

I am still looking for a solution to tell Cypress not to clear cache (e.g .html, .js, .css files) between tests.

lukeapage commented 4 years ago

We are in the same situation.I considered making a git patch that overwrites the right file in cypress after download :s doable but a pain and needs updating every time cypress releases. If you go for that I’d appreciate a gist!

AshMcConnell commented 4 years ago

Our tests are taking a huge amount of time due to a very large bundle size. It would be great if we could cache the js / html (using the standard browser cache mechanism). Any workarounds?

tnrich commented 4 years ago

We have the same question as @AshMcConnell -- our app is quite large and just loading the bundle (which never changes between loads) adds a HUGE amount of time to our test runs. Has anyone found a way to do this in user-land?

It would be awfully nice if this option had official support as well... @jennifer-shehane @bahmutov

Thanks!

lukeapage commented 4 years ago

You can write some code as part of your build that changes the cypress source.

The cache clear is here: https://github.com/cypress-io/cypress/blob/a22d0cc7563d7cb0cb176100cf93105557301e5a/packages/server/lib/browsers/electron.js#L301

So depending on your version and location of the cache you can write something like (this is in powershell)

((Get-Content -path node_modules\.azure_pipeline_cypress_cache\4.4.1\Cypress\resources\app\packages\server\lib\browsers\electron.js -Raw) -replace 'return webContents.session.clearCache','if(!global.clearedCache) { global.clearedCache = true; return webContents.session.clearCache(); } return Promise.resolve') | Set-Content -Path node_modules\.azure_pipeline_cypress_cache\4.4.1\Cypress\resources\app\packages\server\lib\browsers\electron.js

but.. I didnt get any speed increase (I have a local server). Once cypress is upgraded to 8.3.2 or 9.x electron we can take advantage of this: https://github.com/electron/electron/pull/23868 Which will need another hack until v12 (probably) changes the default. That might help more.

jennifer-shehane commented 4 years ago

We are beginning to outline the work required to implement this feature after many discussions on what is necessary. This is part of our upcoming roadmap and we hope to provide more information on the details of its implementation and release soon.

jennifer-shehane commented 4 years ago

Hey everyone, we've outlined some work to be done to address some of the concerns of 'Lifecycle Events' in this issue: https://github.com/cypress-io/cypress/issues/8301

It outlines a proposal for a 'session' API, to quickly summarize:

With cy.session, you'll be able to perform UI interactions before taking a "snapshot" of all the session related data and "restore" it before each test.

I recommend reading the entire proposal in https://github.com/cypress-io/cypress/issues/8301 and following there for any updates. Please feel free to express any concerns or questions in that issue concerning the API and add a 👍 for general support.

iansjk commented 4 years ago

In the original post:

One thing we absolutely do not endorse and will never support is writing tests

Looks like there's a missing clause at the end here? I assume it's supposed to read "... never support is writing tests where the state is set using the UI" or something to that effect.

Spandana-test commented 4 years ago

Team, while using cy.lifecycle(), i am getting TypeError "cy.lifecycle is not a function".Please help me on this.

josefsmrz commented 3 years ago

Any update on this? I'd like to use the lifecycle setting to preserve loaded resources like images or css between tests to speed them up as reloading them seems like a waste of time...

jennifer-shehane commented 3 years ago

@iansjk 😂 I have no idea what that was meant to say. The original seems to reflect this same wording, but obviously this was missing some clause.

The Sessions API work is coming along and viewable here: https://github.com/cypress-io/cypress/pull/14350. The WIP documentation is here if you want to see what it enables you do test: https://github.com/cypress-io/cypress-documentation/pull/3427

dwilches commented 3 years ago

Just arrived here searching for a way for Cypress to cache images and js files to speed up my tests. Is there an estimated time for when this improvement will be done?

vishwa-amit commented 3 years ago

looking at the conversations from start to end .. doesn't seem to have a solution or workaround even after this has been discussed for a while :-(

timini commented 3 years ago

Any workaround available to stop cypress clearing localstorage between tests?

GentryRiggen commented 3 years ago

The workaround is easy but ugly. Only have one describe() => it() per spec test. You can use cy.log(message) to achieve a similar feel in the logs. Not ideal but gets around this problem.

cypress-io / cypress