Rewind & Resume Test / Interactive Breakpoint

[anthony-chaudhary]

What would you like?

I'd like to rewind a test to just before failure, make a small change, and resume a test from that given point.

Why is this needed?

For the purpose of faster development.

Becuase

1. It seems to be well established that any debug system, including cypress, slows down performance. Ok that's accepted.
2. The system under test will never be fully optimized, nor will the test suite ever be fully optimal.

For our case specifically in the context of complex image and video annotation this means getting to the desired state can take some time. Overall the challenge is that something that a dev can get to in the normal application state in the order of x is likely 2,3, or 4x+ during test.

This means it takes a lot of time to get meaningful feedback from any changes. This is further hampered that normal tools available like VueJS dev tools appear limited in the test env. And that the console is altered such that many expectations of elements normally available are either not or require extra effort to acquire.

The joint effect of this is a lot of guessing on what will work, and/or spending way too much time in the thought process instead of actually seeing execution. This is especially true for cases where the application works as expected but something in the testing framework is wrong, a bug in cypress, or some other assumption specific to the testing concept is off.

Overall an effective implementation of this would dramatically improve the developer experience.

Other

Given the end to end nature of cypress, and the existing UI components, it seems like a reasonable extension of the existing functions of clicking a given moment in time.

Examples of ways to limit scope on this

1. maybe only works for changes in application code, not in test spec
2. maybe could be rolled into the before context to further limit what contexts it needs to work in

[BlueWinds]

I can certainly see the appeal of time-travel debugging, but it's difficult to implement in an application-agnostic way.

Currently, the way our snapshots work is by examining the DOM - when you view a snapshot, we replace the DOM wholesale with the earlier version, without any attempt to restore the browser JS stack or other browser state.

In order to restore the application under test to the working state at the time of the snapshot, rather than simply a visually similar DOM, we'd have to also:

• Restore all JS event bindings on the elements.
• Restore the internal state of your application in the browser.
• Restore the internal state of and servers the application talks to. What if a later step created a new user on a local server, how would we restore it to the previous state? What if it created a new user on a remote server?

That's not even getting into when the application (or test) change.

Time travel debugging is made possible in the case of something like React+Redux (or Vue), because those tools can assume that the state of the whole world depends on the application's internal data store - we don't have that luxury in an end-to-end testing environment where we don't know the details of the application.

[BlueWinds]

To be slightly more concise - getting your application into a specific state so that you can modify / test it is not something cypress can handle automatically.

If it takes a long time to achieve a particular configuration in your application that you want to work with, then it'd be best to focus on ways to jump-start your application into a particular configuration - can you serialize / deserialize your application state somehow? This is likely to pay big dividends in speeding up your tests, regardless of framework.

[bahmutov]

Example: https://www.cypress.io/blog/2019/10/29/split-a-very-long-cypress-test-into-shorter-ones-using-app-actions/

Sent from my iPhone

On Nov 10, 2021, at 14:38, Blue F @.***> wrote:

 To be slightly more concise - getting your application into a specific state so that you can modify / test it is not something cypress can handle automatically.

If it takes a long time to achieve a particular configuration in your application that you want to work with, then it'd be best to focus on ways to jump-start your application into a particular configuration - can you serialize / deserialize your application state somehow? This is likely to pay big dividends in speeding up your tests, regardless of framework.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[anthony-chaudhary]

Thank you for the kind thoughts - I think there's still a lot more here.

Another way perhaps to think about this is, unless I run the entire test suite I can't say it's all complete. But in reality we work on one test at a time. So in that same context, I am mainly looking for a way to smoothly verify information and fix aspects inside of a test, without having to re-run the entire test. Overall I'm trying to turn a process that currently learns heavily on guess and check into something more deterministic.

Idea 1 - offer user a rollback hook

If we think about something like a database migration, we have upgrade and downgrade() functions

If a user's state is such that reverting to an early state in the test does not actually effect anything relevant, then it can be safely ignored. Otherwise, a rollback() type operator must be supplied.

The user can fill in anything that is required to be changed to redo the test. That may sound like a lot, but it could be a simple as having a reset_state() or something that it calls etc. The point from cypress perspective is that you only need to call rollback() on reversal.

That means from cypress perspective, it would be

1. ResetWhatItCan()
2. CallUserHookForAppState()

The assumption is that a reset_state() may be something that can run nearly instantly, but actually resetting the entire test runner is 100x slower

Alternative 1 - user defined snapshot

Offer a snapshot() function that takes a longer form snapshot Again this doesn't have to be 100% perfect (the same way the current DOM snapshot isn't), but just something that gives the user a greater ability to actually see what will work with the framework.

I recognize there's caveats to either, that I'm not saying it's a cure-all but it would be a big step forward at least for us. Again I realize there's issues there - but just imagine how much better the eventual developer experience could be with that type of feature. Any steps toward it would be really great

Re: @bahmutov One context of debugging the test runner / sub libraries

I think the thing to keep in mind here is one of the main use cases is not regular application state testing directly, but rather the annoying intermediary bugs. For example the other day I had a test fail to get a data-cy element, that was clearly visible, becuase of an obscure issue with an intermediary framework. If we could more readily go back, look at the actual object, make a change, and then instantly verify that change, that would be so much better.

While I'm sure there's optimizations we can do, in our context application state can be very complex to reasonably arrive at and load in cypress. If cypress can instead fix the speed issues so that things load closer to normal speed that would help too, but right now for us at least certain key pages takes 10-20+ seconds to load in cypress just the page... nothing else to change

Keeping the E2E context theme

I think a theme I'm trying to get at here is: In my 2 cents the biggest value of cypress is the end to end aspect. The further that get away from the default of what an application looks like, and the more we start mocking states for the sakes of testing, the more it starts to feel like a regular unit test not a true end to end test. It's really frustrating to have a test pass locally, and then fail in CI becuase a component wasn't minimized or a expected user info box came up etc. Those types of things should be smooth to accept/ignore etc if they aren't the subject of the test. Again I realize part of this is my own learning etc but it just feels like there's so much room for cypress to improve that experience.

Anyway I hope that helps! I get it this sounds like an epic feature. I hope my 2 cents helped

[cypress-app-bot]

This issue has not had any activity in 180 days. Cypress evolves quickly and the reported behavior should be tested on the latest version of Cypress to verify the behavior is still occurring. It will be closed in 14 days if no updates are provided.

[cypress-app-bot]

This issue has been closed due to inactivity.