search

eWert-Online / OSnap

OSnap is a snapshot testing tool, which mainly focuses on speed and ease of use.
https://ewert-online.github.io/OSnap/
152 stars 2 forks source link

Order that tests are run #17

Closed kevin-j-morse closed 2 years ago

kevin-j-morse commented 2 years ago

Is the order that tests are run in pre-determined or random?

I have a web application with a login form which was causing no shortage of grief until I realized that cookies seem to be preserved within each agent so I only need to login as an action on one of the tests.

The problem is that as I add new tests the order seems to change and then the program either crashes or fails the difference depending on when the login form is processed.

I tried numbering the tests but the numeric order didn't seem to be respected.

eWert-Online commented 2 years ago

The order is to a large extend random. I do order the tests based on if they are new or not, so that new snapshots are created as early as possible. But other than that, I take them as they come in from the file system (which is random).

Even if I would provide a way to order the tests, it would not guarantee a specific order, they are run in because of the large amount of parallelism we are using to speed things up.

I have a web application with a login form which was causing no shortage of grief until I realized that cookies seem to be preserved within each agent so I only need to login as an action on one of the tests.

To be honest, this sounds more like a bug to me, than an intended feature. Ideally, you would want your tests to run in isolation. I do get, that logging in (or doing any other actions, which have to repeat for every test) are missing currently.

5 is something, which would address this.

kevin-j-morse commented 2 years ago

Thanks for the reply. Given the information provided, I agree that this behaviour sounds like a bug. Will close this and open another issue to track this.