Allow for random spec ordering when running tests

[joelzimmer]

Current behavior:

When running specs in cypress, the files run in alphabetical order, and tests within each spec run in order. See #2901.

Desired behavior:

It'd be great to pass a flag into cypress (or for it to do some sort of optimization like Jest does) to randomize the order of test runs. This would allow tests that rely on other state to fail and could help increase confidence in how the app works.

Versions

3.1.3, Alpine Linux, Electron

[Strajk]

Just noticed there is a library for randomizing tests in Mocha by the almighty @bahmutov, would be great to integrate it to Cypress :)

https://github.com/bahmutov/rocha

[bahmutov]

One thought: it would be nice to have a callback from Cypress into userspace code with the list of specs and tests in each. The user code then can return the list of tests - in any desired order. I kind of do this in https://github.com/bahmutov/cypress-select-tests projects where I get a callback of all tests found in the spec file and can remove all tests I don't want to run

On Tue, Apr 16, 2019 at 11:20 AM Strajk notifications@github.com wrote:

Just noticed there is a library for randomizing tests in Mocha by the almighty @bahmutov https://github.com/bahmutov, would be great to integrate it to Cypress :)

https://github.com/bahmutov/rocha

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/cypress-io/cypress/issues/2908#issuecomment-483707749, or mute the thread https://github.com/notifications/unsubscribe-auth/ACHApsM-jXyR5MgkhhuybvFWTHANHjBJks5vhepFgaJpZM4ZHKIm .

-- Dr. Gleb Bahmutov, PhD

Schedule video chat / phone call / meeting with me via https://calendly.com/bahmutov gleb.bahmutov@gmail.com @bahmutov https://twitter.com/@bahmutov https://glebbahmutov.com/ https://glebbahmutov.com/blog https://github.com/bahmutov

[hcharley]

Is there an update on this @ Cypress Team?

[heathd]

+1

[mxygem]

+1

[sumitngupta]

+1

[terencechow]

+1

If you share where the location is in this repo that grabs spec files I can take a stab at this. Would it be enough to:

1) randomize the spec files order 2) use https://github.com/bahmutov/cypress-select-tests to randomize the tests within the spec files?

Or is there another reason that randomizing spec file order might break things? (Does something depend on spec file ordering?)

[mncharlton]

I've had a go at randomising the order of the tests (it blocks) within a spec, it's not perfect yet, but is a start - https://www.npmjs.com/package/cypress-random-test-order

[jennifer-shehane]

@mncharlton Could you open a pull request to add this plugin to our documentation?

Instructions here as well as a PR Template Checklist. Thanks!

[mncharlton]

Thanks for the prompt @jennifer-shehane! PR is here: https://github.com/cypress-io/cypress-documentation/pull/3024

[duraz0rz]

Any other movement on this? Seems like the cypress-random-test-order plugin is deprecated by the owner as they don't have time to work on issues.