search

Automattic / wp-e2e-tests

Automated end-to-end tests for WordPress.com
https://github.com/Automattic/wp-calypso
GNU General Public License v2.0
110 stars 25 forks source link

Figure out SKIP_TEST_REGEX replacement for Magellan #506

Open hoverduck opened 7 years ago

hoverduck commented 7 years ago

Since moving to parallel test runs via Magellan in #504, the SKIP_TEST_REGEX envvar no longer functions to dynamically skip certain tests due to the way Magellan loads the spec files. I know we could modify the magellan/mocha plugin to handle it, but I'd rather find a way to handle it outside the plugin. Maybe via the "mocha_args" parameter?

alisterscott commented 6 years ago

I think the only reason we use this is if Mailosaur is down and we don't want to run tests that have a dependency on email, so, could we add an @email to each of these high level specs that have this dependency and skip them this way somehow?

hoverduck commented 6 years ago

I think the only reason we use this is if Mailosaur is down and we don't want to run tests that have a dependency on email

That's the main reason we do it (and why we still have the DISABLE_EMAIL envvar supported, but I can imagine other cases where we'd want to disable a different set of tests. Like if payment services were going to be down for a while we may want to skip all tests with a checkout step.

I'm working on a change to the magellan-mocha-plugin that adds a new filter to support skipping tests on the command line, like --skip=Invite|Signup. It's a pretty basic change by itself (not sure why I didn't think of it before), but I'd like to get my previous changes merged into the upstream repo to limit how far we diverge. https://github.com/TestArmada/magellan-mocha-plugin/pull/18

I just need to make a few minor changes to that, then hopefully they'll merge it, and then I'll push a new PR with the added filter.