feature-by-feature Parallel testing

[JinCoreana]

🤔 What's the problem you're trying to solve?

I am using Cucumber.js + Playwright for E2E testing in a javascript project. I want to run all feature files in parallel but inline command --parallel 3 runs all scenarios in various feature files not in order.

✨ What's your proposed solution?

Is there any cucumber builtin option for parallel testing to run the scenarios in one feature file in order and then move on to the next feature file?

⛏ Have you considered any alternatives or workarounds?

I understand this is possible using JUnit4 but not sure if this can be done in Javascript project. https://cucumber.io/docs/guides/parallel-execution/?lang=java#junit-4

📚 Any additional context?

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[davidjgoss]

The answer here about why we don't have feature-level hooks has some overlap with what you're asking: https://github.com/cucumber/cucumber-js/blob/main/docs/faq.md#why-arent-there-beforefeature-and-afterfeature-hooks

Is there any cucumber builtin option for parallel testing to run the scenarios in one feature file in order and then move on to the next feature file?

That reads as just running in serial, but I think what you want is to have all scenarios from a single feature run in sequence and on the same worker? If so you might be able to lean on custom work assignment, at least to keep one scenario at a time running from a given feature (via pickle.uri) - I don't think you could guarantee them being on the same worker though.

...but inline command --parallel 3 runs all scenarios in various feature files not in order.

Could you say more about what problem this causes for you? Ideally scenarios should work independently.

[JinCoreana]

The answer here about why we don't have feature-level hooks has some overlap with what you're asking: https://github.com/cucumber/cucumber-js/blob/main/docs/faq.md#why-arent-there-beforefeature-and-afterfeature-hooks

Is there any cucumber builtin option for parallel testing to run the scenarios in one feature file in order and then move on to the next feature file?

That reads as just running in serial, but I think what you want is to have all scenarios from a single feature run in sequence and on the same worker? If so you might be able to lean on custom work assignment, at least to keep one scenario at a time running from a given feature (via pickle.uri) - I don't think you could guarantee them being on the same worker though.

...but inline command --parallel 3 runs all scenarios in various feature files not in order.

Could you say more about what problem this causes for you? Ideally scenarios should work independently.

Thank you very much for the reply and links to have a look at, David. Referring to your description, I guess I was using Cucumber in an unintended way. Because I have several scenarios in one feature connected in sequence. Please see the example.

Let's say we have a feature A - New post with 3no. scenarios and feature B - Custom tap with 2no.scenarios like below. scenario A-1 :Create a new post scenario A-2: Edit the content and save changes scenario A-3: Delete the post

scenario B-1: Save an existing report in custom tap scenario B-2: Delete it from custom tap

Then I run 2 workers expecting one to do feature A and the other one to do feature B. However, the outcome is: worker 1: A-1 ➡️ A-2 ➡️ B-1 fails worker 2: A-3 fails and this worker gets frozen

I will try again after adding some more steps to make each scenario finish and start from the same page. Thank you again for the reply!

[davidjgoss]

I will try again after adding some more steps to make each scenario finish and start from the same page

Yeah I think this is a good approach to run with. Like scenarios A-2 and A-3 could start with something like Given an existing post that goes and creates a post ready to be acted on in the When step(s).

[JinCoreana]

This is now working very well! Although I came across another issue with page.evaluate in BeforeAll hook gets skipped in some of the workers sometimes (50% all work fine - 50% this issue happens I would say). I will prepare an example and post it as a separate issue. Thanks a lot!

[davidjgoss]

In case it helps https://github.com/cucumber/cucumber-js/issues/1153