search

vitalets / playwright-bdd

BDD testing with Playwright runner
https://vitalets.github.io/playwright-bdd/
MIT License
278 stars 33 forks source link

Project roadmap for 2024 #83

Open vitalets opened 8 months ago

vitalets commented 8 months ago

I'd like to share my thoughts on playwright-bdd development in 2024. There are 3 main directions:

  1. ✅ Become less dependent on Cucumber internals Currently feature files and step definitions are loaded using Cucumber internal functions. It has several downsides:

    • changes in Cucumber can break playwright-bdd (as it was in #80)
    • tricky run for ESM: need to pass NODE_OPTIONS: '--loader ts-node/esm --no-warnings' for Cucumber although Playwright handles ESM internally without ts-node
    • need to configure importTestFrom manually, impossible to have several test instances (see #46)
    • performance: each Playwright worker loads full Cucumber package although runner is not used

    All these points will be resolved if playwright-bdd will load features and step definitions by own code. For features it can rely only on @cucumber/gherkin and for step definitions it can re-use Playwright's methods for loading test files. Actually, playwright-bdd already uses Playwright's import for loading steps in decorator syntax.

  2. ✅ Support Cucumber reporters Playwright HTML reporter is not the best option for showing BDD test results, it is not designed for it. Background steps are hidden inside beforeHooks (see #78), Scenario Outline requires titles setup (see #67), there are unneeded details (e.g. fixtures). Cucumber's HTML reporter is definitely better for BDD and was proposed a long time ago (see #9). Current idea is to support Cucumber's message formatter and then use it for all other Cucumber compatible reporters and test automation tools (see #82). To validate messages output playwright-bdd should pass Cucumber compatibility-kit. It's important to generate Cucumber messages from Playwright's Blob output, because it allows to shard test runs and finally build Cucumber report with Playwright's merge reports command.

  3. 🔲 Create a tool for writing BDD scenarios This is more an experimental idea. I think about creating an online tool for writing BDD scenarios during collaboration sessions. It should have a simple UI and help people to quickly compose BDD scenarios following the best practices. Additionally, you can import steps from bddgen export command and use them for inline suggestions to speedup the process. The closest existing service is CucumberStudio, but it is not free and has a complicated UI (imho). If you know other similar services, feel free to share in the comments.

Any your ideas and thoughts are welcome here or in the corresponding issues!

ronvoluted commented 8 months ago
  1. Create a tool for writing BDD scenarios If you know other similar services, feel free to share in the comments.

It's not Gherkin or E2E but for some good precedent on collaboration, good DX, nice design and support for various imports/exports there is Stately, a UI for XState finite state machines.

pateljigar commented 4 months ago

Many teams have very common requirement to pass test data in the .csv, .xls, .xml or .json for different scenarios in the feature files. Currently we have a limitation of doing data driven testing using data tables only. This is really limiting lots of teams to design real life data driven tests.

can we please prioritise this in upcoming releases??

vitalets commented 4 months ago

Currently we have a limitation of doing data driven testing using data tables only

@pateljigar could you provide an example?

CalebMacdonaldBlack commented 3 weeks ago

@vitalets What are your plans around the html formatter?

I'm trying to figure out where to focus my efforts regarding some possible to the existing one.

Some options I've considered are:

  1. Contributions to cucumber/react-components and cucumber/html-formatter
  2. Contributions to vitalets/playwright-bdd
  3. Building a custom formatter myself

Some thoughts:

  1. Contributions to the cucumber repos will likely be limited to changes that are within the scope of that project. So no playwright stuff. Fixes and enhancements to existing stuff is probably welcomed, but additions will probably more difficult to get accepted.
  2. Playwright-bdd looks to have recently changed how it depends on cucumber/html-formatter. Instead of importing the package, the code is copied instead. I suspect the intention is to build a html-formatter tailored to playwright-bdd using cucumber/html-formatter as a base.
  3. I'm not apposed to building my own customer formatter, but I would prefer to collaborate with others if I have the opportunity.

@vitalets Do you have plans for building a html reporter for playwright-bdd? If so I would love to contribute.

vitalets commented 3 weeks ago

Hi @CalebMacdonaldBlack

I think it's better to follow option 1 - contributions to cucumber/react-components and cucumber/html-formatter. Cucumber team plans to make reporters more universal and less bound to Cucumber implementation. Recently they added external attachments and links list, that gives great opportunities to enrich this reporter with Playwright stuff, like trace-viewer and annotations.

Playwright-bdd looks to have recently changed how it depends on cucumber/html-formatter. Instead of importing the package, the code is copied instead.

No. Playwright-bdd still imports cucumber/html-formatter.

Building own formatter for BDD is also a way to go. You can try to adopt existing Playwright html formatter for showing BDD stuff. We have an issue for that https://github.com/vitalets/playwright-bdd/issues/122. It can be even you personal project, that will be referenced in Playwright-bdd docs.

Building own Playwright-bdd HTML formatter from scratch is not planned for now.

CalebMacdonaldBlack commented 3 weeks ago

Thanks for the update @vitalets

I'll have to think on this for a little bit.