Proposal: Switch to a test case-centric model

davidjgoss commented 10 months ago

🤔 What's the problem you're trying to solve?

The current React components and HTML formatter are very feature-centric - that is, they faithfully reproduce the content of the source feature files, which are then decorated with the data from the test runs. This is a really nice way to visualise it when things are simple, but it becomes confusing where a step or hook appears once on-screen but can have multiple results from different test cases if it's in e.g. a background or a scenario outline.

This is tricky enough, but then we want to represent retries which adds another dimension of complexity.

I think the design makes the formatter's purpose a bit unclear. Does it want to be a way for people to read your feature files as documentation, with the results sprinkled on for convenience? Or does it want to be a useful report for people trying to keep their pipelines green? I think we need to pick one, and it should be the latter (we can always look at a "documentation" formatter separately).

Finally, I think displaying things in terms of the original feature files is hiding the true nature of Cucumber's test case model from users who would probably be more successful overall if they understood it better.

✨ What's your proposed solution?

Redesign the formatter to:

Still be grouped by feature
Within each feature, display the results by test case - not scenario
Provide a "go to definition"-style function for locating the source line(s) in the feature file

This would be a big rework but I think it will result in a simpler codebase (without the constant translation between features and pickles and test cases) and a more useful formatter for Cucumber users.

Other potential upsides:

Less coupling to the feature file as a source format, which could be good if we start to look at other source formats like Excel or CSV again
More freedom to design in other useful stuff like timing etc (start and finish timestamps for steps can be useful for debugging) without having to make it fit within the feature file metaphor

Concrete example

Given this Gherkin:

Feature:

  Background:
    Given some preliminary stuff

  Scenario:
    Given some other stuff
    When a thing happens
    Then everything is fine

And a Before hook named "preflight check" And an After hook named "save video" Then the formatter should clearly output the test case like:

Hook: preflight check
Given some preliminary stuff
Given some other stuff
When a thing happens
Then everything is fine
Hook: save video

mpkorstanje commented 10 months ago

This makes a lot of sense. I think we should do this.

I do think it would help to display the test cases in the hierarchy of the feature file. So a test case is nested under Feature -> Rule -> Scenario Outline -> Example Group -> Example Number + Pickle Name.

I think the hierarchy would provide enough information for users to understand exactly which part of the feature file matches which test case.

luke-hill commented 10 months ago

Sounds grand.

I envisage a way in which instead of expanding a feature and getting the plaintext from said feature, you'd get a series of expandable test cases. In each of these test cases it would gather/extract the text from the run (So hooks/backgrounds/gwts e.t.c.) - Basically as you described.

mattwynne commented 10 months ago

This is really interesting. I think the heritage of that set of components came out of a time when we had been building Cucumber Reports, successor to Jam/Cucumber Pro, successor to Relish, and we were really trying to create "living documentation". So it absolutely was aimed at being "a way for people to read your feature files as documentation, with the results sprinkled on for convenience".

However, that was then and this is now. I think you're right that more people want test results than want documentation (with some test results sprinkled in) and if you're saying that it's hard to get to good test results with things as they are, then it makes sense to trade off differently so we can deliver good test results.

I still wholeheartedly believe in documentation being important, but it seems to me that if we have good test-case result reporting at the core, we can layer documentation on top, with the ability to drill down the test results analytics.

I wouldn't even necessarily be married to grouping them by feature, at least not always. I think grouping by, for example, tag might be just as useful. So having the test-case/pickle be the unit of display, and different ways of grouping/filtering/navigating them might be a nice way to think about things.

blaisep commented 10 months ago

Heavy plus one from me.

This is exactly what I was hoping to be able to use in vitrina.readthedocs.io

On Tue, Nov 14, 2023, 8:36 PM Matt Wynne @.***> wrote:

This is really interesting. I think the heritage of that set of components came out of a time when we had been building Cucumber Reports https://reports.cucumber.io, successor to Jam/Cucumber Pro, successor to Relish, and we were really trying to create "living documentation". So it absolutely was aimed at being "a way for people to read your feature files as documentation, with the results sprinkled on for convenience".

However, that was then and this is now. I think you're right that more people want test results than want documentation (with some test results sprinkled in) and if you're saying that it's hard to get to good test results with things as they are, then it makes sense to trade off differently so we can deliver good test results.

I still wholeheartedly believe in documentation being important, but it seems to me that if we have good test-case result reporting at the core, we can layer documentation on top, with the ability to drill down the test results analytics.

I wouldn't even necessarily be married to grouping them by feature, at least not always. I think grouping by, for example, tag might be just as useful. So having the test-case/pickle be the unit of display, and different ways of grouping/filtering/navigating them might be a nice way to think about things.

— Reply to this email directly, view it on GitHub https://github.com/cucumber/html-formatter/issues/283, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAB6D6A25YOISEP42J5VOWTYEQMBVAVCNFSM6AAAAAA7HJY7RKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQMJRGY3DCMZXGI . You are receiving this because you are subscribed to this thread.Message ID: @.***>

cucumber / html-formatter