Group report's tests by scenario

[neojp]

It would be nice if we could group reports by scenario so it's easier to find the test you are looking for.

[garris]

Do you mean for scenarios with multiple screenshots e.g. using selectorExpansion or multiple selectors?

I think I'd need some more detail to make sure I understand the use case -- but yeah -- the front end could go a long way towards reducing cognitive load on things that are related.

Fair warning -- you will find some tech debt in there... And some lemurs. Watch out for the lemurs.

☮️

[neojp]

And some lemurs. Watch out for the lemurs.

Hah!

---

Do you mean for scenarios with multiple screenshots e.g. using selectorExpansion or multiple selectors?

Right, I didn't think of that. I should be clearer. I'm talking about grouping a set of viewports by scenario:

{
  "viewports": [
    { "label": "2xl", "width": 1920, "height": 1080 },
    { "label": "xl", "width": 1280, "height": 800 },
    { "label": "lg", "width": 1024, "height": 768 },
    { "label": "md", "width": 768, "height": 1024 },
    { "label": "sm", "width": 640, "height": 768 }
  ],
  "scenarios": [
    { "label": "Home", "url": "http://localhost/" }
  ]
}

That way you don't see 5 rows of "Home" tests, but rather just 1 row saying "Home"; once clicked on it, it expands to display the viewports available within that test. It gets out of hand very quickly once you have many scenarios with multiple viewports (I have a small site with 21 scenarios and 5 viewports and another one coming up with way more than that).

Examples

Collapsed

Expanded

---

Does it make sense? Would you do it any differently?

[garris]

Yes, makes a lot of sense. There are some filter states, some display options and there is print view. There are also primitive (but useful) result-paging controls (maintaining a group-paging / result-paging relationship for that interaction would be cool). I think all of these features need to be considered because they would interact with the grouping behavior.

But yes, grouping of viewports is a great idea -- except maybe when there is only one viewport -- then it's extra noise. I wonder if the collapse interaction is necessary -- seems to me the visual indentation gets you most of the way there.

And finally, I would wrap the groups in a subtle group boundary since in some cases group headers are not visible on the screen in relation to their child elements.

I think if we can address these issues it would be a solid enhancement.

[neojp]

There are also primitive [...] result-paging controls [...]. I think all of these features need to be considered because they would interact with the grouping behavior.

I haven't seen any pagination features. Where can I find more info?

But yes, grouping of viewports is a great idea -- except maybe when there is only one viewport -- then it's extra noise.

I'd like to see this implemented as an optional feature in the filter button. That way we preserve the previous functionality for people that wouldn't like it grouped or think it gets in the way for the current report.

I wonder if the collapse interaction is necessary -- seems to me the visual indentation gets you most of the way there.

The main reason I'd like to see a collapse/expand is to avoid noise when I'm running 75 scenarios with 5 viewports. I want to identify if there are any failed tests under an scenario without having to scroll too much. I know I can search and filter by failed tests, but I think this can be useful.

We could keep them expanded by default, and a button to toggle all of them at once.

And finally, I would wrap the groups in a subtle group boundary since in some cases group headers are not visible on the screen in relation to their child elements.

Sure.

[garris]

These are really useful -- allows user to keep eyes in the same position while result elements are incremented. Keyboard controls would be even better.

[gpaciga]

This is a different use case, but I've been toying with something similar, to group results based on their labels, basically parsing them like folder paths.

I happen to be generating scenario labels from file names, so it makes sense for us to group results in the same way. There's some improvements to be made (in particular, it doesn't play nice with either of the other filter methods), but wondering if this might be useful to others as well?

[neojp]

@garris These are really useful -- allows user to keep eyes in the same position while result elements are incremented. Keyboard controls would be even better.

Gotcha, I saw those and didn't think of them as "pagination".

@gpaciga I happen to be generating scenario labels from file names, so it makes sense for us to group results in the same way.

Something like archives/test1 and archives/test2 in the labels?

[gpaciga]

Something like archives/test1 and archives/test2 in the labels?

Exactly. e.g. with labels liks

tests/archives/subsection1/*
tests/archives/subsection1/*
tests/archives/subsection3/*
tests/docs/*
tests/home/*

I'll see

tests
|- archives
|- docs
|- home

When I click on the archives line, the tests are filtered to only include those with a label beginning with tests/archives/*, and the nav section will be updated to look like

tests
|- archives
    |- subsection1
    |- subsection2
    |- subsection3

so I can drill down further into one of those subsections, if I want, or click back on tests to go back up a level.

Still a work in progress so I'm playing around with how it works, but I'm liking it so far.

[neojp]

I also like this idea. What if we make a "Group by" section in the filter and let people choose?

[garris]

I also like this idea. What if we make a "Group by" section in the filter and let people choose?

Group-by field would be bad ass. Maybe that belongs in the config section.

Would probably necessitate auto-reading-and-writing those params to the html report directory. Or including with config.

[garris]

@gpaciga thats cool!

[gpaciga]

I have an idea how I could extend my POC with a dropdown for different grouping options. I'll take a stab at it in the next day or two and report back.

[gpaciga]

Ok, it isn't quite what I had in mind for "group by" options but I have something workable here: https://github.com/gpaciga/BackstopJS/tree/feature/navigable-report-by-label

I've added two new settings, disabled by default, so nothing new will show up unless you opt in (bottom right here):

Each one adds a new navigation/filter section (wasn't quite sure what to call it), one broken down by viewport label and one broken down by scenario label. Both use slashes / to create hierarchies, if they exist, otherwise it just makes one list. For example, if I enable the viewport filter on the smoke tests, I get this:

Then if I click on any one of those viewports, I'll see only tests with that viewport.

Both can be enabled at once:

The one thing I don't really like is that it's hard to see what the currently selected option is, but I can probably do something to address that, maybe just by saving the last selected filter in state, similar to how filteredStatus keeps track of which all/pass/fail button is selected.

This isn't the same as grouping things on the page, which was the original ask, since you only see one group at a time, but it might still address the same problem.

@garris what do you think? Is this something that's worth opening a PR for or is this not the right direction? Would the settings make more sense as config options to control whether they're enabled by default or not?

[garris]

@gpaciga thanks for mocking this up! I am not sure we should go in exactly this way. For anyone who wants to see a particular viewport they can just filter with the free text filter field. The group-by-delimiter was a cool idea -- you could just add a text field in that drop down for a user-specified option. That said -- the feature is pretty complex -- maybe best to start with a simple group by label switch that visually groups all the viewports together and take it from there. 👍

[gpaciga]

Free text isn't perfect, since it could match anywhere, not just the viewport. I also like seeing the stats broken down by viewport/label. But yeah that might be good enough for some.

maybe best to start with a simple group by label switch that visually groups all the viewports together and take it from there

Ok, just visually grouping by viewport---without changing which tests are on the page---is a different feature, I think.

The filter-by-label-delimiter option actually almost does the same thing, because tests with the same label but different viewports, selectors, etc, get grouped together. e.g., the "pkra bug test" has two viewports and two selectors, so there are 4 tests under that label. The visual presentation and navigation is different, but you get better usability for large a large number of tests, even if you don't have a delimiter in your labels.