Built-in html formatter generates a blank html report if scenarios have not finished yet completely

[seleniumghost]

👓 What did you see?

When running tests using built-in html formatter, it generates a blank HTML file until all scenarios finished completely.

✅ What did you expect to see?

HTML report to be generated should not be blank after a scenario has been finished

📦 Which tool/library version are you using?

cucumber 8.0.0 cucumber-html-formatter 19.2.0 ruby 3.1.1 Windows 10

🔬 How could we reproduce it?

Steps to reproduce the behavior:

1. Have more than one scenario in a test.
2. Execute cucumber --format HTML --out report.html
3. Open the html report when test is still running and html file is blank.

📚 Any additional context?

As I have not found it in the issue list, I'm picking up the issue mentioned in comment https://github.com/cucumber/cucumber-ruby/issues/1524#issuecomment-824267664 as it shows up as unacceptable drawback compared with older versions, especially when having a lot of scenarios or long scenarios or even both of this. Additionally, when issues like https://github.com/cucumber/cucumber-ruby/issues/1524 occur, the whole report is affected and not only a part of it.

written by marnen:

Unfortunately, the way that’s done makes the Cucumber HTML report much less useful than it used to be. Back in the day, Cucumber used to use a similar approach to RSpec: it wrote straight HTML as scenarios ran, so that the file would render something useful in the browser before it was completely written. That makes it ideal for monitoring a long run with lots of scenarios.

Current Cucumber HTML output, though, renders nothing until the run is complete (I think because window.CUCUMBER_MESSAGES isn’t well-formed JSON/JS till the final closing bracket is written), which means I need to wait for the entire run to finish before I see any output. Is there an HTML formatter for current Cucumber that doesn’t have this drawback? I haven’t found one, and I really miss the old RSpec-like behavior. (And if this is off-topic, I’ll raise a new issue or ask on the list.)

answered by mattwynne:

In fact, this kind of live monitoring of a test run from a browser should now be easier to do, given that the results are streamed out as little ndjson packets. The problem would be getting that stream from your Cucumber process to your browser. It's a really cool idea, but yes it is off-topic for this thread I think :) If you're interested in pursuing it, let's create a new issue (in https://github.com/cucumber/cucumber) and discuss it there.

---

This text was originally generated from a template, then edited by hand. You can modify the template here.

[mpkorstanje]

I don't think process monitoring is reasonable to expect. There are plugins better suited to that. However displaying a message when CUCUMBER_MESSAGES is not defined would avoid future confusion.

[seleniumghost]

Well, just to avoid misunderstanding and to be competely clear on that, I do not expect process monitoring, but partial output in the html report when a part (scenario) is finished. This means I expect behaviour which existed in former Cucumber versions. The quotes should be seen as additional supporting opinions only, which shall mean that there are other users as well seeing a problem in this changed behaviour.

[ciaranmcnulty]

Just to check, is this in a case where cucumber is writing to the filesystem and you're opening the file in your web browser?

[seleniumghost]

A) It occurs at any time when the test is still running and not every scenario in it has been completed yet.

B) It occurs if the 1st scenario is finished and the test currently runs in the 2nd scenario.

C) It stops occurring when the complete test run is finished (Takes a few seconds after that of course).

The file size of the html report file is 22 KB at the start. This file size does not change for A) and not for B) and only changes with C) to e.g. 2327 KB. This means it occurs when cucumber currently is not writing to the html file.

[aurelien-reeves]

I don't think the actual html-formatter would be able to show any progress. It aims to generate an all-inclusive static report once the execution is done.

Yes, at some point, it kinda shows some progress, but that was only a side effect of how it was built before.

What you are requesting would require a brand new dedicated formatter which would actually be able to process messages stream as they arrive and thus monitor the execution based on that stream.

[mpkorstanje]

This means I expect behaviour which existed in former Cucumber versions.

Sounds like a case of Hyrums law in action.

That reports could be viewed while test execution was in progress is merely a coincidence. It is only by the willingness of a browser to interpret and render malformed html documents that this could be done.

I don't see a reason to support this.

https://www.hyrumslaw.com/

[seleniumghost]

What you are requesting would require a brand new dedicated formatter which would actually be able to process messages stream as they arrive and thus monitor the execution based on that stream.

I just request what was working before, for many many years. Nothing new from this retro-perspective. I understand that this might be quite a turn-around to get back there and if there were any alternatives I would have not taken the effort to open and discuss this issue, but I found no alternative and user marnen mentioned above did not either.

I don't see a reason to support this.

The reasons have been told at start:

1. If you have a lot of scenarios or long running scenarios or even both of this, you see nothing until all is finished. This is a major drawback.
2. But this also means that if the test execution aborts to whatever reason, all is lost.
3. If there is any issue in the formatter itself, not only a part, but the whole report is affected.

That reports could be viewed while test execution was in progress is merely a coincidence. It is only by the willingness of a browser to interpret and render malformed html documents that this could be done.

No, I just checked this. The former partial html report was not malformed, but containing the partial content.

Sounds like a case of Hyrums law in action.

Wow, how easy it can be to block an issue of expected behaviour which was working in former versions and where also other users see a problem within. This is kind of unexpected hybris which makes me feel really sad for this world. :-(

[mpkorstanje]

https://xkcd.com/1172/

[aslakhellesoy]

I think maybe this can be solved without too much effort. Rather than writing a window.CUCUMBER_MESSAGES array at the end of the run, we could write messages as they come in - wrapped in a JavaScript function call:

<script>processMessage({...})</script>
<script>processMessage({...})</script>
<script>processMessage({...})</script>

Every time another line is added, the UI would update. We'd have to make sure it doesn't impact performance when a finished page is loaded, but I don't think that's hard to do either.

[aslakhellesoy]

Thanks for raising this issue @seleniumghost - I think fixing this would be an improvement. I might take a crack at it one day, if I can find some time.

[mpkorstanje]

Great, so now we have:

1. A blank report when opened prematurely with terrible UX and with no fix
2. A proposed commitment to a solution that is unlikely to be met.

[mpkorstanje]

We'd have to make sure it doesn't impact performance when a finished page is loaded, but I don't think that's hard to do either.

Just because it happens to work doesn't mean we should do it. That's how you get complex bugs that are hard to solve for use cases nobody should use.

There are better solutions to progress monitoring and they don't involve relying on a browsers handling of malformed files.

Keep it simple, stupid.

[aurelien-reeves]

I agree with @mpkorstanje

Also it would add an overload to the already too big html report

[seleniumghost]

Thanks for raising this issue @seleniumghost - I think fixing this would be an improvement. I might take a crack at it one day, if I can find some time.

Thank you for picking this up. I really appreciate that and hope you can find some time soon as it is a major obstacle to move to a Cucumber version with the new html formatter for us.

[aslakhellesoy]

I've got this working in #121. This adds 20 extra bytes for each message (because each message needs to be wrapped in <script>p( );</script>. This is negligible compared to the size of the bundled JavaScript.

[mpkorstanje]

You've not got it working. You've hacked it together.

You now have to make sure the report doesn't fail when messages are missing or out of order (think parallel execution). And you have to make sure of that in the future. For every Cucumber implementation.

It's not worth the complexity.

[mpkorstanje]

Owh and another fun case to deal with. What if Cucumber stops writing half way through? Do we now have a report that says the run has passed? No failures were written after all.

And nobody will see that the html is malformed. So now we also need an in progress indicator to signal that.

[aurelien-reeves]

It is not because it is possible to make it work using hacks that we should actually do that. @mpkorstanje perfectly explained why it would be wrong.

I suggest to not support that as part of the html-formatter.

If anyone would like that support, I suggest to fork the formatter and to support it on its own.

[aslakhellesoy]

You've not got it working. You've hacked it together.

It's a draft.

I've verified that it works in automated tests as well as in a manual test as described in the README.md in the pr.

You now have to make sure the report doesn't fail when messages are missing or out of order (think parallel execution). And you have to make sure of that in the future. For every Cucumber implementation.

My change does not change the order in which messages are written to the HTML page nor the order in which they are processed by the Cucumber-React library.

If a message formatter emits messages out of order, then that is a bug in Cucumber that needs to be addressed. If it were the case, it would also break HTML reports in the current implementation.

Consumers should assume messages are in order. That said, I'm not aware that any Cucumber implementations emit messages out of order, even for parallel execution.

Your argument doesn't make any sense to me.

It's not worth the complexity.

Please be more specific. Where exactly is the added complexity?

Owh and another fun case to deal with. What if Cucumber stops writing half way through? Do we now have a report that says the run has passed? No failures were written after all.

You'll have a report that looks like the following:

5 of the scenarios are in an unknown state (which means they have not been executed). If this isn't a clear enough signpost that the run did not finish, we should address this in Cucumber-React.

You also said in Slack:

It took me 5 minutes to come up with a problem case for progressively loading html reports. And I'm sure I can find more if I put in the time and effort.

Are you referring to the one about messages out of order above? Can you share any other problem cases?

But the thing is I don't want to. From a technical point of view you are building on undefined browser behavior.

I think you are wrong about this. Since the dawn of the WWW, browsers have been designed specifically to render partial pages, because people have slow Internet connections. Browsers do not need balanced tags to render a page. The HTML5 spec doesn't require it either. XHTML is dead.

That's not a good basis for solid software. Period. That's where this discussion should have ended.

I'll end the discussion when I have seen valid arguments against my implementation. So far I haven't seen any.

And more often then not the problems caused by hacks like this come back.

What exactly makes this a "hack"? You need to be more specific. I am leaning on functionality that is supported in all modern browsers - rendering of partial content.

But you also have a habit of moving into new projects before that happens. So right now I'm rather annoyed that instead of seeking consensus you're just doing an implementation.

I am seeking consensus by sharing a draft PR and refuting arguments I believe are invalid.

[aurelien-reeves]

The fact that it relies on the user keeping refreshing the browser to show the progress, without actually any notification indicating that it is actually in progress or finished, illustrates that this remains a hack and is error-prone

When opening a report, the user actually have no clue if the execution is finished or not

It may even lead other users into thinking the execution is finished while it is not.

If a user open a report before the end of the execution, he may think the execution is finished and think there is something wrong with the "unknwon" scenarios.

In a big report with dozen or more scenarios, that may be an issue.

[aurelien-reeves]

Also, that would prevent us - or at least make this far more complicated that this is already the case - to optimize the report weight.

To keep that "feature" working, we would not be able anymore to post-process messages before generating the report.

[mpkorstanje]

If a message formatter emits messages out of order, then that is a bug in Cucumber that needs to be addressed. If it were the case, it would also break HTML reports in the current implementation.

It can't be a bug. There is no order specified in the message protocol. Every order is currently correct. Messages can be written in reverse order and it would be correct.

And at best you can introduce a partial order because of parallel execution - the format would lose it's write streamability otherwise. And you know this, it's what got you looking into petrinets but we never included it in the specs.

Currently the html formatter accepts the messages in any order because it processes them as a batch. However this doesn't mean the report will be correct for any partial set of messages.

https://github.com/cucumber/react-components/blob/ad4eb5cfb98afe348266e19860dc3f74cc68cfee/src/components/app/EnvelopesWrapper.tsx#L18-L22

[mpkorstanje]

Since the dawn of the WWW, browsers have been designed specifically to render partial pages, because people have slow Internet connections. Browsers do not need balanced tags to render a page.

Slow pages are a different problem from incomplete pages.

[mpkorstanje]

5 of the scenarios are in an unknown state (which means they have not been executed). If this isn't a clear enough signpost that the run did not finish, we should address this in Cucumber-React.

At least now we're getting somewhere. You're starting to see atleast one feature creep in. However the argument is that we shouldn't even bother because none of the additional work is worth it. So we should not. Instead keep it simple, stupid.

[marnen]

The fact that it relies on the user keeping refreshing the browser to show the progress, without actually any notification indicating that it is actually in progress or finished, illustrates that this remains a hack and is error-prone

If indication of it being finished is an issue: RSpec solved this years ago, by having some simple JS that updates a little progress bar at the top, but still allows the HTML to render incrementally. I assume that pretty much the exact same code would work in Cucumber.

Or we could keep it even simpler: just write something like “Done. 100 scenarios executed” at the end of the document.

@mpkorstanje I get that you don’t personally find incremental reports useful, but please recognize that your usage patterns are not everyone’s usage patterns. I relied on the incremental behavior for a long time, and more importantly perhaps, I encountered resistance to using Cucumber at a recent job where I introduced it, because a colleague complained that he couldn’t review the report without waiting for the entire run.

I’d be happy to see the option of an incrementally written HTML test report with no JS whatsoever, frankly; I don’t really need anything that the JS gives me, and it seems to make the report harder to write in a way that’s meaningful during the run.

[mattwynne]

Thanks for your perspective @marnen. RSpec is doing something very similar to what Aslak has suggested.

[mattwynne]

I'm glad to see that this thread has calmed down, but I still think it's important to remind everyone here of our code of conduct.

I'd particularly like to draw your attention to this guiding principle:

We are kind. We are warm, courteous, patient, respectful and encouraging. Disagreements, both social and technical, happen all the time and Cucumber is no exception. These differences are where we can learn the most — if everyone thought the same all the time life would be boring! We’re respectful of other people’s opinions, experience, their skills, their commitments and their efforts. We make an effort to be compassionate towards one another, to assume good intention of others, and to seek to understand first before seeking to be understood.

I don't think this thread sets a great example of this principle.

I'm not going to call anyone out in public, but please examine your own behaviour and reflect what you could do to stick to this principle. Everyone here deserves to have their experience and perspective treated with respect.

[rajebo1805]

in new version: 7.13.0, Html reports are generated with empty response. Any help would be helpful

[saikabade]

Hello, I'm experiencing this issue of reports being blank frequently. I do see it happening more often with one of the feature files (Product Page feature) which has total 5 tests of which 2 are simple scenarios and 3 are scenario outlines (with 1,1 and 3 examples), the report generated is blank often. Upgraded from 9.3.0 to version 9.6.0 and no luck. Any suggestions or work-arounds would be greatly appreciated! TIA

[mattwynne]

This issue is about the fact that the report is blank if scenarios have not finished yet.

If you have an issue where the Cucumber test run is finished, and the report is blank, could you please raise that as a separate issue?

[saikabade]

@mattwynne created a new issue: https://github.com/cucumber/html-formatter/issues/277, would be great if someone can be assigned. Thanks!