Support for attachments / embeddings as per Cucumber and Behave

Johnlon commented 2 weeks ago

🤔 What's the problem you're trying to solve?

In cucumber the Scenario class provides a means to attach(bytes, mediatype, name) but we can't see the same capability in godog. How do we add typed attachments to the output report file?

✨ What's your proposed solution?

Add the necessary API and docs

⛏ Have you considered any alternatives or workarounds?

None known

📚 Any additional context?

No response

vearutop commented 2 weeks ago

@Johnlon I'd be happy to spend some time on that, perhaps you can help by providing some more expectations.

It would be very helpful to see an example report (in cucumber JSON "messages" format?) that does what you need (has attachments where they can be defined) in standard-compliant format.

Johnlon commented 2 weeks ago

Hi take a look at cucumber JVM JSON report output..... It has this at the step level ...

"embeddings" [ { "data" : "string of base 64 data", "mine_type" : "string", "name": "string" } ];

In the JVM api see the api like .. Scenario.attach(name, mimetype, data)

The newer "messages" format in Cucumber JVM has the EmbedEvent which also represents this data.

Additionally the new Cucumber HTML formatter (Cucumber React) consumes these embedding messages and displays them correctly.

Re the attach API ..

The same kind of thing is in the main branch of Behave but because Behave looks like it's dying (ie it doesn't make prod releases anymore, 7 years) then I guess folk build their own dist of Behave containing that feature.

Johnlon commented 2 weeks ago

The cuke json schema isnt maintained ... https://github.com/cucumber/cucumber-json-schema/issues/6 but the cuke JVM tools and the Behave patches populate the JSON report with the embeddings I mentioned.

In the messages format the ndjson file format has an object "attachment". cucumber/messages java defines Attachement.java pregenerated and checked in. messages go has messages.go containing "Attachment"

vearutop commented 2 weeks ago

I think this https://github.com/cucumber/messages (json schema) is the current main reporting format (and it is used in godog internally).

What I'm interested in is a full example file that exhibits variable attachments in different places (also if possible the case of multiple attachments per step result), because I'm not super fluent in these semantics and where attachments should be applicable and how the relation is built.

vearutop commented 2 weeks ago

From the implementation perspective I think we can introduce something like ctx := godog.AddAttachment(ctx, attachment) and then catch that information between the steps.

Johnlon commented 2 weeks ago

From the implementation perspective I think we can introduce something like ctx := godog.AddAttachment(ctx, attachment) and then catch that information between the steps.

Steps may add multiple attachments. I can't recall if hooks can also do it but I'm pretty sure they can.

There is a cucumber compatibility kit that would help I suspect make the Json report compatible.

--

I already started looking myself.

I'm looking at godog and various tickets in he backlog and wondering if a much more severe treatment of godog is needed to move it forward, a treatment that would probably allow for attachments along the way. But it might mean forking it (at least for a bit) to make my life easier.

Why don't you also kick the wheels on a fix and we can compare notes when either of us have anything to show for it.

Johnlon commented 2 weeks ago

I think this https://github.com/cucumber/messages (json schema) is the current main reporting format (and it is used in godog internally).

I don't think that's right. On some other ticket I wrote up this problem. The ticket where aslak is saying we should use the "message" format.

Some events in godog aren't in that module and the ones that have similar names have different properties.
I could be wrong but take a look. My recollection is that it is miles off.

Johnlon commented 2 weeks ago

Re my previous comment. This is why I think that godog possibly needs more radical work

Likely support for messages https://github.com/cucumber/godog/issues/341 would require a solution for attachments as attachments are part of the model.

Johnlon commented 1 week ago

I've had a go at it myself here : https://github.com/cucumber/godog/pull/623

Something strange about the coverage - stuff I've not touched being reported and lines I know are tested aren't being counted?

cucumber / godog