search

TNG / JGiven

Behavior-Driven Development in plain Java
http://jgiven.org
Apache License 2.0
436 stars 99 forks source link

JUnit5: failed assumptions are handled like failed test cases in JGiven reports #1625

Open mtthsAtrv opened 3 months ago

mtthsAtrv commented 3 months ago

Hi,

if we execute JGiven tests under JUnit5 and an assumption fails, this is still treated like the test case failed because of an exception.

There was a fix that has been delivered for https://github.com/TNG/JGiven/issues/1465 , but we still observe that effect, even with JGiven version 1.3.1.

After debugging this we observed that method JGivenExtension#afterTestExecution(ExtensionContext) treats all Exceptions the same way and makes no difference whether there was a exception caused by assumptions or other reasons.

grafik

Also see similar Issue https://github.com/TNG/JGiven/issues/185 for Junit 4.

l-1squared commented 3 months ago

@mtthsAtrv sorry for the slow progress. I have been swamped and did not yet find the time to look deeper into the matter

mtthsAtrv commented 3 months ago

Hi @l-1squared ,

thanks for the information :-)

Might I ask you if you already have a raw guess when a solution might be available for this issue?

Thanks a lot

l-1squared commented 3 months ago

I will devote as much time to this as possible. The first thing I want to do is write some JGiven tests so that we have an end-to-end view on failing assumptions. I now want to eradicate this issue once and for all. Then I'll start looking for the actual error. I've linked the branch I'm working on.

mtthsAtrv commented 3 months ago

Thanks a lot for your feedback @l-1squared :-)

We are looking forward to the solution ...

l-1squared commented 3 months ago

At least I now managed to reproduce the issue :)

mtthsAtrv commented 3 months ago

As a quick hack I have locally tried this change in class com.tngtech.jgiven.junit5.JGivenExtension, at first sight it seemed to work.

grafik

But I can not really assess if this is the correct and clean solution to the problem, maybe there are some side effects I do not see.

l-1squared commented 2 months ago

Hi, unfortunately this is taking forever. The reason so far was that I made some harmless looking changes in my testsetup, which turned out to produce a host of weird behavior and errors.

I think I by now drilled down to the actual issue, and thankfully you have already provided me with a pointer. One interesting thing that is not covered by this is, however, that we, for Junit5 seem to record scenarios at a different place in the test lifecycle than we do for junit4 and testng.

Because in my current setup. If I fail for the first step, I will completely omit that the scenario existed (which also doesn't sound quite right) whereas for JUnit5 I don't.

l-1squared commented 2 months ago

Ah, so I think I've figured it out.

But, it appears as if previously, failed assumpitons would have just omitted the scenarios from the report. This does not sit quite right with me so I'm trying to make them behave like failing scenarios, but they will show up with a grey marker.

mtthsAtrv commented 2 months ago

I think that would be fine for us, but could you please - in addition to using another colour - also replace the word "failed" by "skipped" or another description that is consistent with JUnit (surefire) test reports?

Currently ist looks like

grafik

and the word "failed" seems not be quite right on failed assumptions.

l-1squared commented 2 months ago

The plan right now is to introduce a new Step and Scenario status called "ABORTED" that will show up in grey like pending scenarios but will in its behavior be more akin to "failed" scenarios.

mtthsAtrv commented 2 months ago

Thanks a lot for your feedback - that sounds good to us.

Could you please let us know as soon as a (preview) version is available?

mtthsAtrv commented 1 month ago

Hi l-1squared,

I wanted to ask if maybe you could already give us a prediction when the JGiven version fixing this issue might be available?

Would be great, because we could need this information for our Sprint plannings ...

Kind regards

l-1squared commented 1 month ago

HI @mtthsAtrv as much as I would like to give you a date for this. I cannot. The time I can devote to this project depends on a multitude of factors that are in combination hard to obtain estimates from. Just this month it turned out that I could not devote any time (Thus my late response). I can promise you that I keep this the top priority for JGiven. I'm sorry that I cannot deliver a more positive answer for you.

mtthsAtrv commented 1 month ago

Hi @l-1squared ,

thanks a lot for this information.

So we'll sadly have to come to terms with that effect for the next time.

We'll keep watching this issue and are looking forward to a solution ...

Kind regards