Unimplemented steps do not appear as skipped or failed

TrueDub commented 12 years ago

When running Cucumber tests, an unimplemented step will fail the build (correctly), but the Jenkins reporting plugin fails to mark the feature as failed (or even skipped).

On drilling into the feature file, the relevant step is marked in yellow, but the main reporting page does not reflect this.

I have a screenshot of the relevant areas, and the JSON file saved, but cannot add these to the issue. If any further information is required please let me know.

Thanks for your hard work on this plugin, it's extremely useful

kingsleyh commented 12 years ago

Hi - could you send me the screenshot and json file? maybe reply to this in email?

TrueDub commented 12 years ago

Forgive my stupidity - where do I find your email address?

kingsleyh commented 12 years ago

Hi

You can email me at kingsley@mindflowsolutions.com

--k

Sent from my iPhone

On 23 May 2012, at 16:30, TrueDubreply@reply.github.com wrote:

Forgive my stupidity - where do I find your email address?

Reply to this email directly or view it on GitHub: https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java/issues/7#issuecomment-5876067

TrueDub commented 12 years ago

email sent

poum commented 12 years ago

Hi,

First, many thanks for the plugin. I've the same problem as TrueDub: console output 4 skipped test which are yellow in report but shown as passed in graphics and tables ... I've tried with PendingException & @Ignore: same result. If necessary, I can send feature, screenshot and json.

Jenkins: 1.462 cucumber-jvm 1.0.8 plugin: 0.0.9 maven: 3.0.4 java: 1.6 (openjdk)

poum commented 12 years ago

After inspecting the json file, it appears that the "skipped" tests have a "undefined" status instead... The status seams to be correct in xml file.... Maybe cucumber-jvm is guilty, not the plugin ....

poum commented 12 years ago

I've opened an issue for cucumber-jvm: https://github.com/cucumber/cucumber-jvm/issues/328

kingsleyh commented 12 years ago

Hi,

Thanks for your feedback :) Actually it's the plugin - I didn't take any notice of undefined status in the json file - I just marked it orange colour in the report. But I will make some changes to that for the next release.

On 25 May 2012, at 14:29, poum wrote:

After inspecting the json file, it appears that the "skipped" tests have a "undefined" status instead... The status seams to be correct in xml file.... Maybe cucumber-jvm is guilty, not the plugin ....

Reply to this email directly or view it on GitHub: https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java/issues/7#issuecomment-5930298

poum commented 12 years ago

Ok, but I did'nt understand why test are marked "undefined" in json file instead of "skipped" ... Thanks again.

kingsleyh commented 12 years ago

Yeah me either I think that is a bug with cucumber-jvm - it would make for sense if it was marked skipped. On 25 May 2012, at 14:44, poum wrote:

Ok, but I did'nt understand why test are marked "undefined" in json file instead of "skipped" ... Thanks again.

Reply to this email directly or view it on GitHub: https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java/issues/7#issuecomment-5930625

TrueDub commented 12 years ago

Actually it wouldn't - the steps are undefined in that no step definitions exist. Skipped implies they're defined but not executed.

Pleas forgive my brevity - Sent from my iPhone

On 25 May 2012, at 14:47, Kingsley Hendricksereply@reply.github.com wrote:

Yeah me either I think that is a bug with cucumber-jvm - it would make for sense if it was marked skipped. On 25 May 2012, at 14:44, poum wrote:

Ok, but I did'nt understand why test are marked "undefined" in json file instead of "skipped" ... Thanks again.

Reply to this email directly or view it on GitHub: https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java/issues/7#issuecomment-5930625

Reply to this email directly or view it on GitHub: https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java/issues/7#issuecomment-5930691

poum commented 12 years ago

Ok, but I've tried the two cases: PendingException (ok, it could be undefined) but also @Ignore on a fully defined step: IMO, this one, at least, should be marked as skipped, no ?

kingsleyh commented 12 years ago

hi - I have had a look at this - the cucumber json file marks the steps as skipped if they are skipped - so the plugin marks lines blue if they are skipped - this does not currently fail the build.

If the steps are marked as undefined - the plugin marks these lines orange - this does not currently fail the build

so the question is what would you like the plugin to do:

do you want the plugin to fail the build if any undefined steps are found?
do you want the plugin to fail the build if any skipped steps are found?

I can add some configuration to allow you choose if you want undefined or skipped tests to fail the build?

poum commented 12 years ago

For me, the current behaviour is goo for me: build ok for undefined and skipped steps If this step are signaled skipped in reports: this allow stakeholders to have a progress estimation. Eventually, the undefined steps could be marked as 'TODO' (purple for example)...

But I did'nt succeed to have "skipped" in cucumber-jvm report: both where signaled as 'undefined' (I mean the 'throws new PendingException()' and the '@ignore')

kingsleyh commented 12 years ago

Pending release will contain fix

poum commented 12 years ago

OK great ! Many thanks

TrueDub commented 12 years ago

The configuration element would work for me - we'd like to have a build fail due to undefined steps (our Maven builds already fail).

Thanks for all your help

kingsleyh commented 12 years ago

Hi

It's done and in the release I did yesterday - the latest one 0.0.10 in the downloads area

Sent from my iPhone

On 28 May 2012, at 13:43, TrueDubreply@reply.github.com wrote:

The configuration element would work for me - we'd like to have a build fail due to undefined steps (our Maven builds already fail).

Thanks for all your help

Reply to this email directly or view it on GitHub: https://github.com/masterthought/jenkins-cucumber-jvm-reports-plugin-java/issues/7#issuecomment-5964754

poum commented 12 years ago

0.0.10 works very well for me too. Many thanks again.

For the initial issue, I was misleaded by PendingException which cause test to fail. So: not implemented > pending (ok) throw PendingException > fail (?) @Ignore > skipped (ok)

Any idea to make PendingException generating a pending result ?

kingsleyh commented 12 years ago

I'll have to investigate a bit more for PendingException - if the json result shows that step as failed then all I can do is report it as failed. It may be possible to parse the error message and report it as pending that way - but to be honest I think it's better if it fails in the case of PendingException - those steps should be removed or implemented before checking in really.

poum commented 12 years ago

I fully agree with you: the plugin behaviour is correct, but I'm not sure cucumber-jvm behaviour is ... I considered the PendingException as a special one, specifically designed to mark step as Pending, not failing... I will up my issue in cucumber-jvm

damianszczepanik / cucumber-reporting-jenkins

Unimplemented steps do not appear as skipped or failed #7