search

jenkinsci / junit-plugin

Allows JUnit-format test results to be published
https://plugins.jenkins.io/junit
MIT License
76 stars 335 forks source link

Testcases marked by behave's reporter with status Untested are considered Passed #262

Open hesteban-tuenti opened 3 years ago

hesteban-tuenti commented 3 years ago

When generating the xml files from behave's junit reporter , if a feature has not been executed for instance because a Runtime Error has been generated in a hook and the execution stopped,all the testcases within the feature are reported as untested. In the example the feature has 275 testcases, none of which have been executed. Captura de pantalla 2021-03-22 a las 16 35 39

After recording the xml data in Jenkins , these scenarios are considered Passed by default Captura de pantalla 2021-03-23 a las 13 26 52

Shouldn't they be considered skipped instead?

Version report

Jenkins version: 2.263.1 Junit plugin version: 1.47

timja commented 3 years ago

Shouldn't behave be outputting it as skipped and not inventing a new status?

Saying that without looking into it too much other than only seeing behave coming up in search results for that status

hesteban-tuenti commented 3 years ago

Thanks , and probably you're right. I think behave has a flag to treat these scenarios as Skipped with other side effects , but still I wouldn't expect this behaviour in the Junit plugin because it has a huge impact on the results published