search

jenkinsci / junit-plugin

Allows JUnit-format test results to be published
https://plugins.jenkins.io/junit
MIT License
76 stars 335 forks source link

Failed testcase counted as pass in Junit #463

Closed Sibicoder closed 1 year ago

Sibicoder commented 1 year ago

Jenkins and plugins versions report

Environment ```Jenkins: 2.346.1 OS: Linux - 4.15.0-189-generic --- ace-editor:1.1 amazon-ecr:1.73.v741d474abe74 ant:475.vf34069fef73c antisamy-markup-formatter:2.7 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 authentication-tokens:1.4 aws-credentials:191.vcb_f183ce58b_9 aws-java-sdk:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-cloudformation:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-codebuild:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ec2:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecr:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ecs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-elasticbeanstalk:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-iam:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-logs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-minimal:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-sns:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-sqs:1.12.246-349.v96b_b_f7eb_a_c3c aws-java-sdk-ssm:1.12.246-349.v96b_b_f7eb_a_c3c aws-secrets-manager-credentials-provider:1.2.0 bootstrap5-api:5.1.3-7 bouncycastle-api:2.26 branch-api:2.1046.v0ca_37783ecc5 build-timeout:1.21 build-timestamp:1.0.3 caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.4 cloudbees-folder:6.740.ve4f4ffa_dea_54 command-launcher:84.v4a_97f2027398 credentials:1139.veb_9579fca_33b_ credentials-binding:523.vd859a_4b_122e6 display-url-api:2.3.6 docker-commons:1.19 durable-task:496.va67c6f9eefa7 echarts-api:5.3.3-1 email-ext:2.91 font-awesome-api:6.1.1-1 git:4.11.4 git-client:3.11.2 github:1.34.5 github-api:1.303-400.v35c2d8258028 github-branch-source:1677.v731f745ea_0cf gradle:1.39.4 handlebars:3.0.8 jackson2-api:2.13.3-285.vc03c0256d517 jakarta-activation-api:2.0.1-1 jakarta-mail-api:2.0.1-1 javax-activation-api:1.2.0-4 javax-mail-api:1.6.2-7 jaxb:2.3.6-1 jdk-tool:55.v1b_32b_6ca_f9ca jjwt-api:0.11.5-77.v646c772fddb_0 jobConfigHistory:1165.v8cc9fd1f4597 jquery3-api:3.6.0-4 jsch:0.1.55.61.va_e9ee26616e7 junit:1119.1121.vc43d0fc45561 junit-realtime-test-reporter:90.v7dc2c937f9a_8 ldap:2.11 mailer:435.v79ef3972b_5c7 matrix-auth:3.1.5 matrix-project:772.v494f19991984 mina-sshd-api-common:2.8.0-36.v8e25ce90d4b_1 mina-sshd-api-core:2.8.0-36.v8e25ce90d4b_1 momentjs:1.1.1 oauth-credentials:0.5 okhttp-api:4.9.3-108.v0feda04578cf pam-auth:1.10 parameterized-trigger:2.45 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:612.v84da_9c54906d pipeline-input-step:449.v77f0e8b_845c4 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2114.v2654ca_721309 pipeline-model-definition:2.2114.v2654ca_721309 pipeline-model-extensions:2.2114.v2654ca_721309 pipeline-rest-api:2.24 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2114.v2654ca_721309 pipeline-stage-view:2.24 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.17.0 popper2-api:2.11.5-2 resource-disposer:0.19 scm-api:621.vda_a_b_055e58f7 script-security:1175.v4b_d517d6db_f0 snakeyaml-api:1.30.2-76.vc104f7ce9870 ssh-credentials:295.vced876c18eb_4 ssh-slaves:1.821.vd834f8a_c390e sshd:3.242.va_db_9da_b_26a_c3 structs:324.va_f5d6774f3a_d testInProgress:1.4 timestamper:1.18 token-macro:308.v4f2b_ed62b_b_16 trilead-api:1.67.vc3938a_35172f variant:59.vf075fe829ccb windows-slaves:1.8.1 workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1188.v0016b_4f29881 workflow-basic-steps:986.v6b_9c830a_6b_37 workflow-cps:2759.v87459c4eea_ca_ workflow-durable-task-step:1199.v02b_9244f8064 workflow-job:1207.ve6191ff089f8 workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:838.va_3a_087b_4055b ws-cleanup:0.42 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Ubuntu

Reproduction steps

Attaching Jenkinsfile, console output and API JSON file of build page

JenkinsFile

pipeline { .... stages { stage('Build') { steps { ..... } } stage('Test') { steps { script { sh ''' NO_COLOR=1 ./node_modules/.bin/cypress run --spec myfile.spec.js --browser chrome --headed --reporter junit --reporter-options mochaFile=output.xml,toConsole=true --config baseUrl=$baseUrl || true echo "$BUILD_URL" ''' } } } stage('Publish JUNIT result') { steps { junit 'output.xml' } }
stage('Publish in Dashboard') { steps { script { sh ''' ... ''' } } } } }

Output tput: No value for $TERM and no -T specified

(Run Starting)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Cypress: 10.11.0 │ │ Browser: Chrome 107 │ │ Node Version: v12.22.12 (/usr/bin/node) │ │ Specs: 1 found (myfile.spec.js) │ │ Searched: myfile.spec.js │ └────────────────────────────────────────────────────────────────────────────────────────────────┘

────────────────────────────────────────────────────────────────────────────────────────────────────

Running: myfile.spec.js (1 of 1)

Oops...we found an error preparing this test file:

myfile.spec.js Syntax error in json file while reading for the js file This occurred while Cypress was compiling and bundling your test code. This is usually caused by:

Fix the error in your code and re-run your tests.

(Results)

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ Tests: 0 │ │ Passing: 0 │ │ Failing: 1 │ │ Pending: 0 │ │ Skipped: 0 │ │ Screenshots: 0 │ │ Video: false │ │ Duration: 0 seconds │ │ Spec Ran: myfile.spec.js │ └────────────────────────────────────────────────────────────────────────────────────────────────┘

(Run Finished)

   Spec                                              Tests  Passing  Failing  Pending  Skipped

┌────────────────────────────────────────────────────────────────────────────────────────────────┐ │ ✖ myfile.spec.js 0ms - - 1 - - │ └────────────────────────────────────────────────────────────────────────────────────────────────┘ ✖ 1 of 1 failed (100%) 0ms - - 1 - -

API JSON of build page { _class: "org.jenkinsci.plugins.workflow.job.WorkflowRun", actions: [ .... { _class: "hudson.model.CauseAction", causes: [ { _class: "hudson.model.Cause$UserIdCause", shortDescription: "Started by user admin", userId: "admin", userName: "admin" } ] }, { _class: "org.jenkinsci.plugins.workflow.libs.LibrariesAction" }, { }, { _class: "org.jenkinsci.plugins.workflow.cps.EnvActionImpl" }, { _class: "hudson.tasks.junit.TestResultAction", failCount: 0, skipCount: 0, totalCount: 1, urlName: "testReport" }, { }, { }, { _class: "org.jenkinsci.plugins.displayurlapi.actions.RunDisplayAction" }, { _class: "org.jenkinsci.plugins.pipeline.modeldefinition.actions.RestartDeclarativePipelineAction" }, { }, { _class: "org.jenkinsci.plugins.workflow.job.views.FlowGraphAction" }, { }, { }, { } ], artifacts: [ ], building: false, description: null, displayName: "#217", duration: 54736, estimatedDuration: 65417, executor: null, fullDisplayName: "MY JOB #217", id: "217", keepLog: false, number: 217, queueId: 10810, result: "SUCCESS", timestamp: 1669182140642, url: "http://localhost:8080/job/MY JOB/217/", changeSets: [ ], culprits: [ ], inProgress: false, nextBuild: null, previousBuild: { number: 216, url: "http://localhost:8080/job/MY JOB/216/" } }

Expected Results

JUnit should return failed count properly. Script exited due to syntax error, but junit still reports it as passed

Actual Results

JUnit returns failed and skipped cases as 0 even though the script has exited due to syntax error

Anything else?

No response

uhafner commented 1 year ago

It seems that your tests did not run due to a compile error. What is the content of the XML result file? What is the log output of the JUnit plugin?

Sibicoder commented 1 year ago

Yes. Tests failed due to compile error.

Log output of JUnit plugin

┌──────────────────────────┐ │ Tests: 0 │ │ Passing: 0 │ │ Failing: 1 │ │ Pending: 0 │ │ Skipped: 0 │ │ Screenshots: 0 │ │ Video: fals │ │ Duration: 0 seconds │ │ Spec Ran: myfile.spec.js │ └──────────────────────────┘

tput: No value for $TERM and no -T specified

(Run Finished)

   Spec                                              Tests  Passing  Failing  Pending  Skipped

┌──────────────────────────────────┐ │ ✖ myfile.spec.js 0ms - - 1 - - │ └──────────────────────────────────┘ ✖ 1 of 1 failed (100%) 0ms - - 1 - -

Content of XML file below

<?xml version="1.0" encoding="UTF-8"?>
<testsuites name="Mocha Tests" time="52.642" tests="1" **failures="0">**
  <testsuite name="Root Suite" timestamp="2022-11-17T02:44:09" tests="1" file="myfile.spec.js" time="52.635" failures="0">
    <testcase name="testcase name" time="52.497" classname="testcase name">
    </testcase>
  </testsuite>
</testsuites>
uhafner commented 1 year ago

Then the JUnit plugin does report the correct numbers. It does only read the content of the XML file. So you need to adapt your build scripts.

Sibicoder commented 1 year ago

@uhafner npm run command has this options enabled.

-reporter junit --reporter-options mochaFile=output.xml,toConsole=true

XML is generated through this command only. Please help where this bug need to be filed

uhafner commented 1 year ago

From the log above:

Fix the error in your code and re-run your tests.

This looks not like a bug.