search

jenkinsci / xray-connector-plugin

Xray Test Management Connector for Jenkins
https://plugins.jenkins.io/xray-connector/
MIT License
16 stars 13 forks source link

Import multiple Junit multipart xml results to same test execution #72

Open LSatBosch opened 1 year ago

LSatBosch commented 1 year ago

Jenkins and plugins versions report

Environment ```text Jenkins: 2.375.3 OS: Linux - 5.15.0-67-generic Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- Parameterized-Remote-Trigger:3.1.6.3 ace-editor:1.1 ant:481.v7b_09e538fcca antisamy-markup-formatter:159.v25b_c67cd35fb_ apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5 artifactory:3.18.0 authentication-tokens:1.4 authorize-project:1.5.1 blueocean:1.27.3 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.27.3 blueocean-commons:1.27.3 blueocean-config:1.27.3 blueocean-core-js:1.27.3 blueocean-dashboard:1.27.3 blueocean-display-url:2.4.1 blueocean-events:1.27.3 blueocean-git-pipeline:1.27.3 blueocean-github-pipeline:1.27.3 blueocean-i18n:1.27.3 blueocean-jira:1.27.3 blueocean-jwt:1.27.3 blueocean-personalization:1.27.3 blueocean-pipeline-api-impl:1.27.3 blueocean-pipeline-editor:1.27.3 blueocean-pipeline-scm-api:1.27.3 blueocean-rest:1.27.3 blueocean-rest-impl:1.27.3 blueocean-web:1.27.3 bootstrap5-api:5.2.1-3 bouncycastle-api:2.27 branch-api:2.1071.v1a_188a_562481 build-blocker-plugin:1.7.8 build-timeout:1.28 build-token-root:151.va_e52fe3215fc caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.8.1 cloudbees-bitbucket-branch-source:800.va_b_b_9a_a_5035c1 cloudbees-disk-usage-simple:178.v1a_4d2f6359a_8 cloudbees-folder:6.815.v0dd5a_cb_40e0e command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-36.vc008c8fcda_7b_ config-file-provider:3.11.1 copyartifact:686.v6fd37018d7c2 credentials:1224.vc23ca_a_9a_2cb_0 credentials-binding:523.vd859a_4b_122e6 data-tables-api:1.12.1-4 declarative-pipeline-migration-assistant:1.5.6 declarative-pipeline-migration-assistant-api:1.5.6 disk-usage:0.28 display-url-api:2.3.7 docker-build-publish:1.4.0 docker-commons:419.v8e3cd84ef49c docker-java-api:3.2.13-68.va_875df25a_b_45 docker-plugin:1.3.0 docker-workflow:563.vd5d2e5c4007f durable-task:504.vb10d1ae5ba2f echarts-api:5.4.0-1 email-ext:2.95 envinject:2.901.v0038b_6471582 envinject-api:1.199.v3ce31253ed13 external-monitor-job:203.v683c09d993b_9 favorite:2.4.1 font-awesome-api:6.2.1-1 git:5.0.0 git-client:4.2.0 git-parameter:0.9.18 github:1.37.0 github-api:1.303-417.ve35d9dd78549 github-branch-source:1703.vd5a_2b_29c6cdc gradle:2.3.2 groovy:453.vcdb_a_c5c99890 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 instance-identity:142.v04572ca_5b_265 ionicons-api:45.vf54fca_5d2154 ivy:2.4 jackson2-api:2.14.2-319.v37853346a_229 jakarta-activation-api:2.0.1-3 jakarta-mail-api:2.0.1-3 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-6 javax-mail-api:1.6.2-9 jaxb:2.3.8-1 jdk-tool:63.v62d2fd4b_4793 jenkins-design-language:1.27.3 jersey2-api:2.39.1-1 jira:3.9 jjwt-api:0.11.5-77.v646c772fddb_0 job-dsl:1.82 jobConfigHistory:1207.vd28a_54732f92 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.1-2 jsch:0.1.55.61.va_e9ee26616e7 junit:1189.v1b_e593637fa_e kubernetes:3900.va_dce992317b_4 kubernetes-client-api:6.4.1-215.v2ed17097a_8e9 kubernetes-credentials:0.10.0 ldap:671.v2a_9192a_7419d localization-support:1.2 mailer:448.v5b_97805e3767 mapdb-api:1.0.9-28.vf251ce40855d matrix-auth:3.1.6 matrix-project:785.v06b_7f47b_c631 maven-plugin:3.21 mercurial:1260.vdfb_723cdcc81 metrics:4.2.13-420.vea_2f17932dd6 mina-sshd-api-common:2.9.2-62.v199162f0a_2f8 mina-sshd-api-core:2.9.2-62.v199162f0a_2f8 momentjs:1.1.1 monitoring:1.92.0 nodejs:1.6.0 oic-auth:2.5 okhttp-api:4.10.0-132.v7a_7b_91cef39c pam-auth:1.10 parameterized-trigger:2.45 persistent-parameter:1.3 pipeline-build-step:487.va_823138eee8b_ pipeline-github-lib:42.v0739460cda_c4 pipeline-graph-analysis:202.va_d268e64deb_3 pipeline-groovy-lib:629.vb_5627b_ee2104 pipeline-input-step:466.v6d0a_5df34f81 pipeline-milestone-step:111.v449306f708b_7 pipeline-model-api:2.2125.vddb_a_44a_d605e pipeline-model-definition:2.2125.vddb_a_44a_d605e pipeline-model-extensions:2.2125.vddb_a_44a_d605e pipeline-rest-api:2.32 pipeline-stage-step:305.ve96d0205c1c6 pipeline-stage-tags-metadata:2.2125.vddb_a_44a_d605e pipeline-stage-view:2.32 pipeline-utility-steps:2.15.1 plain-credentials:143.v1b_df8b_d3b_e48 plugin-util-api:2.20.0 popper-api:1.16.1-3 popper2-api:2.11.6-2 powershell:2.0 publish-over:0.22 publish-over-ssh:1.24 pubsub-light:1.17 rebuild:1.34 resource-disposer:0.21 role-strategy:587.588.v850a_20a_30162 scm-api:631.v9143df5b_e4a_a script-security:1229.v4880b_b_e905a_6 snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4 sonar:2.15 sse-gateway:1.26 ssh-agent:327.v230ecd01f86f ssh-credentials:305.v8f4381501156 ssh-slaves:2.877.v365f5eb_a_b_eec sshd:3.275.v9e17c10f2571 stashNotifier:1.28 structs:324.va_f5d6774f3a_d swarm:3.39 timestamper:1.23 token-macro:321.vd7cc1f2a_52c8 trilead-api:2.84.v72119de229b_7 variant:59.vf075fe829ccb workflow-aggregator:596.v8c21c963d92d workflow-api:1208.v0cc7c6e0da_9e workflow-basic-steps:1010.vf7a_b_98e847c1 workflow-cps:3653.v07ea_433c90b_4 workflow-durable-task-step:1241.v1a_63e465f943 workflow-job:1289.vd1c337fd5354 workflow-multibranch:733.v109046189126 workflow-scm-step:408.v7d5b_135a_b_d49 workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c xray-connector:2.6.1 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Controller: Jenkins: 2.375.3 OS: Linux - 5.15.0-67-generic Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

Node: OS: Windows Server 2022 Jenkins Agent executed as service with Jenkins Wrapper

Reproduction steps

  1. Run Junit tests with Jenkins scripted pipeline (Linux Controler, Windows node) via maven mvn test -Dtest="TestSuite"
  2. The test results are saved in the target folder as xml files
  3. Archive the test results in Jenkins junit testResults: /target\**\*.xml/, skipPublishingChecks: true

  4. The tests reports get reported to the same test execution in Jira with the following code in the JENKINSFILE: 

    step([$class: 'XrayImportBuilder', 
    endpointName: '/junit/multipart',
    endpoint: "JUNIT_MULTIPART",
    importFilePath: /target\**\*.xml/, 
    importToSameExecution: 'true',  // uses junit multipart endpoint for all if false is used here
    importInParallel: 'false', 
    serverInstance: jiraInstance,
    inputTestInfoSwitcher: 'fileContent',
    testImportInfo: // test import info
    """{
        "fields": {
                "project": {"key": "KEY"},
                "summary": "Automatic build with JenkinsTestSuite",
                "issuetype": {"name": "Test"},
                "components": [{"name":"PROGRAM"}]
        }
    }""",
    inputInfoSwitcher: 'fileContent',
    importInfo: // test execution import info
    """{
        "fields": {
            "project": {"key": "MSCTOOL"},
            "summary": "Jenkins automatic Test",
            "issuetype": {"name": "Test Execution"},
            "components": [{"name":"PROGRAM"}],
            "customfield_28348": ${testPlanKey}
        }
    }"""
])

If we use importToSameExecution: 'false' all works as expected. Multiple test executions get created with one test linked each. Our required fields components and summary are read in for each file from the given json string. The discribed issue only happens if we want to import to the same test execution.

Expected Results

The import of all found xml files via the junit multipart endpoint specified in the calling of the XrayImportBuilder. The creation of new test issues in Jira if no matching are found which all get imported to the same test execution.

Actual Results

The first file is imported via the junit/multipart endpoint and has the required fields components and summary read in from the json given, but the following other files are imported via the Junit xml endpoint only, resulting in the creation of the test issues to be rejected by Jira.

Log ``` 11:06:17 Starting XRAY: Results Import Task... 11:06:17 ########################################################## 11:06:17 #### Xray is importing the execution results #### 11:06:17 ########################################################## 11:06:17 File found: C:\PATH\TO\target\surefire-reports\TEST-RegressionTest.xml 11:06:17 File found: C:\PATH\TO\target\target\surefire-reports\TEST-RegressionTest2.xml 11:06:17 File found: C:\PATH\TO\target\target\surefire-reports\TEST-TestSuite.xml 11:06:17 Starting to import results from TEST-RegressionTest.xml 11:06:20 Response: (200) {"testExecIssue":{"id":"3330519","key":"KEY-2934","self":"URL"},"testIssues":{"success":[{"id":"3330379","key":"KEY-2930","self":"URL"}]},"infoMessages":[]} 11:06:20 Successfully imported JUnit XML multipart results from TEST-RegressionTest.xml 11:06:20 Starting to import results from TEST-RegressionTest2.xml 11:06:20 Response: (200) {"testExecIssue":{"id":"3330519","key":"KEY-2934","self":"URL"},"testIssues":{"error":[{"messages":["You must specify a summary of the issue.","Component/s is required."],"testDefinition":"RegressionTest2."}]},"infoMessages":[]} 11:06:20 Successfully imported JUnit XML results from TEST-RegressionTest2.xml 11:06:20 Starting to import results from TEST-TestSuite.xml 11:06:20 Response: (200) {"testExecIssue":{"id":"3330519","key":"KEY-2934","self":"URL"},"infoMessages":[]} 11:06:20 Successfully imported JUnit XML results from TEST-TestSuite.xml 11:06:20 XRAY_ISSUES_MODIFIED: KEY-2934;KEY-2930 11:06:20 XRAY_TEST_EXECS: KEY-2934 11:06:20 XRAY_IS_REQUEST_SUCCESSFUL: true 11:06:20 XRAY_RAW_RESPONSE: {"testExecIssue":{"id":"3330519","key":"KEY-2934","self":"URL"},"testIssues":{"success":[{"id":"3330379","key":"KEY-2930","self":"URL"}]},"infoMessages":[]};{"testExecIssue":{"id":"3330519","key":"KEY-2934","self":"URL"},"testIssues":{"error":[{"messages":["You must specify a summary of the issue.","Component/s is required."],"testDefinition":"RegressionTest2."}]},"infoMessages":[]};{"testExecIssue":{"id":"3330519","key":"KEY-2934","self":"URL"},"infoMessages":[]} 11:06:20 XRAY_TESTS: KEY-2930 ```

Anything else?

We have to mandatory fields in our Jira Server Instance: components and summary. These fields are not defined in the Junit-XML results. The json results are read in for each file found if we set importToSameExecution: 'false'.