trends just show 1 build information

ghost commented 1 year ago

Jenkins and plugins versions report

Environment

```text Jenkins: 2.332.3 OS: Linux - 5.10.162+ Java: 11.0.15 - Eclipse Adoptium (OpenJDK 64-Bit Server VM) --- ace-editor:1.1 analysis-core:1.96 ansible:1.1 ansicolor:1.0.2 antisamy-markup-formatter:155.v795fb_8702324 apache-httpcomponents-client-4-api:4.5.13-138.v4e7d9a_7b_a_e61 authentication-tokens:1.4 basic-branch-build-strategies:1.3.2 bitbucket-oauth:0.12 blueocean:1.26.0 blueocean-autofavorite:1.2.5 blueocean-bitbucket-pipeline:1.26.0 blueocean-commons:1.26.0 blueocean-config:1.26.0 blueocean-core-js:1.26.0 blueocean-dashboard:1.26.0 blueocean-display-url:2.4.1 blueocean-events:1.26.0 blueocean-executor-info:1.26.0 blueocean-git-pipeline:1.26.0 blueocean-github-pipeline:1.26.0 blueocean-i18n:1.26.0 blueocean-jira:1.26.0 blueocean-jwt:1.26.0 blueocean-personalization:1.26.0 blueocean-pipeline-api-impl:1.26.0 blueocean-pipeline-editor:1.26.0 blueocean-pipeline-scm-api:1.26.0 blueocean-rest:1.26.0 blueocean-rest-impl:1.26.0 blueocean-web:1.26.0 bootstrap4-api:4.6.0-5 bootstrap5-api:5.2.0-1 bouncycastle-api:2.26 branch-api:2.1051.v9985666b_f6cc build-timeout:1.24 build-with-parameters:73.v3ed13cb_d6cca caffeine-api:2.9.3-65.v6a_47d0f4d1fe checks-api:1.7.5 cloudbees-bitbucket-branch-source:784.v7fcdc7c670f6 cloudbees-disk-usage-simple:0.10 cloudbees-folder:6.740.ve4f4ffa_dea_54 command-launcher:90.v669d7ccb_7c31 commons-lang3-api:3.12.0-36.vd97de6465d5b_ commons-text-api:1.10.0-27.vb_fa_3896786a_7 config-file-provider:3.10.0 credentials:1087.1089.v2f1b_9a_b_040e4 credentials-binding:523.vd859a_4b_122e6 cucumber-reports:5.7.4 cucumber-testresult-plugin:0.10.1 dashboard-view:2.447.vda_124dd35f11 dependency-check-jenkins-plugin:5.2.0 dependency-track:4.2.0 display-url-api:2.3.6 docker-commons:1.21 docker-workflow:563.vd5d2e5c4007f downstream-build-cache:1.7 durable-task:503.v57154d18d478 echarts-api:5.3.3-1 email-ext:2.92 extended-read-permission:3.2 favorite:2.4.1 font-awesome-api:6.1.2-1 generic-webhook-trigger:1.86.1 git:4.11.5 git-client:3.11.2 git-parameter:0.9.18 git-server:1.11 github:1.34.3.1 github-api:1.303-400.v35c2d8258028 github-branch-source:1677.v731f745ea_0cf google-kubernetes-engine:0.8.7 google-login:1.7 google-metadata-plugin:0.3.1 google-oauth-plugin:1.0.7 gradle:2.2 handlebars:3.0.8 handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953 htmlpublisher:1.31 jackson2-api:2.14.1-313.v504cdd45c18b jakarta-activation-api:2.0.1-2 jakarta-mail-api:2.0.1-2 javadoc:226.v71211feb_e7e9 javax-activation-api:1.2.0-5 javax-mail-api:1.6.2-8 jaxb:2.3.7-1 jdk-tool:63.v62d2fd4b_4793 jenkins-design-language:1.26.0 jira:3.7.1 jjwt-api:0.11.5-77.v646c772fddb_0 jnr-posix-api:3.1.7-3 job-dsl:1.78.3 jquery:1.12.4-1 jquery-detached:1.2.1 jquery3-api:3.6.1-1 jsch:0.1.55.61.va_e9ee26616e7 junit:1143.1145.v81b_b_9579a_019 kubernetes:3724.v0920c1e0ec69 kubernetes-client-api:5.12.2-193.v26a_6078f65a_9 kubernetes-credentials:0.9.0 ldap:2.11 lockable-resources:2.18 login-theme:1.1 mailer:435.438.v5b_81173f5b_a_1 mapdb-api:1.0.9-28.vf251ce40855d mask-passwords:3.3 matrix-auth:3.1.5 matrix-project:772.v494f19991984 maven-plugin:3.20 mercurial:1260.vdfb_723cdcc81 metrics:4.1.6.2 mina-sshd-api-common:2.9.2-50.va_0e1f42659a_a mina-sshd-api-core:2.9.2-50.va_0e1f42659a_a miniorange-saml-sp:1.0.18 momentjs:1.1.1 multiple-scms:0.8 oauth-credentials:0.5 okhttp-api:4.9.3-108.v0feda04578cf pam-auth:1.10 parameterized-scheduler:1.1 pipeline-aggregator-view:1.11 pipeline-build-step:2.18 pipeline-github-lib:38.v445716ea_edda_ pipeline-graph-analysis:195.v5812d95a_a_2f9 pipeline-groovy-lib:612.v84da_9c54906d pipeline-input-step:466.v6d0a_5df34f81 pipeline-milestone-step:101.vd572fef9d926 pipeline-model-api:2.2118.v31fd5b_9944b_5 pipeline-model-declarative-agent:1.1.1 pipeline-model-definition:2.2118.v31fd5b_9944b_5 pipeline-model-extensions:2.2118.v31fd5b_9944b_5 pipeline-multibranch-defaults:2.1 pipeline-rest-api:2.28 pipeline-stage-step:293.v200037eefcd5 pipeline-stage-tags-metadata:2.2118.v31fd5b_9944b_5 pipeline-stage-view:2.28 pipeline-utility-steps:2.14.0 plain-credentials:139.ved2b_9cf7587b plugin-util-api:2.17.0 popper-api:1.16.1-3 popper2-api:2.11.6-1 prometheus:2.0.11 pubsub-light:1.16 resource-disposer:0.20 role-strategy:569.v7476f8e4fe29 saml:2.333.vc81e525974a_c scm-api:631.v9143df5b_e4a_a script-security:1190.v65867a_a_47126 simple-theme-plugin:103.va_161d09c38c7 slack:631.v40deea_40323b snakeyaml-api:1.31-84.ve43da_fb_49d0b sonar:2.14 sse-gateway:1.25 ssh-agent:295.v9ca_a_1c7cc3a_a_ ssh-credentials:277.280.v1e86b_7d0056b_ ssh-slaves:1.821.vd834f8a_c390e sshd:3.242.va_db_9da_b_26a_c3 structs:324.va_f5d6774f3a_d timestamper:1.21 token-macro:321.vd7cc1f2a_52c8 trilead-api:1.67.vc3938a_35172f variant:59.vf075fe829ccb workflow-aggregator:590.v6a_d052e5a_a_b_5 workflow-api:1200.v8005c684b_a_c6 workflow-basic-steps:994.vd57e3ca_46d24 workflow-cps:2729.2732.vda_e3f07b_5a_f8 workflow-cps-global-lib:609.vd95673f149b_b workflow-durable-task-step:1190.vc93d7d457042 workflow-job:1189.va_d37a_e9e4eda_ workflow-multibranch:716.vc692a_e52371b_ workflow-scm-step:400.v6b_89a_1317c9a_ workflow-step-api:639.v6eca_cd8c04a_a_ workflow-support:839.v35e2736cfd5c ws-cleanup:0.43 yet-another-build-visualizer:1.16 ```

What Operating System are you using (both controller, and any agents involved in the problem)?

Jeknins master is running at "container optimized OS" at Google's kubernetes GKE Jenkins executor based at jenkins/inbound-agent:4.13-2-jdk11

Reproduction steps

build with tests and creating cucumber reports several times
open "Trends" in one of reports linked to the Jenkins build

I don't think the reproduction is so easy or nobody recognize that this is broken .... Maybe it is more related to the point that we create 2 reports in each build?

At bit simplified, our Jenkinsfile looks like

pipeline {
  options {
    ansiColor('xterm')
    disableConcurrentBuilds()
    timeout(time: 180, unit: 'MINUTES')
    buildDiscarder(logRotator(artifactDaysToKeepStr: "30", daysToKeepStr: "30"))
    skipDefaultCheckout()
  }
  agent {
    kubernetes {
      defaultContainer "jnlp"
      yamlFile "e2e-test.yml"
    }
  }
  stages {
    stage('test browser1') {
      steps {
        // execute cypress at browser1
      }
      post {
        always {
          cucumber buildStatus: "FAILURE",
            reportTitle: "Browser1 Result - BuildNo: ${BUILD_NUMBER}",
            fileIncludePattern: "**/*.cucumber.json",
            trendsLimit: 30,
            classifications: [
              ["key": "Browser", "value": "browser1"],
            ]
        }
      }
    }
    stage('test browser2') {
      steps {
        // execute cypress at browser2
      }
      post {
        always {
          cucumber buildStatus: "UNSTABLE",
            reportTitle: "Browser2 Result - BuildNo: ${BUILD_NUMBER}",
            fileIncludePattern: "**/*.cucumber.json",
            trendsLimit: 30,
            classifications: [
              ["key": "Browser", "value": "browser2"],
            ]
        }
      }
    }
  }
  post {
    cleanup {
      deleteDir()
    }
  }
}

After execution the job directory looks like

jenkins@jenkins-master-statefulset-0:~/jobs/MyProject$ ls -ltr
-rw-r--r--  1 jenkins 2000 8892 Apr 19 15:32 config.xml
-rw-r--r--  1 jenkins 2000    4 Apr 20 12:23 nextBuildNumber
drwxr-sr-x  2 jenkins 2000 4096 Apr 20 12:45 cucumber-reports_8cf47d89-5162-3e81-9c2d-3482f7875129
drwxr-sr-x  2 jenkins 2000 4096 Apr 20 12:52 cucumber-reports_ff9d066e-0fe2-395a-84b7-57748727782d
drwxrwsr-x 63 jenkins 2000 4096 Apr 20 12:52 builds
jenkins@jenkins-master-statefulset-0:~/jobs/MyProject$ cat cucumber-reports_8cf47d89-5162-3e81-9c2d-3482f7875129/cucumber-trends.json 
{
  "buildNumbers" : [ "908" ],
  "passedFeatures" : [ 25 ],
  "failedFeatures" : [ 0 ],
  "totalFeatures" : [ 25 ],
  "passedScenarios" : [ 201 ],
  "failedScenarios" : [ 0 ],
  "totalScenarios" : [ 201 ],
  "passedSteps" : [ 1143 ],
  "failedSteps" : [ 0 ],
  "skippedSteps" : [ 0 ],
  "pendingSteps" : [ 0 ],
  "undefinedSteps" : [ 0 ],
  "totalSteps" : [ 1143 ],
  "durations" : [ 794817000000 ]
  jenkins@jenkins-master-statefulset-0:~/jobs/MyProject$ cat cucumber-reports_ff9d066e-0fe2-395a-84b7-57748727782d/cucumber-trends.json 
{
  "buildNumbers" : [ "908" ],
  "passedFeatures" : [ 1 ],
  "failedFeatures" : [ 2 ],
  "totalFeatures" : [ 3 ],
  "passedScenarios" : [ 10 ],
  "failedScenarios" : [ 2 ],
  "totalScenarios" : [ 12 ],
  "passedSteps" : [ 38 ],
  "failedSteps" : [ 2 ],
  "skippedSteps" : [ 2 ],
  "pendingSteps" : [ 0 ],
  "undefinedSteps" : [ 0 ],
  "totalSteps" : [ 42 ],
  "durations" : [ 203391000000 ]

And the build directory

jenkins@jenkins-master-statefulset-0:~/jobs/MyProject$ ls -ltr builds/908/
total 776
-rw-r--r-- 1 jenkins 2000   2199 Apr 20 12:23 changelog5159779998907979172.xml
drwxr-sr-x 2 jenkins 2000   4096 Apr 20 12:23 workflow
-rw-r--r-- 1 jenkins 2000      0 Apr 20 12:23 changelog3553345163011733707.xml
drwxr-sr-x 3 jenkins 2000   4096 Apr 20 12:23 libs
-rw-r--r-- 1 jenkins 2000      0 Apr 20 12:25 changelog7151498134792390453.xml
-rw-r--r-- 1 jenkins 2000      0 Apr 20 12:25 changelog9764055075027512866.xml
-rw-r--r-- 1 jenkins 2000    644 Apr 20 12:26 changelog11318944069272154331.xml
-rw-r--r-- 1 jenkins 2000      0 Apr 20 12:26 changelog9076530242835235578.xml
drwxr-sr-x 7 jenkins 2000   4096 Apr 20 12:45 cucumber-html-reports_8cf47d89-5162-3e81-9c2d-3482f7875129
drwxr-sr-x 7 jenkins 2000   4096 Apr 20 12:52 cucumber-html-reports_ff9d066e-0fe2-395a-84b7-57748727782d
-rw-r--r-- 1 jenkins 2000   2190 Apr 20 12:52 log-index
-rw-r--r-- 1 jenkins 2000 577485 Apr 20 12:52 log
-rw-r--r-- 1 jenkins 2000 184004 Apr 20 12:52 build.xml

Expected Results

The results of the still existing reports are shown in the Trends section.

Actual Results

Only the current result is shown in the Trends section. Screenshot from 2023-04-20 15-03-20

Anything else?

No response

damianszczepanik commented 1 year ago

Yes, I guess this is because you generate report twice. Or rather two reports. This is not supported, not tested and I believe second report overwrite first one.

One of the problem is that https://github.com/damianszczepanik/cucumber-reporting/blob/master/src/main/java/net/masterthought/cucumber/ReportBuilder.java#L46 is not configurable.

ghost commented 1 year ago

Both reports are shown correctly in the Jenkins build. The second report does not overwrite the first one, because a suffix is added to the static base directory. The Jenkins build directory looks like

...
drwxr-sr-x 7 jenkins 2000   4096 Apr 25 11:38 cucumber-html-reports_089606fb-f6be-3b3a-ad73-71c560919b13
drwxr-sr-x 7 jenkins 2000   4096 Apr 25 11:31 cucumber-html-reports_e9437810-61f7-368f-85a8-45a20aca3219
...

But meanwhile i understood that the trend feature also would work well if this problem would not occur, because we have to much different types of tests mixed up in one Jenkins job (full / partially / single scenario execution at different, selectable browsers). So a trend over the last X builds would flicker like crazy.

jenkinsci / cucumber-reports-plugin