search

serenity-bdd / serenity-core

Serenity BDD is a test automation library designed to make writing automated acceptance tests easier, and more fun.
http://serenity-bdd.info
Other
720 stars 519 forks source link

Serenity-BDD report publishing within Jenkins (when tests fail) #2878

Closed Mo0rBy closed 1 year ago

Mo0rBy commented 2 years ago

I'm using Serenity-BDD for my test automation framework and I really like it, however I've ran into an issue when publishing the Serenity report onto Jenkins. When a test either errors or fails, when I run mvn serenity:aggregate to generate the report, a file called pagesource<20randomNumbers>.html is generated, and it is the HTML and CSS code of the page that the Selenium webdriver can "see" when the test fails.

Now for some reason, when this file is present in the target/site/serenity directory, I am unable to publish the report on Jenkins using the HTML publisher plugin. When the pagesource<numbers>.html file is present, the HTML publisher plugin is unable to tar the directory and the following error is seen :

FATAL: HTML Publisher failure
java.io.IOException: Failed to extract <JenkinsDirectories>/target/site/serenity/**/*
    at hudson.FilePath.readFromTar(FilePath.java:2982)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2742)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2706)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2694)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2677)
    at htmlpublisher.HtmlPublisher.publishReports(HtmlPublisher.java:242)
    at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:62)
    at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:44)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.util.concurrent.ExecutionException: java.io.IOException: Entry 'pagesource6725183915334433591.html' closed at '0' before the '89846' bytes specified in the header were written
        at hudson.remoting.Channel$2.adapt(Channel.java:1036)
        at hudson.remoting.Channel$2.adapt(Channel.java:1030)
        at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
        at hudson.FilePath.copyRecursiveTo(FilePath.java:2745)
        ... 12 more
    Caused by: java.io.IOException: Entry 'pagesource6725183915334433591.html' closed at '0' before the '89846' bytes specified in the header were written
        at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.closeArchiveEntry(TarArchiveOutputStream.java:429)
        at hudson.util.io.TarArchiver.visit(TarArchiver.java:117)
        at hudson.util.DirScanner.scanSingle(DirScanner.java:51)
        at hudson.util.DirScanner$Glob.scan(DirScanner.java:146)
        at hudson.FilePath.writeToTar(FilePath.java:2943)
        at hudson.FilePath$CopyRecursiveRemoteToLocal.invoke(FilePath.java:2908)
        at hudson.FilePath$CopyRecursiveRemoteToLocal.invoke(FilePath.java:2893)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3492)
        at hudson.remoting.UserRequest.perform(UserRequest.java:211)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:376)
        at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
        at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:121)
        at java.base/java.lang.Thread.run(Unknown Source)
        Suppressed: java.io.IOException: This archive contains unclosed entries.
            at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:291)
            at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:309)
            at hudson.util.io.TarArchiver.close(TarArchiver.java:124)
            at hudson.FilePath.writeToTar(FilePath.java:2942)
            ... 12 more
        Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 61.0.35.157/61.0.35.157:46554
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
            at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
            at hudson.remoting.Channel$2.adapt(Channel.java:1034)
            at hudson.remoting.Channel$2.adapt(Channel.java:1030)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2745)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2706)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2694)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2677)
            at htmlpublisher.HtmlPublisher.publishReports(HtmlPublisher.java:242)
            at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:62)
            at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:44)
            at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Truncated TAR archive
    at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:743)
    at java.base/java.io.InputStream.read(InputStream.java:205)
    at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1309)
    at org.apache.commons.io.IOUtils.copy(IOUtils.java:978)
    at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282)
    at org.apache.commons.io.IOUtils.copy(IOUtils.java:953)
    at hudson.util.IOUtils.copy(IOUtils.java:52)
    at hudson.FilePath.readFromTar(FilePath.java:2972)
    ... 13 more

The primary error within this block is: Suppressed: java.util.concurrent.ExecutionException: java.io.IOException: Entry 'pagesource6725183915334433591.html' closed at '0' before the '89846' bytes specified in the header were written

I've tried to delete this pagesource file with shell command in Jenkins but I was unable to do so due to permissions, and I'd rather not mess with Jenkins' permissions. I also tried using the includes parameter for the HTML publisher plugin such that it would exclude this pagesource file, like this: includes: '**/!(pagesource)*'. However, this resulted in a different error

ERROR: Directory '<JenkinsDirectories>/target/site/serenity' exists but failed copying to '<JenkinsDirectories>/builds/33/htmlreports/Serenity_20Report'.

Is there anyway to customise Serenity reporting such that this pagesource file is never generated. I can understand that it might be used to help debug the failing test, however, myself and my team don't need this feature and I can't see it being used for any other purpose. I can't see where it is used within the Serenity report itself, so I believe that I don't need it at all.

Here's a screenshot of where button that links to the pagesource file. Again, this button can link to nothing, myself and my team don't need it, but we do need the rest of report to be posted onto our Jenkins instance.

Screenshot 2022-07-31 at 18 11 24
Mo0rBy commented 2 years ago

I've managed to find 1 solution to this problem, it's not quite what I wanted, but it'll until I find or am shown another way.

Serenity takes screenshots during the test and this functionality can be customised in the serenity.properties file. The options available are:

  1. FOR_EACH_ACTION
  2. BEFORE_AND_AFTER_EACH_STEP
  3. AFTER_EACH_STEP
  4. FOR_FAILURES
  5. DISABLED

I simply disabled the screenshot functionality and the pagesource<numbers>.html is no longer generated. More about managing Serenity screenshots here

I believe the pagesource.html file is generated so that a tester can debug why the test is failing or erroring out. Disabling screenshots means that the following page isn't available in the Serenity report, meaning that there is no need to generate the pagesource.html file (the HTML source button links to the pagesource.html file).

Screenshot 2022-07-31 at 20 39 16

If anyone finds a better solution to this problem that doesn't involve disabling screenshots, please post it, as I'd prefer to have screenshots enabled so that it is easier for our BA's to see the user journey.

wakaleo commented 2 years ago

Should be fixed in the next release

dragos-panzaru-md commented 1 year ago

@wakaleo I am using the latest release of serenity 3.6.9 and the issue is still not fixed. Any timeline on the fix? The reason I ask this is that initially it was supposed to be fixed in the next release after October 22, which was 3.3.6 and now we are at 3.6.9 and still there is no fix

wakaleo commented 1 year ago

If you have a support contract for Serenity support, let me know so I can take a look. In other cases, as for open source projects in general there is no SLA for bug fixes.

dragos-panzaru-md commented 1 year ago

@wakaleo I understand and agree with your point of view regarding open source projects, but your previous comment was misleading in regard to when we will have the fix for this issue.

sfkb commented 1 year ago

@wakaleo Any update on this. We have a similar but not the same case. For us, it happens regardless if a test case fails or if Screenshots are enabled or disabled. It happens in all cases. Sometimes it works, and the job runs successfully. Like for @dragos-panzaru-md we can not see it's fixed with recent versions.


java.io.IOException: Failed to extract /home/jenkins/agent/workspace/43.serenity_report_fails_jenkins/test/ui/target/site/serenity/**/*
    at hudson.FilePath.readFromTar(FilePath.java:2992)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2750)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2714)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2702)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2685)
    at htmlpublisher.HtmlPublisher.publishReports(HtmlPublisher.java:242)
    at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:62)
    at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:44)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.util.concurrent.ExecutionException: java.io.IOException: This archive contains unclosed entries.
        at hudson.remoting.Channel$2.adapt(Channel.java:1036)
        at hudson.remoting.Channel$2.adapt(Channel.java:1030)
        at hudson.remoting.FutureAdapter.get(FutureAdapter.java:66)
        at hudson.FilePath.copyRecursiveTo(FilePath.java:2753)
        ... 12 more
    Caused by: java.io.IOException: This archive contains unclosed entries.```
suadhika commented 1 year ago

@wakaleo I am also facing the same issue as @sfkb. If the tests are run locally the report is getting generated but in Jenkins with publish html plugin I am getting the exact same error as @sfkb

Mo0rBy commented 1 year ago

I found a solution to this problem!!

So for some reason, the pagesource<LotsOfNumbers>.html files have some different permissions to all the other files generated in the report directories, no idea why this is the case. Anyway, I have just simply done a chmod -R 777 <mySerenityReportDirectory> to give full permissions to the files for all users.

After this I can very easily use a tar shell command in my pipeline and do whatever I want with the report files. In my case, I am using the archiveArtifacts plugin to add the report to the build in a way that allows me to download it.

Some caveats to this solution that people should bear in mind. I am using Jenkins with Kubernetes so I have a container with Maven installed and I am running both my test execution command (mvn clean verify [options]) and the permission change command (chmod -R 777) inside this container. Running the chmod outside the container results in operation not permitted when attempting totar the files, again I think this has something to do with permissions of the generated files.

Here's some of my pipeline code:


            container('maven') {
                stage('Run tests') {
                    FAILED_STAGE = env.STAGE_NAME
                    catchError(buildResult: 'SUCCESS', stageResult: 'UNSTABLE') {

                        currentBuild.displayName = params.BRANCH
                        currentBuild.description = params.TAG_FILTER;

                        sh("""
                            mvn -B clean integration-test \
                                -Denvironment=${params.ENVIRONMENT.toLowerCase()} \
                                -P ${params.TAG_FILTER.toLowerCase()} \
                                serenity:aggregate
                        """)
                    }
                    sh('chmod -R 777 target/site/serenity')
                }
            }

            stage('Publish test report') {
                FAILED_STAGE = env.STAGE_NAME
                cucumber buildStatus: 'null', fileIncludePattern: '**/cucumber.json', jsonReportDirectory: 'target/cucumber-reports', reportTitle: 'Cucumber Test Report'

                sh('tar -czf SerenityReport.tar.gz target/site/serenity')
                archiveArtifacts(artifacts: "SerenityReport.tar.gz", fingerprint: true)
            }
suadhika commented 1 year ago

@Mo0rBy I have tried the below in my jenkins file:

stages {
    stage('Run Test Cases') {
      steps {
        container('testcontainer') {
          sh """
            pwd
            ls
            echo ${WORKSPACE}
            gradle --info clean test aggregate -Denvironment=${params.APP_ENV} -Dtags='envr:'${params.APP_ENV} -Dcucumber.options='--tags @envr='${params.APP_ENV}
          """
            sh ('chmod -R 777 target/site/serenity')
                }
            }
        }
    }

  post {
    always {
      publishHTML (target : [allowMissing: false,
        alwaysLinkToLastBuild: false,
        keepAll: true,
        reportDir: 'target/site/serenity',
        reportFiles: 'index.html',
        reportName: 'my-test-serenity-bdd-reports',
        reportTitles: 'The DW Report'])
      sendNotification(message: params.APP_ENV, slackChannel: '#my-automation-build') {
        }
      sendNotification(buildResult: currentBuild.currentResult, slackChannel: '#my-automation-build') {
        }
    }
  }

But still I get the below error:

[htmlpublisher] Archiving HTML reports...
[htmlpublisher] Archiving at BUILD level /home/jenkins/agent/workspace/MY-QA/QA_XYZ_TestRun/target/site/serenity to /var/jenkins_home/jobs/xxxx/jobs/yyyy/builds/123/htmlreports/qa-my-test-bdd-reports
FATAL: HTML Publisher failure
java.io.IOException: Failed to extract /home/jenkins/agent/workspace/xxxx/yyyy/target/site/serenity/**/*
    at hudson.FilePath.readFromTar(FilePath.java:2864)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2642)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2606)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2594)
    at hudson.FilePath.copyRecursiveTo(FilePath.java:2577)
    at htmlpublisher.HtmlPublisher.publishReports(HtmlPublisher.java:242)
    at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:62)
    at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:44)
    at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:829)
    Suppressed: java.util.concurrent.ExecutionException: java.io.IOException: This archive contains unclosed entries.
        at hudson.remoting.Channel$2.adapt(Channel.java:1037)
        at hudson.remoting.Channel$2.adapt(Channel.java:1031)
        at hudson.remoting.FutureAdapter.get(FutureAdapter.java:65)
        at hudson.FilePath.copyRecursiveTo(FilePath.java:2645)
        ... 12 more
    Caused by: java.io.IOException: This archive contains unclosed entries.
        at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:291)
        at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:309)
        at hudson.util.io.TarArchiver.close(TarArchiver.java:125)
        at hudson.FilePath.writeToTar(FilePath.java:2824)
        at hudson.FilePath.access$4300(FilePath.java:214)
        at hudson.FilePath$CopyRecursiveRemoteToLocal.invoke(FilePath.java:2787)
        at hudson.FilePath$CopyRecursiveRemoteToLocal.invoke(FilePath.java:2774)
        at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3329)
        at hudson.remoting.UserRequest.perform(UserRequest.java:211)
        at hudson.remoting.UserRequest.perform(UserRequest.java:54)
        at hudson.remoting.Request$2.run(Request.java:369)
        at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:117)
        at java.base/java.lang.Thread.run(Thread.java:834)
        Suppressed: hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from 192.168.218.0/192.168.218.0:53281
            at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1795)
            at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
            at hudson.remoting.Channel$2.adapt(Channel.java:1035)
            at hudson.remoting.Channel$2.adapt(Channel.java:1031)
            at hudson.remoting.FutureAdapter.get(FutureAdapter.java:65)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2645)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2606)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2594)
            at hudson.FilePath.copyRecursiveTo(FilePath.java:2577)
            at htmlpublisher.HtmlPublisher.publishReports(HtmlPublisher.java:242)
            at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:62)
            at htmlpublisher.workflow.PublishHTMLStepExecution.run(PublishHTMLStepExecution.java:44)
            at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
            at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
            at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
            at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
            at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
            at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.IOException: Truncated TAR archive
    at org.apache.commons.compress.archivers.tar.TarArchiveInputStream.read(TarArchiveInputStream.java:743)
    at java.base/java.io.InputStream.read(InputStream.java:205)
    at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1309)
    at org.apache.commons.io.IOUtils.copy(IOUtils.java:978)
    at org.apache.commons.io.IOUtils.copyLarge(IOUtils.java:1282)
    at org.apache.commons.io.IOUtils.copy(IOUtils.java:953)
    at hudson.util.IOUtils.copy(IOUtils.java:53)
    at hudson.FilePath.readFromTar(FilePath.java:2854)
    ... 13 more
Mo0rBy commented 1 year ago

@suadhika I notice you are trying to use the HTML Publisher plugin and I want to warn you that for the report to look as it should when opened through Jenkins, you must disable the content-security-policy which is something that is not advisable as far as I can tell.

I know the plugin uses tar (as you can see from the stacktrace). Can you try using an sh(tar....) command to see if that works correctly? If that works then, my guess is that the tar command being used in the HTML Publisher plugin is being executed with a different configuration and I don't know how you would be able to change that. If the sh(tar.....) does not work, then there must be something incorrect with the permissions of the report files still.

suadhika commented 1 year ago

@Mo0rBy My mistake I had not catch-ed the error from gradle build/test, hence it was not executing and skipping the chmod line statement.

dragos-panzaru-md commented 1 year ago

@suadhika Can you give an example how you caught the error from gradle build/test?

Mo0rBy commented 1 year ago

@dragos-panzaru-md He's used the catchError() function from the Pipeline: Basic Steps plugin: https://www.jenkins.io/doc/pipeline/steps/workflow-basic-steps/#catcherror-catch-error-and-set-build-result-to-failure

What this does is execute the code within the catchError() block and if there is an error, you exit the catchError() block and then continue executing the rest of your pipeline.

Here's a really easy to follow tutorial

gem-garimakhare commented 2 weeks ago

Can someone please share the Jenkinsfile for getting report publishHTML configuration @suadhika @dragos-panzaru-md @wakaleo