search

authorjapps / zerocode

A community-developed, free, opensource, automated testing framework for microservices APIs, Kafka(Data Streams) and Load testing. Zerocode Open Source enables you to create, change and maintain your automated test scenarios via simple JSON or YAML files. Visit documentation below:
https://zerocode-tdd.tddfy.com
Apache License 2.0
905 stars 399 forks source link

No CSV report files being generated (java 17 and Gradle 7.4.2) #473

Open nickcotter opened 3 years ago

nickcotter commented 3 years ago

I'm using gradle and the ZeroCodeMultiLoadRunner to run some load tests and the tests execute correctly but no csv files are being generated in the target folder, only target/logs/zerocode_rest_bdd_logs.log and the regular junit output in build/test-results. What am I missing?

This is part of my gradle file - I'm running the tests in a separate source config:

sourceSets {
    loadTest {
        java {
            compileClasspath += main.output + test.output
            runtimeClasspath += main.output + test.output
            srcDirs = ["src/loadTest/java"]
        }
        resources {
            srcDirs = ["src/loadTest/resources"]
        }
    }
}

configurations {
    loadTestImplementation.extendsFrom testImplementation
}

dependencies {
    ... various deps ...
    loadTestImplementation "org.jsmart:zerocode-tdd:1.3.+"
}

task loadTest(type: Test, group: "verification") {
    include '**/*LoadTest.class'
    testClassesDirs = sourceSets.loadTest.output.classesDirs
    classpath = sourceSets.loadTest.runtimeClasspath
    outputs.upToDateWhen { false }

    systemProperty 'zerocode.junit', 'gen-smart-charts-csv-reports'
}

Here is the load test class:

@LoadWith("load-config.properties")
@TestMappings({
        @TestMapping(testClass = CounterRecorderTest.class, testMethod = "whenRecordIsCalledThenCounterIsIncremented"),
        @TestMapping(testClass = DistributionSummaryRecorderTest.class, testMethod = "whenRecordIsCalledThenTimingsAreRecorded"),
        @TestMapping(testClass = TimerRecorderTest.class, testMethod = "whenRecordIsCalledThenTimingsAreRecorded")})
@RunWith(ZeroCodeMultiLoadRunner.class)
public class RecorderLoadTest {
}
nirmalchandra commented 3 years ago

@nickcotter , Can you check the Gradle settings documentation here!

Looks like Gradle needs this settings for CSV to be generated at the target folder. e.g.

test {
    systemProperty 'zerocode.junit', 'gen-smart-charts-csv-reports'
    include 'org/jsmart/zerocode/testhelp/tests/HelloWorldGitHubSuite.class'
}
nickcotter commented 3 years ago

@nickcotter , Can you check the Gradle settings documentation here!

Looks like Gradle needs this settings for CSV to be generated at the target folder. e.g.

test {
    systemProperty 'zerocode.junit', 'gen-smart-charts-csv-reports'
    include 'org/jsmart/zerocode/testhelp/tests/HelloWorldGitHubSuite.class'
}

I already had the systemProperty present, and the include line includes what appears to be a specific test for the demo code.

Anyway, adding it has no effect - still no CSV file.

authorjapps commented 3 years ago

systemProperty 'zerocode.junit', 'gen-smart-charts-csv-reports'

Hi @nickcotter , That should do the job for Gradle as Gradle by default is not notified by JUnit listeners.

Here is a repo for you to refer to, it works. You can clone and try at your local machine, with and without the above system property set. GitHub repo link: https://github.com/authorjapps/zerocode-hello-world/blob/master/build.gradle

I suggest doing a clean and then, rerun with the above settings.

include 'org/jsmart/zerocode/testhelp/tests/HelloWorldGitHubSuite.class'

You can ignore this line and configure your test suite in similar fashion 👍

nickcotter commented 3 years ago

systemProperty 'zerocode.junit', 'gen-smart-charts-csv-reports'

Hi @nickcotter , That should do the job for Gradle as Gradle by default is not notified by JUnit listeners.

Here is a repo for you to refer to, it works. You can clone and try at your local machine, with and without the above system property set. GitHub repo link: https://github.com/authorjapps/zerocode-hello-world/blob/master/build.gradle

I suggest doing a clean and then, rerun with the above settings.

include 'org/jsmart/zerocode/testhelp/tests/HelloWorldGitHubSuite.class'

You can ignore this line and configure your test suite in similar fashion 👍

I've looked at this project to see what's different, and unfortunately, I can't see anything obvious. My project is a submodule of a larger one and I guess there may be some hidden dependency issue, but I don't know how to diagnose that if so. I can see in the logs that it thinks it has written the CSV file:

2021-01-22 11:45:11,388 [Test worker] WARN  org.jsmart.zerocode.core.runner.parallel.LoadProcessor - 
-----------------------------------------------------------------------------------------------------------
==>> Completed this load-run!
==>> Number of load tests ran : 160
==>> View the detailed performance results in the 'target/zerocode-junit-granular-report.csv' folder.
-----------------------------------------------------------------------------------------------------------

Unfortunately when I try to build the demo I get an error about not being able to find @Scenario:

Crowley:zerocode-hello-world nick$ ./gradlew clean build
Parallel execution is an incubating feature.

> Task :compileJava
/private/tmp/zerocode-hello-world/src/main/java/org/jsmart/zerocode/testhelp/tests/uuid/UUIDTest.java:4: error: cannot find symbol
import org.jsmart.zerocode.core.domain.Scenario;
                                      ^
  symbol:   class Scenario
  location: package org.jsmart.zerocode.core.domain
/private/tmp/zerocode-hello-world/src/main/java/org/jsmart/zerocode/testhelp/tests/randomnumber/RandomNumberTest.java:4: error: cannot find symbol
import org.jsmart.zerocode.core.domain.Scenario;
                                      ^
  symbol:   class Scenario
  location: package org.jsmart.zerocode.core.domain
/private/tmp/zerocode-hello-world/src/main/java/org/jsmart/zerocode/testhelp/tests/uuid/UUIDTest.java:14: error: cannot find symbol
    @Scenario("helloworld_uuid/hello_world_uuid.json")
     ^
  symbol:   class Scenario
  location: class UUIDTest
/private/tmp/zerocode-hello-world/src/main/java/org/jsmart/zerocode/testhelp/tests/randomnumber/RandomNumberTest.java:14: error: cannot find symbol
    @Scenario("helloworld_random_number/hello_world_random_numbers.json")
     ^
  symbol:   class Scenario
  location: class RandomNumberTest
Note: /private/tmp/zerocode-hello-world/src/main/java/org/jsmart/zerocode/zerocodejavaexec/httpclient/CustomHttpClient.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
4 errors

> Task :compileJava FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':compileJava'.
> Compilation failed; see the compiler error output for details.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 4s
2 actionable tasks: 1 executed, 1 up-to-date
authorjapps commented 3 years ago

Guys, Can you have a look as you have used this with Gradle? @ankitshubham97, @Umang, @BeTheCodeWithYou Appreciate your input here.

BeTheCodeWithYou commented 3 years ago

@nickcotter - could you please try setting systemProperty for smart chart reports out of your load test task? Like as below

test { systemProperty 'zerocode.junit', 'gen-smart-charts-csv-reports' }

nickcotter commented 3 years ago

Unfortunately I already had this setting.

On Wed, 3 Feb 2021 at 08:09, {Neeraj : Sidhaye} notifications@github.com wrote:

@nickcotter https://github.com/nickcotter - could you please try setting systemProperty for smart chart reports out of your load test task? Like as below

test { systemProperty 'zerocode.junit', 'gen-smart-charts-csv-reports' }

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/authorjapps/zerocode/issues/473#issuecomment-772318627, or unsubscribe https://github.com/notifications/unsubscribe-auth/AATKEGGYEI6ILB6B2PICQ73S5EAEBANCNFSM4WOQI3EA .

BeTheCodeWithYou commented 3 years ago

@nickcotter ok. Thanks. Could you please share sample structure of your projects, submodules, and where you ZeroCode test project sits with them etc. It will help us to provide some pointers.

arran-auctane commented 2 years ago

Think I've found I'm having a similar issue with multi-module gradle setups.

The CSV files appear to be wrote to a target directory within the submodule directory, but it's then attempting to read the CSV file from a target directory in the project root.

So wrote to .../project/submodule/target/zerocode-test-reports But fails trying to read back from .../project/target/zerocode-test-reports

Exception while deserializing to ZeroCodeReport. Details: java.io.FileNotFoundException: /Users/[USER]/projects/[PROJECT]/target/zerocode-test-reports/class com.....test.example.ExampleTest#deleteExampleWithExistingIdShouldSuccessfullyReturnc0fe7cc5-432d-4e10-9533-a18767f34bfc.json (No such file or directory)
java.lang.RuntimeException: Exception while deserializing to ZeroCodeReport. Details: java.io.FileNotFoundException: /Users/[USER]/projects/[PROJECT]/target/zerocode-test-reports/class com.....test.example.ExampleTest#deleteExampleWithExistingIdShouldSuccessfullyReturnc0fe7cc5-432d-4e10-9533-a18767f34bfc.json (No such file or directory)
    at org.jsmart.zerocode.core.report.ZeroCodeReportGeneratorImpl.lambda$readZeroCodeReportsByPath$9(ZeroCodeReportGeneratorImpl.java:364)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
    at java.base/java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1625)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
    at org.jsmart.zerocode.core.report.ZeroCodeReportGeneratorImpl.readZeroCodeReportsByPath(ZeroCodeReportGeneratorImpl.java:368)
    at org.jsmart.zerocode.core.report.ZeroCodeReportGeneratorImpl.generateCsvReport(ZeroCodeReportGeneratorImpl.java:222)
    at org.jsmart.zerocode.jupiter.extension.ParallelLoadExtension.beforeEach(ParallelLoadExtension.java:64)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeEachCallbacks$2(TestMethodTestDescriptor.java:163)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.lambda$invokeBeforeMethodsOrCallbacksUntilExceptionOccurs$6(TestMethodTestDescriptor.java:199)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeMethodsOrCallbacksUntilExceptionOccurs(TestMethodTestDescriptor.java:199)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.invokeBeforeEachCallbacks(TestMethodTestDescriptor.java:162)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:129)
    at org.junit.jupiter.engine.descriptor.TestMethodTestDescriptor.execute(TestMethodTestDescriptor.java:66)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:151)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.invokeAll(SameThreadHierarchicalTestExecutorService.java:41)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$6(NodeTestTask.java:155)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$8(NodeTestTask.java:141)
    at org.junit.platform.engine.support.hierarchical.Node.around(Node.java:137)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.lambda$executeRecursively$9(NodeTestTask.java:139)
    at org.junit.platform.engine.support.hierarchical.ThrowableCollector.execute(ThrowableCollector.java:73)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.executeRecursively(NodeTestTask.java:138)
    at org.junit.platform.engine.support.hierarchical.NodeTestTask.execute(NodeTestTask.java:95)
    at org.junit.platform.engine.support.hierarchical.SameThreadHierarchicalTestExecutorService.submit(SameThreadHierarchicalTestExecutorService.java:35)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestExecutor.execute(HierarchicalTestExecutor.java:57)
    at org.junit.platform.engine.support.hierarchical.HierarchicalTestEngine.execute(HierarchicalTestEngine.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:107)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:88)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:54)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:67)
    at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:52)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
    at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
    at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
    at org.junit.platform.launcher.core.SessionPerRequestLauncher.execute(SessionPerRequestLauncher.java:53)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.processAllTestClasses(JUnitPlatformTestClassProcessor.java:99)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor$CollectAllTestClassesExecutor.access$000(JUnitPlatformTestClassProcessor.java:79)
    at org.gradle.api.internal.tasks.testing.junitplatform.JUnitPlatformTestClassProcessor.stop(JUnitPlatformTestClassProcessor.java:75)
    at org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.stop(SuiteTestClassProcessor.java:61)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:568)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
    at org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
    at org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
    at org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
    at jdk.proxy1/jdk.proxy1.$Proxy2.stop(Unknown Source)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker$3.run(TestWorker.java:193)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100)
    at org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60)
    at org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:133)
    at org.gradle.process.internal.worker.child.SystemApplicationClassLoaderWorker.call(SystemApplicationClassLoaderWorker.java:71)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.run(GradleWorkerMain.java:69)
    at worker.org.gradle.process.internal.worker.GradleWorkerMain.main(GradleWorkerMain.java:74)

Then had also noticed I can't enable the chartAndDashBoardGeneration option as it's hardcoded to false in the junit5 ParelleLoadExtension...

        //-------------------------------------------
        //       On/Off Extent-Report switch
        //-------------------------------------------
        // Load the key 'chart.dashboard.generation'
        // from 'loadPropertiesFile'
        //-------------------------------------------
        boolean chartAndDashBoardGenerationEnabled = false;

This is with Java 17 and Gradle 7.4.2