Incorrect and multiple JMeter dependencies downloaded to `build/jmeter/lib` cause JMeter runs to fail

DKroot commented 1 year ago

Describe the bug The JMeter runs in our project using id 'de.qualersoft.jmeter' version '2.4+'. These runs started to fail suddenly with no build changes with the following error in JMeter logs:

2023-05-19 12:24:49,072 ERROR o.a.j.JMeter: Uncaught exception in thread Thread[Thread Group 1-33,5,main]
java.lang.NoClassDefFoundError: Could not initialize class org.codehaus.groovy.runtime.InvokerHelper
    at groovy.lang.GroovyObjectSupport.getDefaultMetaClass(GroovyObjectSupport.java:46) ~[groovy-3.0.13.jar:3.0.13]
    at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:32) ~[groovy-3.0.13.jar:3.0.13]
    at groovy.lang.Binding.<init>(Binding.java:36) ~[groovy-3.0.13.jar:3.0.13]
    at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl$2.<init>(GroovyScriptEngineImpl.java:222) ~[groovy-jsr223-3.0.13.jar:3.0.13]
    at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:222) ~[groovy-jsr223-3.0.13.jar:3.0.13]
    at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:155) ~[groovy-jsr223-3.0.13.jar:3.0.13]
    at javax.script.AbstractScriptEngine.eval(AbstractScriptEngine.java:231) ~[java.scripting:?]
    at org.apache.jmeter.functions.Groovy.execute(Groovy.java:120) ~[ApacheJMeter_functions-5.5.jar:5.5]
    at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:138) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:113) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.testelement.property.FunctionProperty.getStringValue(FunctionProperty.java:100) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.testelement.AbstractTestElement.getPropertyAsString(AbstractTestElement.java:280) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.IfController.getCondition(IfController.java:170) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.IfController.next(IfController.java:231) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.GenericController.nextIsAController(GenericController.java:222) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.GenericController.next(GenericController.java:175) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.GenericController.nextIsAController(GenericController.java:222) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.GenericController.next(GenericController.java:175) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.GenericController.nextIsAController(GenericController.java:222) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.GenericController.next(GenericController.java:175) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.control.LoopController.next(LoopController.java:134) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.threads.AbstractThreadGroup.next(AbstractThreadGroup.java:99) ~[ApacheJMeter_core-5.5.jar:5.5]
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:303) ~[ApacheJMeter_core-5.5.jar:5.5]
    at java.lang.Thread.run(Thread.java:833) ~[?:?]
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.ExceptionInInitializerError [in thread "Thread Group 1-9"]
    at org.codehaus.groovy.runtime.InvokerHelper.<clinit>(InvokerHelper.java:88) ~[groovy-3.0.13.jar:3.0.13]
    ... 24 more

It happened on a couple of occasions in the last year. Once the runs start to fail, they fail consistently even after total project directory removal and recreation. Troubleshooting of the issue reveals that multiple versions of JMeter dependencies are present in build/jmeter/lib. Notice groovy-3.0.13.jar and groovy-3.0.17.jar. The original JMeter 5.5 distirbution actually has groovy-3.0.11.jar. Hence the dependencies seem to be incorrect: not the ones which come with JMeter.

-bash-4.2$ ls build/jmeter/lib
accessors-smart-2.4.8.jar        commons-io-2.11.0.jar                     groovy-json-3.0.13.jar          jodd-core-5.0.13.jar                   oro-2.0.8.jar
accessors-smart-2.4.9.jar        commons-jexl-2.1.1.jar                    groovy-json-3.0.17.jar          jodd-lagarto-5.0.13.jar                ph-commons-10.1.6.jar
annotations-16.0.2.jar           commons-jexl3-3.2.1.jar                   groovy-jsr223-3.0.13.jar        jodd-log-5.0.13.jar                    ph-css-6.5.0.jar
ApacheJMeter-5.5.jar             commons-lang3-3.12.0.jar                  groovy-jsr223-3.0.17.jar        jodd-props-5.0.13.jar                  plot-api-jvm-3.1.1.jar
apiguardian-api-1.1.2.jar        commons-logging-1.2.jar                   groovy-sql-3.0.13.jar           jorphan-5.5.jar                        plot-base-portable-jvm-2.2.1.jar
asm-9.3.jar                      commons-math3-3.6.1.jar                   groovy-sql-3.0.17.jar           json-path-2.6.0.jar                    plot-builder-portable-jvm-2.2.1.jar
base-portable-jvm-2.2.1.jar      commons-net-3.8.0.jar                     groovy-templates-3.0.13.jar     json-smart-2.4.10.jar                  plot-common-portable-jvm-2.2.1.jar
batik-anim-1.14.jar              commons-pool2-2.11.1.jar                  groovy-templates-3.0.17.jar     json-smart-2.4.8.jar                   plot-config-portable-jvm-2.2.1.jar
batik-awt-util-1.14.jar          commons-text-1.9.jar                      groovy-xml-3.0.13.jar           jsoup-1.15.1.jar                       reactive-streams-1.0.4.jar
batik-bridge-1.14.jar            darklaf-core-2.7.3.jar                    groovy-xml-3.0.17.jar           jtidy-r938.jar                         rhino-1.7.14.jar
batik-codec-1.14.jar             darklaf-extensions-rsyntaxarea-0.3.4.jar  hamcrest-2.2.jar                junit                                  rsyntaxtextarea-3.2.0.jar
batik-constants-1.14.jar         darklaf-macos-2.7.3.jar                   hamcrest-core-2.2.jar           junit-4.13.2.jar                       Saxon-HE-11.3.jar
batik-css-1.14.jar               darklaf-native-utils-2.7.3.jar            hamcrest-date-2.0.8.jar         kotlin-logging-jvm-2.0.5.jar           serializer-2.7.2.jar
batik-dom-1.14.jar               darklaf-platform-base-2.7.3.jar           httpasyncclient-4.1.5.jar       kotlin-stdlib-1.6.21.jar               slf4j-api-1.7.36.jar
batik-ext-1.14.jar               darklaf-property-loader-2.7.3.jar         httpclient-4.5.13.jar           kotlin-stdlib-common-1.6.21.jar        svgSalamander-1.1.2.4.jar
batik-gvt-1.14.jar               darklaf-theme-2.7.3.jar                   httpclient-4.5.14.jar           kotlin-stdlib-jdk7-1.6.21.jar          swing-extensions-laf-support-0.1.3.jar
batik-i18n-1.14.jar              darklaf-utils-2.7.3.jar                   httpcore-4.4.15.jar             kotlin-stdlib-jdk8-1.6.21.jar          swing-extensions-visual-padding-0.1.3.jar
batik-parser-1.14.jar            darklaf-windows-2.7.3.jar                 httpcore-4.4.16.jar             kotlinx-coroutines-core-jvm-1.5.2.jar  tika-core-1.28.3.jar
batik-script-1.14.jar            dec-0.1.2.jar                             httpcore-nio-4.4.15.jar         kotlinx-coroutines-swing-1.5.2.jar     tika-parsers-1.28.3.jar
batik-shared-resources-1.14.jar  dnsjava-2.1.9.jar                         httpcore-nio-4.4.16.jar         kotlinx-html-jvm-0.7.3.jar             vis-svg-portable-jvm-2.2.1.jar
batik-svg-dom-1.14.jar           error_prone_annotations-2.10.0.jar        httpmime-4.5.13.jar             lets-plot-batik-2.2.1.jar              xalan-2.7.2.jar
batik-svggen-1.14.jar            ext                                       httpmime-4.5.14.jar             lets-plot-common-2.2.1.jar             xercesImpl-2.12.2.jar
batik-transcoder-1.14.jar        freemarker-2.3.31.jar                     jackson-annotations-2.13.4.jar  log4j-1.2-api-2.17.2.jar               xml-apis-1.4.01.jar
batik-util-1.14.jar              freemarker-2.3.32.jar                     jackson-annotations-2.13.5.jar  log4j-api-2.17.2.jar                   xml-apis-ext-1.3.04.jar
batik-xml-1.14.jar               geronimo-jms_1.1_spec-1.1.1.jar           jackson-core-2.13.4.jar         log4j-core-2.17.2.jar                  xmlgraphics-commons-2.7.jar
bsf-2.4.0.jar                    groovy-3.0.13.jar                         jackson-core-2.13.5.jar         log4j-slf4j-impl-2.17.2.jar            xmlpull-1.1.3.1.jar
bsh-2.0b6.jar                    groovy-3.0.17.jar                         jackson-databind-2.13.4.2.jar   mail-1.5.0-b01.jar                     xmlresolver-4.2.0-data.jar
caffeine-2.9.3.jar               groovy-datetime-3.0.13.jar                jackson-databind-2.13.5.jar     miglayout-core-5.3.jar                 xmlresolver-4.2.0.jar
checker-qual-3.19.0.jar          groovy-datetime-3.0.17.jar                javax.activation-1.2.0.jar      miglayout-swing-5.3.jar                xstream-1.4.19.jar
commons-codec-1.15.jar           groovy-dateutil-3.0.13.jar                jcharts-0.7.5.jar               mongo-java-driver-2.11.3.jar
commons-collections-3.2.2.jar    groovy-dateutil-3.0.17.jar                jcl-over-slf4j-1.7.36.jar       mxparser-1.2.2.jar
commons-collections4-4.4.jar     groovy-jmx-3.0.13.jar                     jmespath-core-0.5.1.jar         neo4j-java-driver-4.4.11.jar
commons-dbcp2-2.9.0.jar          groovy-jmx-3.0.17.jar                     jmespath-jackson-0.5.1.jar      neo4j-java-driver-4.4.9.jar

To Reproduce Steps to reproduce the behavior: It's not entirely clear what triggered this, but it might have to do something to do with the global Gradle cache.

Expected behavior build/jmeter/lib has to consist of what comes with a particular version of JMeter. Otherwise, the integrity of JMeter is violated, and its behavior can be unpredictable.

Data/Screenshots Portions of build.gradle:

jmeter {
  // Main property file (`-p` / `--propfile`).
  mainPropertyFile = file('src/test/jmeter/user.properties')

  // JMeter property map passed to JMeter via `-J` options, overriding properties set above
  jmeterProperties = jmProperties

  //region Expand heap
  // The following max heap sizes had to be used: 1G for >= 250 VUsers, 2G for >= 500 VUsers.
  maxHeap = '2g'
  //endregion

  //logOutputFile = // Defaults to <buildDir>/logs/jmeter.log
  //jvmArgs = // Additional JVM arguments that will be passed to the jvm directly.

  tool {
    version = 5.5 // version of the jmeter-runner: defaults to '5.4.1'.
  }
}

def jmeterTestResultsDir = "${project.buildDir}/test-results/jmeter"

tasks.register('jmFailOnErrors') {
  doLast {
    assert file("${jmeterTestResultsDir}/errors.csv").length() == 0: 'Some performance test samples failed'
  }
}

tasks.register('jmRun', JMeterRunTask) {
  description 'Executes the performance tests'
  // TBD Consolidate custom `${project.buildDir}/jmeter-report` with the main test results directory
  // Empty `${project.buildDir}/reports/jmeter` is an unused run artifact
  outputs.dirs jmeterTestResultsDir, "${project.buildDir}/jmeter-report", "${project.buildDir}/reports/jmeter"
  outputs.file "${project.buildDir}/logs/jmeter.log"
  jmxFile = PERF_TEST_SUITE_FILE_NAME
  finalizedBy jmFailOnErrors

  doFirst {
    println "Starting JMeter CLI run on JVM ${System.properties['java.vendor.version']}"
    println "JMeter properties: ${jmProperties.toString()}"
  }
}
// Always clean otherwise it might be skipped as UP-TO-DATE
// This has to be specified separately from the `jmRun` registration because `cleanJmRun` is not yet available there
jmRun.dependsOn cleanJmRun

System (please complete the following information):

OS: Red Hat Enterprise Linux Server
OS-Version: release 7.9 (Maipo)
Gradle-Version: 7.4.2
Plugin-Version: 2.4.0

Additional context

I was able to fix this problem for good by doing two things:

Adding removal of ${project.buildDir}/jmeter before each jmRun
Adding --no-build-cache Gradle switch

Doing just step 1 is not enough.

However, this now means that JMeter gets downloaded on every run.

mathze commented 1 year ago

Hi @DKroot,

I've tried to reproduce the issue with a minimal setup. As far as I can tell, the only groovy-dependency in this is 3.0.11 coming with org.apache.jmeter:ApacheJMeter_core:5.5. So my educated guess is that you use any additional jmeterLibrary or jmeterPlugin dependencies which introduce the newer versions as a transitive dependency.

My suggestion would be to call gradlew dependencies and check which are those libraries and downgrade them to an appropriate version.

If you can not downgrade the main dependency, there would be the possibility to force or exclude the required groovy version (see here) but I wouldn't suggest this as this could lead to other issues.

DKroot commented 1 year ago

I don't have jmeterLibrary nor jmeterPlugin nor explicit org.apache.jmeter dependency. Here are my dependencies:

plugins {
  //region Server-side stack
  id 'java'

  /*
  Adds Spring Boot tasks and configurations depending on other plug-ins.
  The version of this plug-in determines versions of Spring Boot and its BOM (managed dependencies).

  NOTE: temporarily enable `spring-boot-properties-migrator` dependency below after all Sprint Boot upgrades.
  */
  id 'org.springframework.boot' version '2.6.+'

  // Enables using the `spring-boot-dependencies` BOM and Spring Boot Starters inheriting versions from the BOM
  id 'io.spring.dependency-management' version '1.0.+'

  /*
  Adds Lombok dependency to `annotationProcessor` and `compileOnly` configurations of each source set.
  The `javadoc` task will be configured to read the `delombok`-ed sources instead of the actual sources.
  Recommended to use by Lombok.
  */
  id 'io.freefair.lombok' version '6.4.+'
  //endregion

  //region Client-side stack
  // (automatically downloaded) `Node.js` and `npm` support
  id 'com.github.node-gradle.node' version '3.4.+'
  //endregion

  // Basic tasks for building and pushing Docker images: https://github.com/palantir/gradle-docker
  // Complemented with custom tasks using `docker` CLI.
  id 'com.palantir.docker' version '0.32.+'
  // Tasks for starting, stopping, and cleaning up a named container
  id 'com.palantir.docker-run' version '0.32.+'

  // Performance testing using JMeter
  id 'de.qualersoft.jmeter' version '2.4.+'

  // Static code analysis using SonarQube
  id 'org.sonarqube' version '3.3'

  // Prints task metadata and dependency information
  id 'org.barfuin.gradle.taskinfo' version '1.4.+'
}

dependencies {
  //region Spring Boot-managed dependencies
  /*
  Versions are specified by the Sprint Boot BOM, which is added by `io.spring.dependency-management`.
  See the list of all managed dependencies here:
  https://docs.spring.io/spring-boot/docs/current/reference/html/appendix-dependency-versions.html
  */

  annotationProcessor 'org.springframework.boot:spring-boot-configuration-processor'

  // Core starter, including auto-configuration support, logging and YAML
  implementation 'org.springframework.boot:spring-boot-starter'

  // Starter for building RESTful web apps using `JAX-RS` and `Jersey`. An alternative to `spring-boot-starter-web`.
  implementation 'org.springframework.boot:spring-boot-starter-jersey'

  /*
  Starter to provide the web app servlet, serving static content using Spring MVC and Tomcat as the default
  embedded container.
  */
  implementation 'org.springframework.boot:spring-boot-starter-web'
  //implementation 'org.springframework.boot:spring-boot-starter-tomcat'

  // Starter for using Spring Security
  implementation 'org.springframework.boot:spring-boot-starter-security'

  // Starter for using Spring Data JPA with Hibernate
  implementation 'org.springframework.boot:spring-boot-starter-data-jpa'
  implementation 'org.apache.tomcat:tomcat-jdbc'

  //region JDBC drivers
  runtimeOnly 'net.sourceforge.jtds:jtds'

  runtimeOnly 'com.oracle.database.jdbc:ojdbc11' // for Spring Boot 2.5.+
  //  runtimeOnly 'com.oracle.database.jdbc:ojdbc10' // for Spring Boot 2.4.x-
  //endregion

  // Starter for developer testing with JUnit, Hamcrest and Mockito
  // `implementation` configuration also enables application use of `MockHttpSession` and `MockHttpServletRequest`
  implementation 'org.springframework.boot:spring-boot-starter-test'

  implementation 'org.slf4j:slf4j-api'
  implementation 'org.slf4j:log4j-over-slf4j'
  implementation 'org.apache.commons:commons-lang3'

  /*
  Spring Boot Developer Tools will be automatically disabled when running a JAR or if the app is started from a
  special classloader.

  Note: Dev Tools will:
      1. Disable the Spring caching options by default
      2. Enable DEBUG logging for the web logging group
      3. Automatically restart the app whenever files on the classpath change
      4. Embed `LiveReload` server
  */
  developmentOnly 'org.springframework.boot:spring-boot-devtools'

  /*
  Prints diagnostics at startup, but also temporarily migrate properties at runtime for you.
  */
  //  runtimeOnly 'org.springframework.boot:spring-boot-properties-migrator'
  //endregion

  //region Other dependencies: specific (fixed or dynamic) versions. If you omit a version, the latest will be used.

  // Google Guava: a common-purpose library
  implementation 'com.google.guava:guava:29.+'

  // FindBugs: provides null-check annotations
  implementation 'com.google.code.findbugs:jsr305:3.+'

  implementation 'io.jsonwebtoken:jjwt-api:0.+'
  runtimeOnly 'io.jsonwebtoken:jjwt-impl:0.+'
  runtimeOnly 'io.jsonwebtoken:jjwt-jackson:0.+'

  // Support for ANSI color codes on Windows: not working with Logback
  // runtimeOnly 'org.fusesource.jansi:jansi:2.3.+'

  //implementation 'io.springfox:springfox-boot-starter:3.+'
  implementation 'io.springfox:springfox-swagger2:3.+'
  //TODO How do we access the Swagger UI?
  //  implementation 'io.springfox:springfox-swagger-ui:3.+'
  implementation 'io.swagger:swagger-jersey2-jaxrs:1.6.+'

  // Backward compatibility support for JUnit 4 tests
  testImplementation('org.junit.vintage:junit-vintage-engine') {
    exclude group: 'org.hamcrest', module: 'hamcrest-core'
  }

  testImplementation 'com.opengamma.strata:strata-basics:2.+'
  //endregion
}

mathze commented 1 year ago

Hi, with the information you provided, I was able to drill down the issue, at least for the groovy 3.0.17 version, to the io.spring.dependency-management plugin. The plugin seams to override any dependencies it can find. Also those which are managed by jmeter-plugin. As this behaviour is out of my hand, please refer to the dependency-management plugin documentation, if there is a possibility to avoid this. If I remember correctly the plugin offer a way to set a fixed version for a dedicated dependency.

Still unclear to me is where the 3.0.13 came from, but I wasn't able to apply all plugins & dependencies. Maybe it's also a side effect of dependency-plugin in combination with those plugins/dependencies.

If you cannot force groovy version to 3.0.11, an alternative approach would be to isolate the jmeter-stuff in an isolated subproject.

qualersoft / jmeter-gradle-plugin

Incorrect and multiple JMeter dependencies downloaded to `build/jmeter/lib` cause JMeter runs to fail #169