server.rmi.ssl.keystore.file property not working during test run in distributed mode

Expected behavior

When I set the property server.rmi.ssl.keystore.file=/secrets/rmi_keystore.jks I expect the jmeter worker to use that file path to look for the rmi_keystore.jks file instead of the default behavior of looking for it in the jmeter bin folder.

Actual behavior

rmi_keystore.jks is located in /secrets/rmi_keystore.jks
the same rmi_keystore.jks is also loaded into the jmeter master node
the jmeter worker starts up succesfully when server.rmi.ssl.keystore.file=/secrets/rmi_keystore.jks is set
- it did not start up successfully when server.rmi.ssl.keystore.file was not set
while the jmeter worker starts up successfully it runs into an error of not being able to find rmi_keystore.jks file when a test is actually started by the jmeter master node
- the jmeter worker seems to use the default behavior of looking for the rmi_keystore.jks in the jmeter bin directory when the test starts

worker logs:


Executing the jmeter startup command: JVM_ARGS="-Dsun.lang.ClassLoader.allowArraySyntax=true" /jmeter/apache-jmeter-5.6.3/bin/jmeter-server -p /configz/jmeter-user-props.properties -Jserver.rmi.ssl.disable=false -Dserver.rmi.localport=50000 -Dserver_port=1099 -Dserver.rmi.ssl.keystore.file=/secrets/rmi_keystore.jks
WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
WARN StatusConsoleListener The use of package scanning to locate plugins is deprecated and will be removed in a future release
2024-03-12 16:32:52,480 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN
2024-03-12 16:32:52,550 INFO o.a.j.JMeter: Setting System property: server_port=1099
2024-03-12 16:32:52,550 INFO o.a.j.JMeter: Setting JMeter property: server.rmi.ssl.disable=false
2024-03-12 16:32:52,550 INFO o.a.j.JMeter: Setting System property: server.rmi.localport=50000
2024-03-12 16:32:52,550 INFO o.a.j.JMeter: Setting System property: server_port=1099
2024-03-12 16:32:52,550 INFO o.a.j.JMeter: Setting System property: server.rmi.ssl.keystore.file=/secrets/rmi_keystore.jks
2024-03-12 16:32:52,555 INFO o.a.j.JMeter: Copyright (c) 1998-2024 The Apache Software Foundation
2024-03-12 16:32:52,555 INFO o.a.j.JMeter: Version 5.6.3
2024-03-12 16:32:52,555 INFO o.a.j.JMeter: java.version=11.0.22
2024-03-12 16:32:52,555 INFO o.a.j.JMeter: java.vm.name=OpenJDK 64-Bit Server VM
2024-03-12 16:32:52,555 INFO o.a.j.JMeter: os.name=Linux
2024-03-12 16:32:52,555 INFO o.a.j.JMeter: os.arch=amd64
2024-03-12 16:32:52,556 INFO o.a.j.JMeter: os.version=5.15.0-1057-azure
2024-03-12 16:32:52,556 INFO o.a.j.JMeter: file.encoding=UTF-8
2024-03-12 16:32:52,556 INFO o.a.j.JMeter: java.awt.headless=true
2024-03-12 16:32:52,556 INFO o.a.j.JMeter: Max memory     =524288000
2024-03-12 16:32:52,556 INFO o.a.j.JMeter: Available Processors =1
2024-03-12 16:32:52,588 INFO o.a.j.JMeter: Default Locale=English (EN)
2024-03-12 16:32:52,588 INFO o.a.j.JMeter: JMeter  Locale=English (EN)
2024-03-12 16:32:52,588 INFO o.a.j.JMeter: JMeterHome=/jmeter/apache-jmeter-5.6.3
2024-03-12 16:32:52,588 INFO o.a.j.JMeter: user.dir  =/jmeter/apache-jmeter-5.6.3/bin
2024-03-12 16:32:52,589 INFO o.a.j.JMeter: PWD       =/jmeter/apache-jmeter-5.6.3/bin
2024-03-12 16:32:52,589 INFO o.a.j.JMeter: IP: 10.240.0.145 Name: worker1 FullName: worker1
2024-03-12 16:32:52,589 INFO o.a.j.JMeter: user.classpath=/home/jmeter-support-files
2024-03-12 16:32:52,589 INFO o.a.j.JMeter: Adding to classpath and loader: /home/jmeter-support-files
Using local port: 50000
2024-03-12 16:32:52,697 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2024-03-12 16:32:52,697 INFO o.a.j.r.RmiUtils: Local IP address=10.240.0.145
2024-03-12 16:32:52,878 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=worker1/10.240.0.145,localport=50000]]
Created remote object: UnicastServerRef2 [liveRef: [endpoint:[10.240.0.145:50000,SSLRMIServerSocketFactory(host=worker1/10.240.0.145, keyStoreLocation=/secrets/rmi_keystore.jks, type=JKS, trustStoreLocation=/secrets/rmi_keystore.jks, type=JKS, alias=rmi),SSLRMIClientSocketFactory(keyStoreLocation=/secrets/rmi_keystore.jks, type=JKS, trustStoreLocation=/secrets/rmi_keystore.jks, type=JKS, alias=rmi)](local),objID:[430ed322:18e338279da:-7fff, -3945686868058120489]]]
2024-03-12 16:32:53,035 INFO o.a.j.e.RemoteJMeterEngineImpl: Starting backing engine on 1099
2024-03-12 16:32:53,035 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2024-03-12 16:32:53,037 INFO o.a.j.r.RmiUtils: Local IP address=10.240.0.145
2024-03-12 16:32:53,037 INFO o.a.j.e.RemoteJMeterEngineImpl: IP address is a site-local address; this may cause problems with remote access.
    Can be overridden by defining the system property 'java.rmi.server.hostname' - see jmeter-server script file
2024-03-12 16:32:53,037 INFO o.a.j.e.RemoteJMeterEngineImpl: Creating RMI registry (server.rmi.create=true)
2024-03-12 16:32:53,037 INFO o.a.j.r.RmiUtils: System property 'java.rmi.server.hostname' is not defined, using localHost address
2024-03-12 16:32:53,037 INFO o.a.j.r.RmiUtils: Local IP address=10.240.0.145
2024-03-12 16:32:53,046 INFO o.a.j.r.SSLRMIServerSocketFactory: Created SSLSocket: [SSL: ServerSocket[addr=worker1/10.240.0.145,localport=1099]]
2024-03-12 16:32:53,047 INFO o.a.j.e.RemoteJMeterEngineImpl: Bound to RMI registry on port 1099
2024-03-12 16:34:07,556 INFO o.a.j.s.SampleEvent: List of sample_variables: ]
2024-03-12 16:34:07,619 INFO o.a.j.s.BatchSampleSender: Using batching for this run. Thresholds: num=100, time=60000
2024-03-12 16:34:07,619 INFO o.a.j.s.DataStrippingSampleSender: Using DataStrippingSampleSender for this run with stripAlsoOnError: true
2024-03-12 16:34:07,736 INFO o.a.j.e.RemoteJMeterEngineImpl: Creating JMeter engine on host 10.240.0.145 base '.'
2024-03-12 16:34:07,736 INFO o.a.j.e.RemoteJMeterEngineImpl: Remote client host: 10.240.0.123
2024-03-12 16:34:07,742 INFO o.a.j.s.FileServer: Default base='/jmeter/apache-jmeter-5.6.3/bin'
2024-03-12 16:34:07,744 INFO o.a.j.s.FileServer: Set new base='.'
2024-03-12 16:34:07,747 INFO o.a.j.e.StandardJMeterEngine: Applying properties {}
2024-03-12 16:34:07,749 INFO o.a.j.e.RemoteJMeterEngineImpl: Running test
Starting the test on host 10.240.0.145 @ 2024 Mar 12 16:34:07 GMT (1710261247749)
2024-03-12 16:34:07,764 INFO o.a.j.e.StandardJMeterEngine: Running the test!
2024-03-12 16:34:07,764 INFO o.a.j.s.SampleEvent: List of sample_variables: ]
2024-03-12 16:34:08,345 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-casutg-2.10.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,361 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/di-kafkameter-1.2.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,367 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-httpraw-0.1.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,373 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-udp-0.4.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,377 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-tst-2.5.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,381 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-autostop-0.1.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,383 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-prmctl-0.4.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,434 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-functions-2.1.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,440 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/di-extended-csv-2.1.1-sre.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:08,478 INFO o.a.j.r.ClassFinder: Will scan jar /jmeter/apache-jmeter-5.6.3/lib/ext/jmeter-plugins-csl-0.1.jar with filter ExtendsClassFilter [parents=[interface org.apache.jmeter.functions.Function], inner=false, contains=null, notContains=null]. Consider exposing JMeter plugins via META-INF/services, and add JMeter-Skip-Class-Scanning=true manifest attribute so JMeter can skip classfile scanning
2024-03-12 16:34:09,051 ERROR o.a.j.s.RemoteListenerWrapper: testStarted(host) on 10.240.0.145
java.rmi.ConnectIOException: Exception creating connection to: 10.240.0.123; nested exception is: 
    java.io.FileNotFoundException: rmi_keystore.jks (No such file or directory)
    at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:635) ~[?:?]
    at java.rmi/sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:209) ~[?:?]
    at java.rmi/sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:196) ~[?:?]
    at java.rmi/sun.rmi.server.UnicastRef.invoke(UnicastRef.java:132) ~[?:?]
    at java.rmi/java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(RemoteObjectInvocationHandler.java:217) ~[?:?]
    at java.rmi/java.rmi.server.RemoteObjectInvocationHandler.invoke(RemoteObjectInvocationHandler.java:162) ~[?:?]
    at com.sun.proxy.$Proxy19.testStarted(Unknown Source) ~[?:?]
    at org.apache.jmeter.samplers.RemoteListenerWrapper.testStarted(RemoteListenerWrapper.java:78) [ApacheJMeter_core.jar:5.6.3]
    at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:249) [ApacheJMeter_core.jar:5.6.3]
    at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:432) [ApacheJMeter_core.jar:5.6.3]
    at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
    at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
    at java.base/java.lang.Thread.run(Thread.java:829) [?:?]
Caused by: java.io.FileNotFoundException: rmi_keystore.jks (No such file or directory)
    at java.base/java.io.FileInputStream.open0(Native Method) ~[?:?]
    at java.base/java.io.FileInputStream.open(FileInputStream.java:219) ~[?:?]
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157) ~[?:?]
    at java.base/java.io.FileInputStream.<init>(FileInputStream.java:112) ~[?:?]
    at org.apache.jmeter.rmi.SSLRMIClientSocketFactory.loadStore(SSLRMIClientSocketFactory.java:124) ~[ApacheJMeter_core.jar:5.6.3]
    at org.apache.jmeter.rmi.SSLRMIClientSocketFactory.createSocket(SSLRMIClientSocketFactory.java:82) ~[ApacheJMeter_core.jar:5.6.3]
    at java.rmi/sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:617) ~[?:?]
    ... 14 more
2024-03-12 16:34:09,056 ERROR o.a.j.s.RemoteTestListenerWrapper: Exception on testStarted on host 10.240.0.145

Steps to reproduce the problem

worker startup command: JVM_ARGS="-Dsun.lang.ClassLoader.allowArraySyntax=true" jmeter/apache-jmeter-5.6.3/bin/jmeter-server -p /configz/jmeter-user-props.properties -Jserver.rmi.ssl.disable=false -Dserver.rmi.localport=50000 -Dserver_port=1099 -Jserver.rmi.ssl.keystore.file=/secrets/rmi_keystore.jks
start a test with a basic test plan that doesn't use any plugins (ex: JSR223 sampler that run log.info messages)

JMeter Version

5.6.3

Java Version

11.0.22

OS Version

Linux [5.15.0-1057-azure]

apache / jmeter

server.rmi.ssl.keystore.file property not working during test run in distributed mode #6248