johrstrom / jmeter-prometheus-plugin

A Prometheus Listener for Apache JMeter that exposes results in an http API
Apache License 2.0
164 stars 109 forks source link

PrometheusMetricsConfig can't run in distributed mode #93

Open GaspardZeBear opened 3 years ago

GaspardZeBear commented 3 years ago

Hi,

Bug ? : Jmeter 5.2.1, plugin 0.6.0

Issue : When remotly launched, the prometheus plugin 0.6.0 generates an exception when trying to increment in jsr223 script counters defined as metrics . It works when launched locally. The counters defined into a prometheus listener work in both cases..

Illustration with the plugin demo scenario :

https://github.com/johrstrom/jmeter-prometheus-plugin/blob/master/docs/examples/simple_prometheus_example.jmx

Hereunder details

Thanks for help ! Your plugin is great

Local launch

curl -vvv http://local:9270 2>&1 | grep jsr223 | grep total < # HELP jsr223_can_fail_total success ratio of the can_fail_sampler < # TYPE jsr223_can_fail_total counter < jsr223_can_fail_total 16.0 < # HELP jsr223_assertions_total default help string < # TYPE jsr223_assertions_total counter < jsr223_assertions_total{label="less than 2kB",} 16.0 < # HELP jsr223_count_total the total number of samplers < # TYPE jsr223_count_total counter < jsr223_count_total{label="can_fail_sampler",} 16.0 < # HELP jsr223_animals_total default help string < # TYPE jsr223_animalstotal counter **< jsr223_animals_total{color="G",size="S",mammal="N",} 1.0 < jsr223_animals_total{color="R",size="L",mammal="N",} 1.0 < jsr223_animals_total{color="R",size="S",mammal="Y",} 2.0 < jsr223_animals_total{color="B",size="M",mammal="Y",} 1.0 < jsr223_animalstotal{color="R",size="M",mammal="Y",} 1.0** < # HELP jsr223_success_total the total number of successful samplers < # TYPE jsr223_success_total counter < jsr223_success_total{label="can_fail_sampler",} 10.0

Remote launch : jsr223_animals_total not present

curl -vvv http://remote: 9270 2>&1 | grep jsr223 | grep total < # HELP jsr223_can_fail_total success ratio of the can_fail_sampler < # TYPE jsr223_can_fail_total counter < jsr223_can_fail_total 32.0 < # HELP jsr223_count_total the total number of samplers < # TYPE jsr223_count_total counter < jsr223_count_total{label="can_fail_sampler",} 32.0 < # HELP jsr223_success_total the total number of successful samplers

Traces

2020-10-21 08:50:20,174 WARN c.g.j.l.PrometheusListener: Prometheus server has not yet been initialized, doing it now 2020-10-21 08:50:20,175 ERROR c.g.j.c.CollectorElement: Didn't create new collector because of error, java.lang.NullPointerException: null at com.github.johrstrom.collector.CollectorElement.makeNewCollectors(CollectorElement.java:70) [prometheus-plugin-0.6.0.jar:?] at com.github.johrstrom.config.PrometheusMetricsConfig.testStarted(PrometheusMetricsConfig.java:44) [prometheus-plugin-0.6.0.jar:?] at com.github.johrstrom.config.PrometheusMetricsConfig.testStarted(PrometheusMetricsConfig.java:59) [prometheus-plugin-0.6.0.jar:?] at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:208) [ApacheJMeter_core.jar:5.2.1] at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:381) [ApacheJMeter_core.jar:5.2.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262] ...... 2020-10-21 08:50:23,367 ERROR o.a.j.p.j.s.JSR223Sampler: Problem in JSR223 script JSR223 Sampler, message: javax.script.ScriptException: java.lang.NullPointerException: Cannot invoke method labels() on null object javax.script.ScriptException: java.lang.NullPointerException: Cannot invoke method labels() on null object at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:324) ~[groovy-all-2.4.16.jar:2.4.16] at org.codehaus.groovy.jsr223.GroovyCompiledScript.eval(GroovyCompiledScript.java:72) ~[groovy-all-2.4.16.jar:2.4.16] at javax.script.CompiledScript.eval(CompiledScript.java:92) ~[?:1.8.0_262] at org.apache.jmeter.util.JSR223TestElement.processFileOrScript(JSR223TestElement.java:223) ~[ApacheJMeter_core.jar:5.2.1] at org.apache.jmeter.protocol.java.sampler.JSR223Sampler.sample(JSR223Sampler.java:71) [ApacheJMeter_java.jar:5.2.1] at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:627) [ApacheJMeter_core.jar:5.2.1] at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:551) [ApacheJMeter_core.jar:5.2.1] at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:490) [ApacheJMeter_core.jar:5.2.1] at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:257) [ApacheJMeter_core.jar:5.2.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262] Caused by: java.lang.NullPointerException: Cannot invoke method labels() on null object at org.codehaus.groovy.runtime.NullObject.invokeMethod(NullObject.java:91) ~[groovy-all-2.4.16.jar:2.4.16] at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:47) ~[groovy-all-2.4.16.jar:2.4.16] at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-all-2.4.16.jar:2.4.16] at org.codehaus.groovy.runtime.callsite.NullCallSite.call(NullCallSite.java:34) ~[groovy-all-2.4.16.jar:2.4.16] at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:47) ~[groovy-all-2.4.16.jar:2.4.16] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116) ~[groovy-all-2.4.16.jar:2.4.16] at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:144) ~[groovy-all-2.4.16.jar:2.4.16] at Script41.run(Script41.groovy:10) ~[?:?] at org.codehaus.groovy.jsr223.GroovyScriptEngineImpl.eval(GroovyScriptEngineImpl.java:321) ~[groovy-all-2.4.16.jar:2.4.16]

johrstrom commented 3 years ago

Thanks for the ticket! It's strange that this would happen to you. Looking back through the code, if this were an issue it would mean nobody could run distributed tests, which doesn't seem to be the case.

Can you comment out the skip tg and see if it runs in a distributed mode then? I think there could be an issue with the fact that there are multiple listeners and the way JMeter initialized everything on the client side before starting the test on the remove server.

GaspardZeBear commented 3 years ago

Hi .. I have disabled the skip tg and launched in distributed mode, the issue is still present

2020-10-21 17:12:11,339 ERROR c.g.j.c.CollectorElement: Didn't create new collector because of error, java.lang.NullPointerException: null at com.github.johrstrom.collector.CollectorElement.makeNewCollectors(CollectorElement.java:70) [prometheus-plugin-0.6.0.jar:?] at com.github.johrstrom.config.PrometheusMetricsConfig.testStarted(PrometheusMetricsConfig.java:44) [prometheus-plugin-0.6.0.jar:?] at com.github.johrstrom.config.PrometheusMetricsConfig.testStarted(PrometheusMetricsConfig.java:59) [prometheus-plugin-0.6.0.jar:?] at org.apache.jmeter.engine.StandardJMeterEngine.notifyTestListenersOfStart(StandardJMeterEngine.java:208) [ApacheJMeter_core.jar:5.2.1] at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:381) [ApacheJMeter_core.jar:5.2.1] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_262] 2020-10-21 17:12:11,781 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 1 : listener tg 2020-10-21 17:12:11,781 INFO o.a.j.e.StandardJMeterEngine: Starting 1 threads for group listener tg. 2020-10-21 17:12:11,781 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error 2020-10-21 17:12:11,781 INFO o.a.j.t.ThreadGroup: Starting thread group... number=1 threads=1 ramp-up=1 delayedStart=false 2020-10-21 17:12:11,781 INFO o.a.j.t.ThreadGroup: Started thread group number 1 2020-10-21 17:12:11,781 INFO o.a.j.e.StandardJMeterEngine: Starting ThreadGroup: 2 : config tg 2020-10-21 17:12:11,781 INFO o.a.j.e.StandardJMeterEngine: Starting 1 threads for group config tg. 2020-10-21 17:12:11,781 INFO o.a.j.e.StandardJMeterEngine: Thread will continue on error 2020-10-21 17:12:11,781 INFO o.a.j.t.ThreadGroup: Starting thread group... number=2 threads=1 ramp-up=1 delayedStart=false 2020-10-21 17:12:11,782 INFO o.a.j.t.JMeterThread: Thread started: 10.0.95.21-listener tg 1-1 2020-10-21 17:12:11,783 INFO o.a.j.t.ThreadGroup: Started thread group number 2 2020-10-21 17:12:11,783 INFO o.a.j.e.StandardJMeterEngine: All thread groups have been started 2020-10-21 17:12:11,783 INFO o.a.j.t.JMeterThread: Thread started: 10.0.95.21-config tg 2-1 2020-10-21 17:12:11,814 INFO o.a.j.p.j.s.J.can_fail_sampler: sleeping for 102 ms 2020-10-21 17:12:11,939 INFO o.a.j.p.j.s.J.can_fail_sampler: sleeping for 1371 ms 2020-10-21 17:12:13,311 INFO o.a.j.p.j.s.J.can_fail_sampler: sleeping for 100 ms 2020-10-21 17:12:13,412 INFO o.a.j.p.j.s.J.can_fail_sampler: sleeping for 207 ms 2020-10-21 17:12:13,620 INFO o.a.j.p.j.s.J.can_fail_sampler: sleeping for 1283 ms 2020-10-21 17:12:14,797 ERROR o.a.j.p.j.s.JSR223Sampler: Problem in JSR223 script JSR223 Sampler, message: javax.script.ScriptException: java.lang.NullPointerException: Cannot invoke method labels() on null object javax.script.ScriptException: java.lang.NullPointerException: Cannot invoke method labels() on null object

johrstrom commented 3 years ago

OK, this could be due to me setting config elements as the running version, an API I don't fully understand and probably just blindly copied from another config element as an example.

https://github.com/johrstrom/jmeter-prometheus-plugin/blob/cbd51a1569565c82d9e814f5b820a82fb3e13783/src/main/java/com/github/johrstrom/config/PrometheusMetricsConfig.java#L43

You can probably comment PrometheusMetricsConfig and it should run. Meaning you've discovered some bug in Prometheus Metrics config elements and how they can't run in distributed mode. Apparently I invented it for my use case which was functional testing and didn't need to run in distributed mode and nobody else uses this :disappointed: .

In any case, this does seem to be a bug with the config element. You seem to be just testing the workflow out, so commenting that should element should allow you to move forward, though it removes that functionality should you need it.

GaspardZeBear commented 3 years ago

Ok thanks. I'll try to have a more detailed look at this issue ... but I need some time to enter the internals of jmeter. Regards