search

apache / jmeter

Apache JMeter open-source load testing tool for analyzing and measuring the performance of a variety of services
https://jmeter.apache.org/
Apache License 2.0
8.25k stars 2.09k forks source link

Similarly, scripts written with jsr223 preprocessor TPS are half less than those written with BeanShell preprocessor #5607

Open asfimport opened 2 years ago

asfimport commented 2 years ago

ko (Bug 65754): JMeter version 5.4.1 java version 1.8.0_161 Running environment windows10

Script as like as two peas, script script TPS 300+, BeanShell preprocessor script script, TPS 600+, script content is the same as jsr223 preprocessor.

Created attachment _%5DUF%5B%28MQ_E%404%5DMK%7EZVHR%60OA.png: Red box mark

Red box mark

Severity: normal OS: All

asfimport commented 2 years ago

ko (migrated from Bugzilla): Created attachment 1639976054%281%29.png: Jsr223 preprocessor configuration

Jsr223 preprocessor configuration
asfimport commented 2 years ago

@FSchumacher (migrated from Bugzilla): Yes, that seems to be odd and I will look into it, but the real question for me is, why do you use Bean Shell and not the default, which is Groovy? (hint Groovy is on par performance wise and definitely nicer to deal with that Bean Shell)

asfimport commented 2 years ago

@pmouawad (migrated from Bugzilla): Hello, Can you share the Test element script content ? Thansk

asfimport commented 2 years ago

@FSchumacher (migrated from Bugzilla): In my tests, I used 'return "OK"' as the script content.

The Bean Shell sampler ticked at something about 50.000 req/s and the JSR223 Bean Shell at about 300 req/s.

The problem with JSR223 and Bean Shell seems to be, that the JSR223 implementation for Bean Shell is not getting cached/compiled and the compilation is extremely expensive (the parser seems to be using exceptions to control the flow of the parser and a class loader instance is set up for each run).

You can get Bean Shell sampler almost equally slow, if you check the box to reset Bean Shell on each iteration :)

I am not sure, whether it was faster in earlier versions of JMeter (I somehow doubt it), but I still wonder, why anyone would choose Bean Shell over Groovy.

asfimport commented 2 years ago

@FSchumacher (migrated from Bugzilla): The attached test plan runs four different Hello World samplers against the clock. On my machine the BeanShell sampler is the fastest, followed by JSR 223 Groovy (not that much slower), JavaScript (quite a lot slower) and way off BeanShell (as JSR223).

It seems, that there is a newer version of BeanShell (2.1.0 according to https://github.com/beanshell/beanshell/releases) which implements JSR223 Compilable interface. Sadly, I didn't find the artefacts on any maven repo.

If you wish to test the new release of Beanshell for yourself, you can remove lib/bsh-2.0b6.jar and replace it with the bsh-2.1.0.jar. Then, you have to edit the file META-INF/services/javax.script.ScriptEngineFactory inside the jar to read bsh.BshScriptEngineFactory instead of bsh.engine.BshScriptEngineFactory. (Otherwise an error message is displayed on JMeter startup and no bsh can be found in the JSR223 sampler)

Created attachment bug65754-bsh-jsr-comparison.jmx: Test plan for comparison of BSH and JSR223 samplers

bug65754-bsh-jsr-comparison.jmx ````xml false true false false saveConfig true true true true true true true false true true false false false true false false false true 0 true true true true true true continue false -1 4 1 true 60 true true "OK" groovy continue false -1 4 1 true 60 65 true true "OK" javascript continue false -1 4 1 true 60 130 true true return "OK"; beanshell continue false -1 4 1 true 60 195 true return "OK"; false ````