Closed simonNeir closed 3 months ago
Hello, thank you for reporting this and providing detailed information.
I have never experienced something like this. I just tried with this test plan (based on the one you provided) and could not reproduce the issue you describe:
@Test
public void testPerformance() throws IOException {
String itemApiRootUrl = "http://myserviceurl.com";
HashMap<Integer, String> itemIdMap = new HashMap<>();
for (int i = 0 ; i < 500; i ++) {
itemIdMap.put(i, "val" + i);
}
TestPlanStats stats = testPlan(
rpsThreadGroup()
.rampToAndHold(4, Duration.ofSeconds(10), Duration.ofMinutes(1))
.children(
counter("REQUEST_NUMBER").startingValue(1),
httpSampler(s -> {
int requestNum = Integer.parseInt(s.vars.get("REQUEST_NUMBER"));
return itemApiRootUrl+ "?param=" + itemIdMap.get(requestNum);
})
),
resultsTreeVisualizer()
).run();
assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofSeconds(5));
}
I guess that it might be related to itemApiRootUrl
variable, but with the information you provide is not clear how is it calculated or if is it being modified during test plan.
Have you tried using resultsTreeVisualizer() to debug the test plan, or even use IDE debugger putting some condition on breakpoints to detect what may be happening when the url is blank whhen jmeter sampler sample method is executed?
Regards
Greetings
Thank you for your swift reply. I tried running the testplan with the resultsTreeVisualizer. Here are the results for both the successful and failed requests:
Sampler result succes:
Sampler result failure:
Request body success:
Request body failure:
Request headers success:
Request headers failure:
Response body success:
Response body failure:
Response headers success:
Response headers failure:
My code goes as followed:
@Test
public void testPerformance() throws IOException {
testPlan(
rpsThreadGroup("TESTEN")
.maxThreads(20)
.rampToAndHold(4, Duration.ofSeconds(10), Duration.ofMinutes(1))
.children(
PersonaliaPerformanceTestUtil.deleteBijkomendeVerblijfplaatsTransaction()
),
resultsTreeVisualizer()
).run();
}
public static DslTransactionController deleteBijkomendeVerblijfplaatsTransaction() {
return transaction("Verwijder de verblijfplaats van een persoon",
counter("REQUEST_NUMBER").startingValue(1),
deleteBijkomendeVerblijfplaatsHttpSampler()
);
}
private static DslHttpSampler deleteBijkomendeVerblijfplaatsHttpSampler() {
return maakHttpSampler(
HttpMethode.DELETE,
"DELETE - Verwijder de verblijfplaats van een persoon",
putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier()
);
}
protected static DslHttpSampler maakHttpSampler(
HttpMethode methode,
String samplerNaam,
Function<DslJsr223PreProcessor.PreProcessorVars, String> urlSupplier
) {
return new DslHttpSampler(samplerNaam, urlSupplier)
.method(methode.getMethode())
.header("x-onderwijs-instellingsnr", onderwijsInstellingsnr)
.header("x-correlation-id", UUID.randomUUID().toString())
.header("Authorization", "Bearer " + personaliaAccessToken);
}
private static Function<DslJsr223PreProcessor.PreProcessorVars, String> putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier() {
return preProcessorVars -> {
int requestNum = Integer.parseInt(preProcessorVars.vars.get("REQUEST_NUMBER"));
// "persoonIdVar" is a variable passed when a chain of samplers is executed.
String persoonIdVar = "PERSOONID_" + requestNum;
String persoonId = preProcessorVars.vars.get(persoonIdVar);
// I have seperate testplan that chains a few samplers. In this case the "persoonId" is passed directly to the next sampler.
// Otherwise I get the "persoonId" from a Map.
if(persoonId != null && !persoonId.isEmpty()) {
return personaliaApiRootUrl + "/" + persoonId + "/verblijfplaats";
} else {
return personaliaApiRootUrl + "/" + persoonIdMap.get(getRequestNumModulo(requestNum)) + "/verblijfplaats";
}
};
}
// I re-use the same "persoonIDs" for my requests. Before my testplan runs, I post a start amount of "persoon" to the app.
// I then save all the "persoonIDs" to a Map. When the testplan with the RPSThreadgroup runs, it re-uses these "persoonIDs".
// I calculate which "persoonID" is next using a modulo.
private static int getRequestNumModulo(int requestNum) {
// Starting amount of threads. The amount of "persoon" that is POSTed to the application.
int startAantalThreads = PersonaliaPerformanceTestConstants.getStartAantalThreads();
// Calculate which "persoonID" to use next. (e.g.: requestNumber % startAmountOfPersoon => 180 % 50 => 30)
int requestNumModulo = requestNum % startAantalThreads;
return requestNumModulo != 0 ? requestNumModulo : startAantalThreads;
}
So yes, the URL is always generated dynamically. personaliaApiRootUrl never changes, but the path that follows does based on the "persoonID". I also have other testplans that are set-up in a simular way, but also use request body suppliers. The same issue occurs where a request is made with an empty URL and empty request body.
I hope this helps, thank you!
It seems like when I code it as followed, I get the same error. Note: the error does not always occur.
public void testPerformance() throws IOException {
String rootUrl = PersonaliaPerformanceTestUtil.getPersonaliaApiRootUrl(); // never changes
String personaliaAccessToken = PersonaliaPerformanceTestUtil.getPersonaliaAccessToken(); // JWT token
TestPlanStats stats = testPlan(
rpsThreadGroup()
.rampToAndHold(20, Duration.ofSeconds(2), Duration.ofSeconds(20))
.children(
transaction("Verwijder de verblijfplaats van een persoon",
counter("REQUEST_NUMBER").startingValue(1),
httpSampler(s -> {
int requestNum = Integer.parseInt(s.vars.get("REQUEST_NUMBER"));
return rootUrl + "/" + PersonaliaPerformanceTestUtil.getPersoonIdMap().get(PersonaliaPerformanceTestUtil.getRequestNumModulo(requestNum)) + "/verblijfplaats";
})
.method(HttpMethode.DELETE.getMethode())
.header("x-onderwijs-instellingsnr", "18")
.header("x-correlation-id", UUID.randomUUID().toString())
.header("Authorization", "Bearer " + personaliaAccessToken)
)
)
).run();
assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofSeconds(5));
}
When using a URL supplier, it also fails. Code goes as followed. Note: the error does not always occur.
@Test
public void testPerformance2() throws IOException {
String personaliaAccessToken = PersonaliaPerformanceTestUtil.getPersonaliaAccessToken();
TestPlanStats stats = testPlan(
rpsThreadGroup()
.rampToAndHold(20, Duration.ofSeconds(2), Duration.ofSeconds(20))
.children(
transaction("Verwijder de verblijfplaats van een persoon",
counter("REQUEST_NUMBER").startingValue(1),
httpSampler("SamplerName", PersonaliaPerformanceTestUtil.putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier())
.method(HttpMethode.DELETE.getMethode())
.header("x-onderwijs-instellingsnr", "18")
.header("x-correlation-id", UUID.randomUUID().toString())
.header("Authorization", "Bearer " + personaliaAccessToken)
)
)
).run();
assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofSeconds(5));
}
public static Function<DslJsr223PreProcessor.PreProcessorVars, String> putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier() {
return preProcessorVars -> {
int requestNum = Integer.parseInt(preProcessorVars.vars.get("REQUEST_NUMBER"));
return personaliaApiRootUrl + "/" + PersonaliaPerformanceTestUtil.persoonIdMap.get(getRequestNumModulo(requestNum)) + "/verblijfplaats";
};
}
I'm not really sure what's happening.
My apologies, I forgot to mention the following exception JexlException$Cancel
:
16:09:59.050 [RPS Thread Group-ThreadStarter 1-1] ERROR org.apache.jmeter.functions.Jexl2Function -- An error occurred while evaluating the expression "props.get('lambdaScript4').run(new('us.abstracta.jmeter.javadsl.core.util.PropertyScriptBuilder$PropertyScriptVars',ctx,log))"
org.apache.commons.jexl2.JexlException$Cancel: org.apache.jmeter.functions.Jexl2Function.execute@94![0,128]: 'props.get('lambdaScript4').run(new ('us.abstracta.jmeter.javadsl.core.util.PropertyScriptBuilder$PropertyScriptVars', ctx, log));' execution cancelled at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1309) at org.apache.commons.jexl2.parser.ASTReference.jjtAccept(ASTReference.java:18) at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:946) at org.apache.commons.jexl2.parser.ASTJexlScript.jjtAccept(ASTJexlScript.java:38) at org.apache.commons.jexl2.Interpreter.interpret(Interpreter.java:232) at org.apache.commons.jexl2.ExpressionImpl.execute(ExpressionImpl.java:107) at org.apache.jmeter.functions.Jexl2Function.execute(Jexl2Function.java:95) at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:138) at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:113) at org.apache.jmeter.testelement.property.FunctionProperty.getStringValue(FunctionProperty.java:100) at org.apache.jmeter.testelement.AbstractTestElement.getPropertyAsString(AbstractTestElement.java:280) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.getPath(HTTPSamplerBase.java:513) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.getUrl(HTTPSamplerBase.java:1084) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1301) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1290) at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651) at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570) at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) at java.base/java.lang.Thread.run(Thread.java:842)
Hello, thank you for the very detailed information.
Seems like the root cause of the issue is hinted in that last exception you shared. Sounds like the pre processor being canceled while is executing, and then the url not be properly solved and sampler then failing with mentioned error. The pre processor in general shoudn't be cancelled by the thread group setup. It might be related with the execution framework you are using for the test.
How are you running the tests? Are this JUnit 5 - 4 tests? Are you using maven sure fire to execute? Or intellij or something? Any particular JVM or JVM setting you are using? Maybe something is interrupting the test execution and that is generating the cancelation in the pre processor.
I haven't been able to reproduce the issue, I will try further. If you can come up with a minimum version of a test plan with no direct dependencies to your logic to reproduce the issue, that could help us trace it and solve it would be awesome.
I just could reproduce (some times, due to race conditions) the issue with this testplan and adding a breakpoint for the particular exception:
testPlan(
rpsThreadGroup()
.rampToAndHold(50, Duration.ofSeconds(10), Duration.ofSeconds(10))
.children(
httpSampler(s -> {
try {
Thread.sleep(1000);
return "https://abstracta.us";
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
})
)
).run();
The reason is due to the combination rpsThreadGroup (which uses ConcurrencyThreadGroup + VariableThroughputTime) + lambdas (which uses jexl pre processor).
When threads need to go down, the variableThroughputTimer sends a stop to the ConcurrencyThreadGroup which interrupts the threads and if jexl is in the middle/executing, then it throws the exception and ends up with the behavior you are noticing. It could also interrupt other things and you might get other errors (eg: some IOException, socketclosed or things like that).
We need to look for some way to fix this behavior.
For the time being I recommend you ignore such errors since they happen when threads are going down (so request could have been done or not). You should see at maximum as much errors as the number of threads required to fulfill the RPS.
Evening
Thanks a lot for looking into it! I did indeed notice that sometimes I would get a Socket Closed Exception as well. Most of the time however I got the “IllegalArgumentException: Host may not be blank”.
I’ll ignore the errors for the time being as you suggested.
I have just tried doing some patching on classes that seem to avoid the issue.
You can define this class in your project:
public class NonInterruptingRpsThreadGroup extends RpsThreadGroup {
private static int timerId = 1;
public NonInterruptingRpsThreadGroup(String name) {
super(name);
}
@Override
public HashTree buildTreeUnder(HashTree parent, BuildTreeContext context) {
HashTree ret = parent.add(buildConfiguredTestElement());
HashTree timerParent = counting == EventType.ITERATIONS ? ret.add(buildTestAction()) : ret;
timerParent.add(buildTimer());
children.forEach(c -> context.buildChild(c, ret));
return ret;
}
private TestElement buildTestAction() {
TestAction ret = new TestAction();
ret.setAction(TestAction.PAUSE);
ret.setDuration("0");
configureTestElement(ret, "Flow Control Action", TestActionGui.class);
return ret;
}
private TestElement buildTimer() {
VariableThroughputTimer ret = new NonInterruptingVariableThroughputTimer();
ret.setData(buildTimerSchedulesData());
configureTestElement(ret, buildTimerName(timerId++), VariableThroughputTimerGui.class);
return ret;
}
public static class NonInterruptingVariableThroughputTimer extends VariableThroughputTimer {
@Override
protected void stopTest() {
// This is actually the main change from the original code of rpsThreadGroup.
JMeterContextService.getContext().getThreadGroup().tellThreadsToStop();
}
}
private String buildTimerName(int id) {
return "rpsTimer" + id;
}
private CollectionProperty buildTimerSchedulesData() {
PowerTableModel table = new PowerTableModel(
new String[]{"Start RPS", "End RPS", "Duration, sec"},
new Class[]{String.class, String.class, String.class});
schedules.forEach(s -> table.addRow(s.buildTableRow()));
return JMeterPluginsUtils.tableModelRowsToCollectionProperty(table, "load_profile");
}
}
And use it instead of the default rpsThreadGroup
, like in this scenario:
testPlan(
new NonInterruptingRpsThreadGroup(null)
.rampToAndHold(50, Duration.ofSeconds(10), Duration.ofSeconds(10))
.children(
httpSampler(s -> {
try {
Thread.sleep(1000);
return "https://abstracta.us";
} catch (InterruptedException e) {
throw new RuntimeException(e);
}
})
)
).run();
We are still not sure of all potential implications of such change in the general picture. We will try contacting talk with the ConcurrencyThreadGroup & VariableThroughputTimer maintainers and see how we proceed.
We would really appreciate you and anyone trying it for a while and let us know if you see any unexpected issues with this patched class.
Regards
Hello @simonNeir , have you tried the NonInterruptingRpsThreadGroup
? If so, have you had any issues with it?
Greetings @rabelenda
Since using the NonInterruptingRpsThreadGroup
, I haven't experienced any issues.
Regards
Closing this since it has been fixed in this version
Greetings
I have an issue when using the RpsThreadgroup. Let's say I run a test with 4 requests per second, a hold duration of 1 minute and a ramp up period of 10 seconds. The test would generate around 280 requests (4rps * 70s).
Let's take a DELETE request as example. I'm using a counter element called "REQUEST_NUMBER" that increments per request.
I'm also using a URLSupplier to generate a custom URL for each request, depending on the REQUEST_NUMBER timer variable. I retrieve the corresponding item ID from a Map to generate the correct URL.
The issue is that once the test runtime is over, the final few requests fail with a "Host may not be blank" Exception. Successful XML results:
Failing XML results:
So it seems like once the test runtime ends, one or more requests with an empty URL are still executed. How can I avoid this or wait until all requests properly finish? Thanks!