abstracta / jmeter-java-dsl

Simple JMeter performance tests API
https://abstracta.github.io/jmeter-java-dsl/
Apache License 2.0
477 stars 59 forks source link

RpsThreadgroup - IllegalArgumentException (Host may not be blank) #257

Closed simonNeir closed 3 months ago

simonNeir commented 8 months ago

Greetings

I have an issue when using the RpsThreadgroup. Let's say I run a test with 4 requests per second, a hold duration of 1 minute and a ramp up period of 10 seconds. The test would generate around 280 requests (4rps * 70s).

Let's take a DELETE request as example. I'm using a counter element called "REQUEST_NUMBER" that increments per request.

private static DslTransactionController deletePropertyOfItemTransaction() {
    return transaction("Delete property of item",
        counter("REQUEST_NUMBER").startingValue(1),
        deletePropertyOfItemHttpSampler()
    );
}
private static DslHttpSampler deletePropertyOfItemHttpSampler() {
    return createHttpSampler(
        HttpMethod.DELETE,
        "DELETE - Delete the property of item",
        deletePropertyOfItemHttpSamplerURLSupplier()
    );
}
protected static DslHttpSampler createHttpSampler(
    HttpMethod method,
    String samplerName,
    Function<DslJsr223PreProcessor.PreProcessorVars, String> urlSupplier
) {
    return new DslHttpSampler(samplerName, urlSupplier)
        .method(method.getMethod());
}

I'm also using a URLSupplier to generate a custom URL for each request, depending on the REQUEST_NUMBER timer variable. I retrieve the corresponding item ID from a Map to generate the correct URL.

private static Function<DslJsr223PreProcessor.PreProcessorVars, String> deletePropertyOfItemHttpSamplerURLSupplier() {
    return preProcessorVars -> {
        int requestNum = Integer.parseInt(preProcessorVars.vars.get("REQUEST_NUMBER"));
        return itemApiRootUrl+ "/" + itemIdMap.get(requestNum) + "/property";
    };
}

The issue is that once the test runtime is over, the final few requests fail with a "Host may not be blank" Exception. Successful XML results:

<responseData class="java.lang.String">{"item_id":"ITEM-ID"}</responseData>
<java.net.URL>http://localhost:PORT/api/item/ITEM-ID/PROPERTY</java.net.URL>

Failing XML results:

<responseData class="java.lang.String">java.lang.IllegalArgumentException: Host may not be blank&#xd;
    at org.apache.http.util.Args.notBlank(Args.java:74)&#xd;
    at org.apache.http.cookie.CookieOrigin.&lt;init&gt;(CookieOrigin.java:52)&#xd;
    at org.apache.jmeter.protocol.http.control.HC4CookieHandler.getCookiesForUrl(HC4CookieHandler.java:228)&#xd;
    at org.apache.jmeter.protocol.http.control.HC4CookieHandler.getCookieHeaderForURL(HC4CookieHandler.java:179)&#xd;
    at org.apache.jmeter.protocol.http.control.CookieManager.getCookieHeaderForURL(CookieManager.java:357)&#xd;
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.setConnectionCookie(HTTPHC4Impl.java:1381)&#xd;
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.setupRequest(HTTPHC4Impl.java:1309)&#xd;
    at org.apache.jmeter.protocol.http.sampler.HTTPHC4Impl.sample(HTTPHC4Impl.java:626)&#xd;
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerProxy.sample(HTTPSamplerProxy.java:66)&#xd;
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1301)&#xd;
    at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1290)&#xd;
    at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651)&#xd;
    at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570)&#xd;
    at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501)&#xd;
    at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268)&#xd;
    at java.base/java.lang.Thread.run(Thread.java:842)&#xd;
</responseData>
<java.net.URL>http:/</java.net.URL>

So it seems like once the test runtime ends, one or more requests with an empty URL are still executed. How can I avoid this or wait until all requests properly finish? Thanks!

rabelenda commented 8 months ago

Hello, thank you for reporting this and providing detailed information.

I have never experienced something like this. I just tried with this test plan (based on the one you provided) and could not reproduce the issue you describe:

@Test
public void testPerformance() throws IOException {
    String itemApiRootUrl = "http://myserviceurl.com";
    HashMap<Integer, String> itemIdMap = new HashMap<>();
    for (int i  = 0 ; i < 500; i ++) {
      itemIdMap.put(i, "val" + i);
    }
    TestPlanStats stats = testPlan(
        rpsThreadGroup()
            .rampToAndHold(4, Duration.ofSeconds(10), Duration.ofMinutes(1))
                .children(
                    counter("REQUEST_NUMBER").startingValue(1),
                    httpSampler(s -> {
                      int requestNum = Integer.parseInt(s.vars.get("REQUEST_NUMBER"));
                      return itemApiRootUrl+ "?param=" + itemIdMap.get(requestNum);
                    })
                ),
        resultsTreeVisualizer()
    ).run();
    assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofSeconds(5));
  }

I guess that it might be related to itemApiRootUrl variable, but with the information you provide is not clear how is it calculated or if is it being modified during test plan.

Have you tried using resultsTreeVisualizer() to debug the test plan, or even use IDE debugger putting some condition on breakpoints to detect what may be happening when the url is blank whhen jmeter sampler sample method is executed?

Regards

simonNeir commented 8 months ago

Greetings

Thank you for your swift reply. I tried running the testplan with the resultsTreeVisualizer. Here are the results for both the successful and failed requests:

Sampler result succes: Sampler-result-success

Sampler result failure: Sampler-result-failure

Request body success: Request-body-success

Request body failure: Request-body-failure

Request headers success: Request-headers-success

Request headers failure: Request-headers-failure

Response body success: Response-body-success

Response body failure: Response-body-failure

Response headers success: Response-headers-success

Response headers failure: Response-headers-failure

My code goes as followed:

@Test
public void testPerformance() throws IOException {
    testPlan(
        rpsThreadGroup("TESTEN")
            .maxThreads(20)
            .rampToAndHold(4, Duration.ofSeconds(10), Duration.ofMinutes(1))
            .children(
                PersonaliaPerformanceTestUtil.deleteBijkomendeVerblijfplaatsTransaction()
            ),
        resultsTreeVisualizer()
    ).run();
}
public static DslTransactionController deleteBijkomendeVerblijfplaatsTransaction() {
    return transaction("Verwijder de verblijfplaats van een persoon",
        counter("REQUEST_NUMBER").startingValue(1),
        deleteBijkomendeVerblijfplaatsHttpSampler()
    );
}
private static DslHttpSampler deleteBijkomendeVerblijfplaatsHttpSampler() {
    return maakHttpSampler(
        HttpMethode.DELETE,
        "DELETE - Verwijder de verblijfplaats van een persoon",
        putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier()
    );
}
protected static DslHttpSampler maakHttpSampler(
    HttpMethode methode,
    String samplerNaam,
    Function<DslJsr223PreProcessor.PreProcessorVars, String> urlSupplier
) {
    return new DslHttpSampler(samplerNaam, urlSupplier)
        .method(methode.getMethode())
        .header("x-onderwijs-instellingsnr", onderwijsInstellingsnr)
        .header("x-correlation-id", UUID.randomUUID().toString())
        .header("Authorization", "Bearer " + personaliaAccessToken);
}
private static Function<DslJsr223PreProcessor.PreProcessorVars, String> putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier() {
    return preProcessorVars -> {
        int requestNum = Integer.parseInt(preProcessorVars.vars.get("REQUEST_NUMBER"));

        // "persoonIdVar" is a variable passed when a chain of samplers is executed.
        String persoonIdVar = "PERSOONID_" + requestNum;
        String persoonId = preProcessorVars.vars.get(persoonIdVar);

        // I have seperate testplan that chains a few samplers. In this case the "persoonId" is passed directly to the next sampler.
        // Otherwise I get the "persoonId" from a Map.
        if(persoonId != null && !persoonId.isEmpty()) {
            return personaliaApiRootUrl + "/" + persoonId + "/verblijfplaats";
        } else {
            return personaliaApiRootUrl + "/" + persoonIdMap.get(getRequestNumModulo(requestNum)) + "/verblijfplaats";
        }
    };
}
// I re-use the same "persoonIDs" for my requests. Before my testplan runs, I post a start amount of "persoon" to the app. 
// I then save all the "persoonIDs" to a Map. When the testplan with the RPSThreadgroup runs, it re-uses these "persoonIDs".
// I calculate which "persoonID" is next using a modulo.
private static int getRequestNumModulo(int requestNum) {
    // Starting amount of threads. The amount of "persoon" that is POSTed to the application.
    int startAantalThreads = PersonaliaPerformanceTestConstants.getStartAantalThreads();

    // Calculate which "persoonID" to use next. (e.g.: requestNumber % startAmountOfPersoon => 180 % 50 => 30)
    int requestNumModulo = requestNum % startAantalThreads;

    return requestNumModulo != 0 ? requestNumModulo : startAantalThreads;
}

So yes, the URL is always generated dynamically. personaliaApiRootUrl never changes, but the path that follows does based on the "persoonID". I also have other testplans that are set-up in a simular way, but also use request body suppliers. The same issue occurs where a request is made with an empty URL and empty request body.

I hope this helps, thank you!

simonNeir commented 8 months ago

It seems like when I code it as followed, I get the same error. Note: the error does not always occur.

    public void testPerformance() throws IOException {
        String rootUrl = PersonaliaPerformanceTestUtil.getPersonaliaApiRootUrl(); // never changes
        String personaliaAccessToken = PersonaliaPerformanceTestUtil.getPersonaliaAccessToken(); // JWT token

        TestPlanStats stats = testPlan(
            rpsThreadGroup()
                .rampToAndHold(20, Duration.ofSeconds(2), Duration.ofSeconds(20))
                .children(
                    transaction("Verwijder de verblijfplaats van een persoon",
                        counter("REQUEST_NUMBER").startingValue(1),
                        httpSampler(s -> {
                            int requestNum = Integer.parseInt(s.vars.get("REQUEST_NUMBER"));

                            return rootUrl + "/" + PersonaliaPerformanceTestUtil.getPersoonIdMap().get(PersonaliaPerformanceTestUtil.getRequestNumModulo(requestNum)) + "/verblijfplaats";
                        })
                            .method(HttpMethode.DELETE.getMethode())
                            .header("x-onderwijs-instellingsnr", "18")
                            .header("x-correlation-id", UUID.randomUUID().toString())
                            .header("Authorization", "Bearer " + personaliaAccessToken)
                    )
                )
        ).run();
        assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofSeconds(5));
    }

When using a URL supplier, it also fails. Code goes as followed. Note: the error does not always occur.

 @Test
    public void testPerformance2() throws IOException {
        String personaliaAccessToken = PersonaliaPerformanceTestUtil.getPersonaliaAccessToken();

        TestPlanStats stats = testPlan(
            rpsThreadGroup()
                .rampToAndHold(20, Duration.ofSeconds(2), Duration.ofSeconds(20))
                .children(
                    transaction("Verwijder de verblijfplaats van een persoon",
                        counter("REQUEST_NUMBER").startingValue(1),
                        httpSampler("SamplerName", PersonaliaPerformanceTestUtil.putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier())
                            .method(HttpMethode.DELETE.getMethode())
                            .header("x-onderwijs-instellingsnr", "18")
                            .header("x-correlation-id", UUID.randomUUID().toString())
                            .header("Authorization", "Bearer " + personaliaAccessToken)
                    )
                )
        ).run();
        assertThat(stats.overall().sampleTimePercentile99()).isLessThan(Duration.ofSeconds(5));
    }
public static Function<DslJsr223PreProcessor.PreProcessorVars, String> putOfDeleteBijkomendeVerblijfplaatsHttpSamplerURLSupplier() {
        return preProcessorVars -> {
            int requestNum = Integer.parseInt(preProcessorVars.vars.get("REQUEST_NUMBER"));

            return personaliaApiRootUrl + "/" + PersonaliaPerformanceTestUtil.persoonIdMap.get(getRequestNumModulo(requestNum)) + "/verblijfplaats";
        };
    }

I'm not really sure what's happening.

simonNeir commented 8 months ago

My apologies, I forgot to mention the following exception JexlException$Cancel:

16:09:59.050 [RPS Thread Group-ThreadStarter 1-1] ERROR org.apache.jmeter.functions.Jexl2Function -- An error occurred while evaluating the expression "props.get('lambdaScript4').run(new('us.abstracta.jmeter.javadsl.core.util.PropertyScriptBuilder$PropertyScriptVars',ctx,log))"

org.apache.commons.jexl2.JexlException$Cancel: org.apache.jmeter.functions.Jexl2Function.execute@94![0,128]: 'props.get('lambdaScript4').run(new ('us.abstracta.jmeter.javadsl.core.util.PropertyScriptBuilder$PropertyScriptVars', ctx, log));' execution cancelled at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:1309) at org.apache.commons.jexl2.parser.ASTReference.jjtAccept(ASTReference.java:18) at org.apache.commons.jexl2.Interpreter.visit(Interpreter.java:946) at org.apache.commons.jexl2.parser.ASTJexlScript.jjtAccept(ASTJexlScript.java:38) at org.apache.commons.jexl2.Interpreter.interpret(Interpreter.java:232) at org.apache.commons.jexl2.ExpressionImpl.execute(ExpressionImpl.java:107) at org.apache.jmeter.functions.Jexl2Function.execute(Jexl2Function.java:95) at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:138) at org.apache.jmeter.engine.util.CompoundVariable.execute(CompoundVariable.java:113) at org.apache.jmeter.testelement.property.FunctionProperty.getStringValue(FunctionProperty.java:100) at org.apache.jmeter.testelement.AbstractTestElement.getPropertyAsString(AbstractTestElement.java:280) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.getPath(HTTPSamplerBase.java:513) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.getUrl(HTTPSamplerBase.java:1084) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1301) at org.apache.jmeter.protocol.http.sampler.HTTPSamplerBase.sample(HTTPSamplerBase.java:1290) at org.apache.jmeter.threads.JMeterThread.doSampling(JMeterThread.java:651) at org.apache.jmeter.threads.JMeterThread.executeSamplePackage(JMeterThread.java:570) at org.apache.jmeter.threads.JMeterThread.processSampler(JMeterThread.java:501) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:268) at java.base/java.lang.Thread.run(Thread.java:842)

rabelenda commented 8 months ago

Hello, thank you for the very detailed information.

Seems like the root cause of the issue is hinted in that last exception you shared. Sounds like the pre processor being canceled while is executing, and then the url not be properly solved and sampler then failing with mentioned error. The pre processor in general shoudn't be cancelled by the thread group setup. It might be related with the execution framework you are using for the test.

How are you running the tests? Are this JUnit 5 - 4 tests? Are you using maven sure fire to execute? Or intellij or something? Any particular JVM or JVM setting you are using? Maybe something is interrupting the test execution and that is generating the cancelation in the pre processor.

I haven't been able to reproduce the issue, I will try further. If you can come up with a minimum version of a test plan with no direct dependencies to your logic to reproduce the issue, that could help us trace it and solve it would be awesome.

rabelenda commented 8 months ago

I just could reproduce (some times, due to race conditions) the issue with this testplan and adding a breakpoint for the particular exception:

testPlan(
        rpsThreadGroup()
            .rampToAndHold(50, Duration.ofSeconds(10), Duration.ofSeconds(10))
            .children(
                httpSampler(s -> {
                  try {
                    Thread.sleep(1000);
                    return "https://abstracta.us";
                  } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                  }
                })
            )
    ).run();

The reason is due to the combination rpsThreadGroup (which uses ConcurrencyThreadGroup + VariableThroughputTime) + lambdas (which uses jexl pre processor).

When threads need to go down, the variableThroughputTimer sends a stop to the ConcurrencyThreadGroup which interrupts the threads and if jexl is in the middle/executing, then it throws the exception and ends up with the behavior you are noticing. It could also interrupt other things and you might get other errors (eg: some IOException, socketclosed or things like that).

We need to look for some way to fix this behavior.

For the time being I recommend you ignore such errors since they happen when threads are going down (so request could have been done or not). You should see at maximum as much errors as the number of threads required to fulfill the RPS.

simonNeir commented 8 months ago

Evening

Thanks a lot for looking into it! I did indeed notice that sometimes I would get a Socket Closed Exception as well. Most of the time however I got the “IllegalArgumentException: Host may not be blank”.

I’ll ignore the errors for the time being as you suggested.

rabelenda commented 8 months ago

I have just tried doing some patching on classes that seem to avoid the issue.

You can define this class in your project:

public class NonInterruptingRpsThreadGroup extends RpsThreadGroup {

  private static int timerId = 1;

  public NonInterruptingRpsThreadGroup(String name) {
    super(name);
  }

  @Override
  public HashTree buildTreeUnder(HashTree parent, BuildTreeContext context) {
    HashTree ret = parent.add(buildConfiguredTestElement());
    HashTree timerParent = counting == EventType.ITERATIONS ? ret.add(buildTestAction()) : ret;
    timerParent.add(buildTimer());
    children.forEach(c -> context.buildChild(c, ret));
    return ret;
  }

  private TestElement buildTestAction() {
    TestAction ret = new TestAction();
    ret.setAction(TestAction.PAUSE);
    ret.setDuration("0");
    configureTestElement(ret, "Flow Control Action", TestActionGui.class);
    return ret;
  }

  private TestElement buildTimer() {
    VariableThroughputTimer ret = new NonInterruptingVariableThroughputTimer();
    ret.setData(buildTimerSchedulesData());
    configureTestElement(ret, buildTimerName(timerId++), VariableThroughputTimerGui.class);
    return ret;
  }

  public static class NonInterruptingVariableThroughputTimer extends VariableThroughputTimer {

    @Override
    protected void stopTest() {
      // This is actually the main change from the original code of rpsThreadGroup.
      JMeterContextService.getContext().getThreadGroup().tellThreadsToStop();
    }

  }

  private String buildTimerName(int id) {
    return "rpsTimer" + id;
  }

  private CollectionProperty buildTimerSchedulesData() {
    PowerTableModel table = new PowerTableModel(
        new String[]{"Start RPS", "End RPS", "Duration, sec"},
        new Class[]{String.class, String.class, String.class});
    schedules.forEach(s -> table.addRow(s.buildTableRow()));
    return JMeterPluginsUtils.tableModelRowsToCollectionProperty(table, "load_profile");
  }

}

And use it instead of the default rpsThreadGroup, like in this scenario:

testPlan(
        new NonInterruptingRpsThreadGroup(null)
            .rampToAndHold(50, Duration.ofSeconds(10), Duration.ofSeconds(10))
            .children(
                httpSampler(s -> {
                  try {
                    Thread.sleep(1000);
                    return "https://abstracta.us";
                  } catch (InterruptedException e) {
                    throw new RuntimeException(e);
                  }
                })
            )
    ).run();

We are still not sure of all potential implications of such change in the general picture. We will try contacting talk with the ConcurrencyThreadGroup & VariableThroughputTimer maintainers and see how we proceed.

We would really appreciate you and anyone trying it for a while and let us know if you see any unexpected issues with this patched class.

Regards

rabelenda commented 8 months ago

Here is the followup in JMeter Plugins group.

rabelenda commented 7 months ago

Hello @simonNeir , have you tried the NonInterruptingRpsThreadGroup? If so, have you had any issues with it?

simonNeir commented 7 months ago

Greetings @rabelenda

Since using the NonInterruptingRpsThreadGroup, I haven't experienced any issues.

Regards

rabelenda commented 3 months ago

Closing this since it has been fixed in this version