Open Model Thread Group: support concurrency limiter

[vlsi]

Use case

Currently OMTG might create an unlimited amount of threads threads in case the application processes requests slower than JMeter spawns threads. JMeter might run out of memory since every JMeterThread uses a cloned test plan. We should report the error and fail gracefully rather than crash with OutOfMemory.

Another use case is to test the system with a limited concurrency level. For instance, the system might have a hard limit on the request concurrency (e.g. 10 concurrent requests maximum), and the users might want to test various load levels.

Possible solution

a) Implement a fixed concurrency limit for OMTG b) Implement variable concurrency limit like the current rate(...) in the schedule string: concurrency(10) rate(10/sec) random_arrivals(10 min)

Open questions

• [ ] What should OMTG do in case the concurrency limit is reached? Should it discard a launch immediately? Should it wait for some time for the thread to become available?
• [ ] How do we report that certain launches were delayed due to concurrency limit?

Possible workarounds

No response

JMeter Version

5.5

Java Version

No response

OS Version

No response

[ra0077]

Hi

I think it's a good idea to have concurrency(10) variable in addition to the others. How do you avoid the crash with OutOfMemory if the concurrency is to high?

About report, If the reporting is with backend listener, send a event (I don't knwow if it always possible) If the reporting is with console log, a message If the reporting is with html report, it will be more complex. But maybe add another file in addition of csv file to add metadata. And put these metadata in a section at the top the html report

[vipergreat]

Hi

Can we has a Thread lifetime option like the Standard Jmeter Thread Group. Because if we specific the concurrency limit it not realistic like production issue. Imagin when the production server is slow. Customer wait too long. When they wait too long they will kill the mobile app and re longin again. We cannot limit them. It's not real.

[vlsi]

When they wait too long they will kill the mobile app and re longin again. We cannot limit them. It's not real.

You are right limiting threads is not realistic for systems like google.com, however, if the limit is reasonably high, then it it might be enough to reach the target workload, and at the same time it might be low enough to keep JMeter from failing with OutOfMemoryError.

---

We cannot limit them. It's not real

There are cases when thread limit is expected for a production system. Imagine there's an internal application with 100 users only. The new users won't automatically appear, and they will not open 20 browser tabs trying to complete their work with 20 tabs simultaneously. So if we set JMeter's limit to 100, and test shows "not enough threads to reach the desired workload", it would effectively mean we need to optimize the software or recruit more workers.

---

"thread lifetime" sounds like an interesting idea. The current "thread lifetime" for OMTG is "single execution". In other words, if OMTG is configured for 100 requests, each of them uses its own JMeterThread with its own variables and cookies.

However, it looks like "thread lifetime" alone is not enough. Imagine the following: thread_lifetime(20 min) rate(1/min) random_arrivals(1 day). Suppose JMeter creates the first request, and it completes within seconds. Which thread should be used for the second request? Should JMeter reuse the existing one (it is still available for ~20 minutes) or should it create a new thread? It seems we would need thread_reuse_percentage(42%) so JMeter could know it should reuse one of the existing threads with 42% (which one?) and it should create a new one with 58%.

So the scenario becomes thread_lifetime(20 min) thread_reuse_percentage(42%) rate(1/min) random_arrivals(1 hour). Overall concurrency limit would fit nicely there as a precaution setting to prevent OOM: thread_lifetime(20 min) thread_reuse_percentage(42%) max_concurrency(100) rate(1/min) random_arrivals(1 day).

[vipergreat]

Hi Vlad

I have some use case to share. I'm reproducing the production issue by using JMeter and I think the Open Model Thread Group is the answer. Actually we comparing with bzm - Free Form Arrival Thread Group but I think OMTG is easier to config. We has APIs as follows

1. /api/welcome
2. /api/login <--- This API was pain
3. /api/verify-input
4. /api/submit-input <--- we will simulate response time from 1 sec to 30 sec
5. /api/logout

We have number of Customer arriving rate from the load balancer. It's around 10k/min (we count only first API - welcome). On that day the API no.4 is slow from 1 seconds to 30-40 seconds. We found API no.2 is pain but another API still fine.

So my hypothesis is when Customers wait API no.4 too long they will kill the app and try login and input data again and again the arriving rate increase to 20-25k/min. So the thread lifetime is represent Customer can't wait and kill the app. It's may config the lifetime to only 1-2 minutes per thread.

[ra0077]

Hi I don't know if it can help you, but in HTTP request, you can configure a timeout

[vipergreat]

Hi I don't know if it can help you, but in HTTP request, you can configure a timeout

Hi The default timeout is 30 sec and this value is acceptable in production. But I want to solve the problem about somany thread stuck when API slow response and it's will cause the OutOfMemory error in JMeter when we use OMTG.

[vlsi]

Actually we comparing with bzm - Free Form Arrival Thread Group but I think OMTG is easier to config.

Glad to hear that.

Have you tried wrapping the test with Runtime Controller so it stops the thread if it takes too long? https://jmeter.apache.org/usermanual/component_reference.html#Runtime_Controller

It won't really "retry the steps in case of timeout reached", however, it would probably enable you to configure an overall timeout for several steps.

[vipergreat]

Actually we comparing with bzm - Free Form Arrival Thread Group but I think OMTG is easier to config.

Glad to hear that.

Have you tried wrapping the test with Runtime Controller so it stops the thread if it takes too long? https://jmeter.apache.org/usermanual/component_reference.html#Runtime_Controller

It won't really "retry the steps in case of timeout reached", however, it would probably enable you to configure an overall timeout for several steps.

Interesting! I will try for this. Thanks!

[vipergreat]

@vlsi The Runtime Controller is working for me in this case. Thank you very much!

[JJena]

We always run into jmeter heap issues because it tries to spawn ~ 2000 threads with open model while constant throughput timer/regular thread group can achieve the same TPS with < 10 threads. If you could add a limit to max thread count, it would be such a relief.

[m42cel]

I would also highly appreciate a thread limit.

We're testing our production load with thousands of relatively heavy requests. If we reach a certain point where the system can't keep up the OMTG will spawn more threads to push the configured throughput which makes everything worse by putting even more load on an overloaded system.

I've seen test runs where we had over 60000 threads beating on our already defeated system.

[Akaoni]

+1 for concurrency limiting.