search

authorjapps / zerocode

A community-developed, free, opensource, automated testing framework for microservices APIs, Kafka(Data Streams) and Load testing. Zerocode Open Source enables you to create, change and maintain your automated test scenarios via simple JSON or YAML files. Visit documentation below:
https://zerocode-tdd.tddfy.com
Apache License 2.0
907 stars 398 forks source link

ramp.up.period.in.seconds=0 does not work #418

Open metrospiegel opened 4 years ago

metrospiegel commented 4 years ago

I guess the log says it all. The last part repeats for inifinity. Nothing gets executed. With ramp.up.period.in.seconds=1 and a large number users he gets quickly hickups till he chokes kompletely thill he is also sending nearly nothing out. We use a custom HTTP conection, bit this should not matter i guess?

2020-07-29 10:55:38,405 [main] WARN org.jsmart.zerocode.parallel.ExecutorServiceRunner - LOAD:

numberOfThreads : 6

rampUpPeriodInSeconds : 0

loopCount : 2500000

2020-07-29 10:55:38,421 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,421 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.421227500 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [pool-2-thread-1] INFO org.jsmart.zerocode.core.runner.parallel.LoadProcessor - pool-2-thread-1 Parallel Junit test- Start. Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [pool-2-thread-2] INFO org.jsmart.zerocode.core.runner.parallel.LoadProcessor - pool-2-thread-2 Parallel Junit test- Start. Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [pool-2-thread-3] INFO org.jsmart.zerocode.core.runner.parallel.LoadProcessor - pool-2-thread-3 Parallel Junit test- Start. Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [pool-2-thread-4] INFO org.jsmart.zerocode.core.runner.parallel.LoadProcessor - pool-2-thread-4 Parallel Junit test- Start. Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [pool-2-thread-5] INFO org.jsmart.zerocode.core.runner.parallel.LoadProcessor - pool-2-thread-5 Parallel Junit test- Start. Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [pool-2-thread-6] INFO org.jsmart.zerocode.core.runner.parallel.LoadProcessor - pool-2-thread-6 Parallel Junit test- Start. Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,436 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.436891800 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-07-29T10:55:38.452522300 2020-07-29 10:55:38,452 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0

authorjapps commented 4 years ago

@metrospiegel , can you try this... clone this working samples and run locally? Then compare where things are going wrong. Link: Try AT Home Performance Tests - Working Examples

Also can you paste here your runners configs?

metrospiegel commented 4 years ago

We checked with: Try AT Home Performance Tests - Working Examples

For reproduction it is sufficent to run the LoadGetTest and chane the load_generation.properties as follows: number.of.threads=300 ramp.up.period.in.seconds=0 loop.count=1000000

He will do some calls in the beginning but in the end will get stuck in repeated: 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-03T09:59:46.912 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 0.0 2020-08-03 09:59:46,912 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - *Start... Time = 2020-08-03T09:59:46.912

In other scenariaos we found that it 'seems' that workers are dying over time. We start with 100 requests coming to the server readily responding and after 1 hour only around 10 request trickle in (even though config would suggest oherwise). Somthing with the sheduling agorythim? We are currently not able to pertain a high load on the server over more than maybe 5 min?

authorjapps commented 4 years ago

@metrospiegel , we are trying to understand the issue. Is the following(1) an issue?, 1)

number.of.threads=300
ramp.up.period.in.seconds=0
loop.count=1000000

but, the below(2) works just fine? 2)

number.of.threads=5
ramp.up.period.in.seconds=0
loop.count=2

Note: No.2 above has less virtual users and fewer loop count than No.1

We start with 100 requests coming to the server readily responding and after 1 hour only around 10 request trickle in

What is the system(laptop or PC?) you use to generate the load? Precisely, can you put some more details e.g.

around 10 request trickle in

I am slightly concerned here, sounds like a very low-end machine is used for such a high load... the machine is unable to spawn more threads(virtual users)! Not entirely sure... but there might be some other underlying issues too.

metrospiegel commented 4 years ago

1) is an issue; As described he is not doing anything in our test. In try at home he makes maybe 2 requests an than he is waiting and doing nothing. 2) no issue.

However we need large input with maximum stress. Server is not the issue (in my opinion).

On the server: 32 cores; 64GB ram; Windows: solely used for load generation (ram and CPU not fully utelized during run at 200 treads ~40%cpu ram no problem at all) Laptop: 4cpu 8 threads; 32GB ram, Windows used for Try AT Home Performance Tests

metrospiegel commented 4 years ago

Importand also to notice that agound 30min in config

number.of.threads=200 ramp.up.period.in.seconds=0 loop.count=1000000

maybe 5 threads are left actively inserting?

In our test he first does make around 100 to 200 requests a sectond but that quickly declines.

authorjapps commented 4 years ago

Laptop: 4cpu 8 threads; 32GB ram, Windows used for Try AT Home Performance Tests

Thanks for the details of the client specs, That's important for now. Server specs are not really of a concern here as of now, but we will address that in a bit(for now I am assuming yours is a single machine/server!)

In our test he first does make around 100 to 200 requests a second but that quickly declines.

Yes, That's expected and that's exactly the issue. The ramping-up of threads in your case, is limited by the laptop which is a low config machine for your desired load requirement.

That's why I asked earlier:

Next questions are:

Once you have the answers, you can easily figure out why the threads(virtual users) are ramping-down!

metrospiegel commented 4 years ago

Thanks for the details of the client specs, That's important for now. Server specs are not really of a concern here as of now, but we will address that in a bit(for now I am assuming yours is a single machine/server!)

It runs 1:1 virtualized. You can assume one machine without any config probems.

Yes, That's expected and that's exactly the issue. The ramping-up of threads in your case, is limited by the laptop which is a low config machine for your desired load requirement.

The Laptop i used only for the try at home stuff for reproduction. At the time i used it only for laod. What i am concerned about is the server.

how much ram was left free before/while you started firing the load ? you need to monitor this if it touches 100% at any time Yes: it bounces 2 times at 90 to 100 and goes to 40

how much CPU %age was used before you started firing the load ? 0 load on the CPUs

what's your payload size of each request(KB or MB)? 20kb

Ram usege is 11GB so no problem there.

The thing for me is, he seems to skip request. It is also not clear to me why it can happen that he is somtimes not firing anything just beacuse ramp up is 0. I woud guess an best effort implementation.

I am in a mode where i am not concerend about the performance measurements and sideffects on the results. I just want to put stress on the server utelizing the full power of the test server. Basically i want to have 0 ramp up, much as possible threads/users and infinity loop.

I will try configure peakload below 80 cpu usage and check what is happening, as it seems to me the only limiting factor.

metrospiegel commented 4 years ago

Ich checked my testcases on the server and ensured 0 payload on the server and never more tane 40% cpu usage during execution. the result is: number.of.threads=10 ramp.up.period.in.seconds=0 loop.count=2000000 -> nothing gests send

number.of.threads=2 ramp.up.period.in.seconds=0 loop.count=2000000 -> nothing gets send

number.of.threads=2 ramp.up.period.in.seconds=1 loop.count=2000000 -> working

The cpu load stayed always below 40. Ram and bandwith also no problem?

authorjapps commented 4 years ago

Set to 1 sec and try e.g.

ramp.up.period.in.seconds=1

Which means all virtual users will be launched in a span of 1sec.

0 to 1 won't make a huge difference in terms of concurrency. But it might help to visualize the issue.

Then try with :

ramp.up.period.in.seconds=2 or 5

and see if it makes any difference...

metrospiegel commented 4 years ago

In all cases the number of requests is dwindeling down ower time. Making 2000000 nearly unreachable (which it does not neet to rach anyways). Since i don't have a spechific measurment scenario and i am not concerned about measurment accuracy is there a prossebility do build in a no synchronization mode to avoid:

2020-08-04 08:51:07,692 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - Waiting for the next test flight to adjust the overall ramp up time, waiting time in the transit now = 500.0 2020-08-04 08:51:08,192 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Start... Time = 2020-08-04T08:51:08.192702100 2020-08-04 08:51:08,192 [main] INFO org.jsmart.zerocode.parallel.ExecutorServiceRunner - main Executor - Finished Time = 2020-08-04T08:51:08.192702100

A mode where he ist just firing after successful exection in the same thread (no sync)? I would make a feature request? I would like to use the same zearocode testcases for this. Not needing to attune the testparameters for the machine at hand. Just get the most out of the server executing the tests (no delay no sync).