CSV file

[ncaridi]

Hi, Thank you for this .

Can you elaborate how does the csv option works. I have CSV data set config in one of my http/s requests, currently the csv is located in the same folder as the jmx file . Do I have to copy the file to the dataset folder and change the CSV dataset config file name to a different path ?

Thank you.

Nedz.

[Rbillon59]

Hi @ncaridi

You just need to put the filename as the path in the CSV dataset config in JMeter.

With the default configuration, JMeter will look in the bin folder of the apache-jmeter directory.

Thus, the start_test.sh script copy the CSV files from your dataset folder to each pods in the apache-jmeter's bin folder.

So you only need to change the CSV path in JMeter to make it work :)

Tip : Locally you can also put your files in the bin folder of JMeter to open your jmx locally without errors

[ncaridi]

Thanks for the fast reply. Thats exactly my config. I have the filename field setup only with the file name. But what I'm more interested in is the split option. Assuming I have 100k rows in my csv file and I'd like 20 workers each getting unique rows the - c flag would be the route to go for? So in theory if I have users.csv file I'll place the whole file in the dataset folder and it will be splitted and placed in the jmeter bin folder along with the copied jmx in each worker?

Another question, maybe I can get another tip from you ;). Assuming I would like to reach 3k rps in my test run what would you recommend the pod specs to go with?

Thanks for your help

On Sat, Dec 11, 2021, 23:54 Rbillon59 @.***> wrote:

Hi @ncaridi https://github.com/ncaridi

You just need to put the filename as the path in the CSV dataset config in JMeter.

With the default configuration, JMeter will look in the bin folder of the apache-jmeter directory.

Thus, the start_test.sh script copy the CSV files from your dataset folder to each pods in the apache-jmeter's bin folder.

So you only need to change the CSV path in JMeter to make it work :)

Tip : Locally you can also put your files in the bin folder of JMeter to open your jmx locally without errors

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rbillon59/jmeter-k8s-starterkit/issues/12#issuecomment-991794137, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWJIGD7WVHS2FHW5VETEE3UQPCAPANCNFSM5J3NQSTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Rbillon59]

Yes, the -c flag on the start_test.sh will split the CSV files in X pieces where X is the number of injectors asked with the -i flag. So in your case, with 100k rows in your users.csv, you'll end with 5k unique rows per injectors if -i 20.

For your second question about the pods spec to reach 3k RPS. I would say it depends on your JMeter scenario and target application. I mean if you've a lot of regular expressions / assertions, JSON path extractions, JSR223 etc... It will cost a lot of ressources per virtual users. Also, in containers and Kubernetes, I'm more into many small injectors rather than few big ones. (Mainly because JMeter is a Java applications and the more heap you give to the JVM the more GC you get). And the slower is your application, the more virtual users you'll need to reach the targeted RPS.

So without any informations with your scenario or your application, I won't be able to answer correctly. I only can say, I would keep a resource allocation per slave pods of 1024 CPU shares (1 CPU core) and 1GB of Ram.

[ncaridi]

Thank you . That's great info. I'll give it a go with the csv and post back with the findings.

On Sun, Dec 12, 2021 at 3:57 AM Rbillon59 @.***> wrote:

Yes, the -c flag on the start_test.sh will split the CSV files in X pieces where X is the number of injectors asked with the -i flag. So in your case, with 100k rows in your users.csv, you'll end with 5k unique rows per injectors if -i 20.

For your second question about the pods spec to reach 3k RPS. I would say it depends on your JMeter scenario and target application. I mean if you've a lot of regular expressions / assertions, JSON path extractions, JSR223 etc... It will cost a lot of ressources per virtual users. Also, in containers and Kubernetes, I'm more into many small injectors rather than few big ones. (Mainly because JMeter is a Java applications and the more heap you give to the JVM the more GC you get). And the slower is your application, the more virtual users you'll need to reach the targeted RPS.

So without any informations with your scenario or your application, I won't be able to answer correctly. I only can say, I would keep a resource allocation per slave pods of 1024 CPU shares (1 CPU core) and 1GB of Ram.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rbillon59/jmeter-k8s-starterkit/issues/12#issuecomment-991820235, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWJIGDWBOMMTMI6IB2ZOHLUQP6QLANCNFSM5J3NQSTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Rbillon59]

You're welcome ! Really glad that my work can be of use.

Let me know your results

[ncaridi]

Hi So before even trying to load the CSV file , I did some runs with static values. I'm really not sure how to got about this , but I get very bad results when I look at the Grafana. for an example : I get 90% / 95% / 99% 2sec / 2sec /3sec but when I look at my logstash logs my response time from my backends are looking at around 100ms.

whats the best way to verify that this delay isn't caused by the k8s / jemter side ?

Thanks

[Rbillon59]

Through the Jolokia dashboard, monitoring the JMeter slaves JVM.

You need to check the garbage collection time.

How much Virtual users and hit per seconds with how many injectors please?

[ncaridi]

Yeah I tried looking there but the dashboards always look empty. Only the top nodes dashboards have data? Also can I enable saving changes to dashboards I make on this page?

On Tue, Dec 14, 2021, 08:33 Rbillon59 @.***> wrote:

Through the Jolokia dashboard, monitoring the JMeter slaves JVM.

You need to check the garbage collection time.

How much Virtual users and hit per seconds with how many injectors please?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rbillon59/jmeter-k8s-starterkit/issues/12#issuecomment-993204171, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWJIGEJ2NUHGYGUR5XAAP3UQ3QNJANCNFSM5J3NQSTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Rbillon59]

Oh yeah, sometime there is a race condition where the resources (jmeter-pods for example) are created before the telegraf operator so they are not monitored.

But since I upgraded the operator because some API were deprecated, it seems that there is a 400 error when the operator try to attach it's telegraf sidecar to the pods.. So bad. I'll try to fix that asap.

To answer your question about Grafana, the default account is admin and the storage is made in persistantVolumes. Thus if you do not delete the persistantVolume you'll keep your saved dashboard.

Another solution would be to download the modified dashboard json and update the configmap with what you modified

[ncaridi]

Thanks , Really interesting. We had a aws with EC2 setup that got much better results on windows machines, and my backends times looks great when I look at the response time against my Load balancer. So I'm really interested to know / see how to better understand what is causing the bottle neck. I tried playing around with the worker / master configuration files and setup a higher request and limit for both ,I see improvement when giving them more CPU but not so much when I add memory.

In regard to you previous question about the Users /Injectors , I was trying a range of 50 to 200 users in my thread group , with a ramp up of 5 seconds . my worker setup was as follows : requests: memory: "2048Mi" cpu: "6" limits: memory: "2048Mi" cpu: "6"

and 6 to 9 injectors at the moment. my results aren't looking great at the moment but I think it's still due to the jmeter and not my backends as far as I can tell.

Thank you .

On Tue, Dec 14, 2021 at 12:17 PM Rbillon59 @.***> wrote:

Oh yeah, sometime there is a race condition where the resources (jmeter-pods for example) are created before the telegraf operator so they are not monitored.

But since I upgraded the operator because some API were deprecated, it seems that there is a 400 error when the operator try to attach it's telegraf sidecar to the pods.. So bad. I'll try to fix that asap.

To answer your question about Grafana, the default account is admin and the storage is made in persistantVolumes. Thus if you do not delete the persistantVolume you'll keep your saved dashboard.

Another solution would be to download the modified dashboard json and update the configmap with what you modified

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rbillon59/jmeter-k8s-starterkit/issues/12#issuecomment-993387389, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWJIGAKHR7PWBTRWHFWOBTUQ4KURANCNFSM5J3NQSTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[Rbillon59]

I fixed the problem here #14

Can you pull the master again and retry ?

I would keep 1 CPU per injectors and multiply it. Be careful with JMeter. The number of virtual users you set in the JMX is the number PER INJECTORS. It's not splitted across the injectors.

I mean, if you put 200 VU in your JMX with 10 injectors, you'll launch 2000 VU

[ncaridi]

Thanks , yeah it's intentional ,e.g. I'd like to load test my login page and figure out how many logins it can handle per second so I test it vs as many unique logins as possible so 2K VU isn't that much right? isn't this the proper way to achieve this ?

[Rbillon59]

It's the right way, but be aware to put timers in your scenario to avoid your injectors to be overloaded.

Because if no timers are set (constant timers per example), your injectors will send as much traffic as possible. But it's unpredictable and so your load test will be unreproducible as identical.

In performance, we call this kind of tests : limit tests or crash tests. Basically, you'll need to put a timer (I'm a fan of the Constant throughput timer with the "per threads" option. It will adapt the waiting time to match the desired throughput by VU). For example, I will put 60 requests per minute per threads. You will be able to predict pretty easily the desired throughput.

Also in limit testing you put rampup = test duration, thus, you'll increase the load over time until you find the bottleneck of your application. In JMeter it's also beneficial because the JVM have the time to manage the threads and the heap to avoid longs GC and allocations if the rampup is too quick.

[Rbillon59]

FYI I fixed the telegraf operator in my noon break. You can pull the master and try again, you should be able to monitor the jmeter JVM. (You should see two containers by pods)

[ncaridi]

Thanks bottom charts still wouldn't show data , but top jvm show data now I did a run with one injector with cpu req /limit =1 on both worker / master other settings remains the same . I set the ramp up to to 50 vu with 10 sec ramp up can you tell me what I should be looking for in this monitoring image ?

repeated the test again with cpu req/ limit = 2 on both worker / master

second run times were down by more than half But i'm still not sure how to determine where bottlebecks are . Thanks .

[Rbillon59]

Hi again,

you better run like 20 minutes tests with 20 minutes rampup with 1500 virtual users spread on like 10 injectors (so 150VU per injectors) and a constant throughput timer with 120 requests per threads per minutes. Which would make 3000 requests per seconds at the very end of the test.

Because looking at the monitoring for a so short period of time is pretty hard. But to answer your question to know what you need to see. You can just check the "Gargage collection time" graph for the jmeter-slaves. I can't properly see the legend, but the scale of the graph seems to show that the injectors are making 1 to 5 GC with an average GC time of 20-30 milliseconds. Nothing fancy here. It's a normal behaviour for G1 GC.

Finding the bottleneck of your application will be on your side with your monitoring. The stack is to monitor the injection platform and get the JMeter result from the JMeter Grafana dashboard. But you need to monitor your targeted application as well if you want to be sure to find the bottleneck.

[ncaridi]

Thank you.

Appreciated.

On Tue, Dec 14, 2021, 19:03 Rbillon59 @.***> wrote:

Hi again,

you better run like 20 minutes tests with 20 minutes rampup with 1500 virtual users spread on like 10 injectors (so 150VU per injectors) and a constant throughput timer with 120 requests per threads. Which would make 3000 requests per seconds at the very end of the test.

Because looking at the monitoring for a so short period of time is pretty hard. But to answer your question to know what you need to see. You can just check the "Gargage collection time" graph for the jmeter-slaves. I can properly see the legend, but the scale of the graph seems to show that the injectors are making 1 to 5 GC with an average GC time of 20-30 milliseconds. Nothing fancy here. It's a normal behaviour for G1 GC.

Finding the bottleneck of your application will be on your side with your monitoring. The stack is to monitor the injection platform and get the JMeter result from the JMeter Grafana dashboard. But you need to monitor your targeted application as well if you want to be sure to find the bottleneck.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Rbillon59/jmeter-k8s-starterkit/issues/12#issuecomment-993788425, or unsubscribe https://github.com/notifications/unsubscribe-auth/AIWJIGFLT3HHHDXVLLZODEDUQ52GVANCNFSM5J3NQSTQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.