kubernauts / jmeter-kubernetes

Load testing as a service (LTaaS) with Apache Jmeter on kubernetes
Apache License 2.0
361 stars 250 forks source link

Unable to get results in Grafana #8

Closed subinmt closed 5 years ago

subinmt commented 6 years ago

I have setup the same in my Kubernetes Cluster. But Nothing showing in the grafana. How can I show the hit/test results in Grafana. I already configured the DataSource in Grafana. But nothing showing in the Grafana.Please help.

infinitydon commented 6 years ago

Check the date interval selection, it defaults to 2017.

subinmt commented 6 years ago

@infinitydon I have corrected the date. But the issue still persists. Please I need an urgent help. I have configured all things like docs except grafana reporter. Because, I don want that PDF sheet. I need to disaplay the hit/load test in my Grafana. So please help.

infinitydon commented 6 years ago

You have to give some more information: 1.) Are you getting any errors when you are starting a test? How are you starting your test? Are you using the sample test script that we provided? 2.) Is there any abnormalities in the jmeter.log in the Master Jmeter Pod?

The PDF reporter is optional, this will not affect your result.

subinmt commented 6 years ago

@infinitydon Yes. I have started the ¨start_test.sh¨ with myown .jmx file. And ¨start_test.sh¨ script successfully executed like below:

+++++++++++ Remote engines have been started Waiting for possible Shutdown/StopTestNow/Heapdump message on port 4445 summary + 1 in 00:00:00 = 8.9/s Avg: 13 Min: 13 Max: 13 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0 summary + 79 in 00:00:05 = 15.7/s Avg: 18 Min: 0 Max: 149 Err: 0 (0.00%) Active: 0 Started: 20 Finished: 20 summary = 80 in 00:00:05 = 15.5/s Avg: 18 Min: 0 Max: 149 Err: 0 (0.00%) Tidying up remote @ Sat Sep 01 02:13:10 UTC 2018 (1523456767325) ... end of run +++++++++++

I am using myown JMeter and also tried with your package, but both are nothing showing in the Grafana. I have added the following parameter to the Jmeter master pod ¨load_test¨ script file:

+++++ -GHOST=35.xxx.xxx.112 -GVU=20 -GRAMP=2 -GLOOP=1 +++++

The -GHOST is my websocket(like message broker) IP. Also I have noted that the connections/hits are comming to my Websocket/Kafka, but the grafana doesn shows anything. See the jmeter.log results in the JMeter Master pod:

++++++++++++++++ 2018-09-01 02:13:03,340 INFO o.a.j.u.JMeterUtils: Setting Locale to en_EN 2018-09-01 02:13:03,363 INFO o.a.j.JMeter: Loading user properties from: /jmeter/apache-jmeter-4.0/bin/user.properties 2018-09-01 02:13:03,364 INFO o.a.j.JMeter: Loading system properties from: /jmeter/apache-jmeter-4.0/bin/system.properties 2018-09-01 02:13:03,364 INFO o.a.j.JMeter: Setting Global property: HOST=35.xxx.xxx.112 2018-09-01 02:13:03,364 INFO o.a.j.JMeter: Setting Global property: VU=20 2018-09-01 02:13:03,364 INFO o.a.j.JMeter: Setting Global property: RAMP=2 2018-09-01 02:13:03,365 INFO o.a.j.JMeter: Setting Global property: LOOP=1 2018-09-01 02:13:03,365 INFO o.a.j.JMeter: Setting System property: server.rmi.ssl.disable=true 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: Copyright (c) 1998-2018 The Apache Software Foundation 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: Version 4.0 r1823414 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: java.version=1.8.0_181 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: java.vm.name=OpenJDK 64-Bit Server VM 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: os.name=Linux 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: os.arch=amd64 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: os.version=4.13.0-1006-gcp 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: file.encoding=UTF-8 2018-09-01 02:13:03,372 INFO o.a.j.JMeter: Max memory =6442450944 2018-09-01 02:13:03,373 INFO o.a.j.JMeter: Available Processors =32 2018-09-01 02:13:03,379 INFO o.a.j.JMeter: Default Locale=English (EN) 2018-09-01 02:13:03,379 INFO o.a.j.JMeter: JMeter Locale=English (EN) 2018-09-01 02:13:03,379 INFO o.a.j.JMeter: JMeterHome=/jmeter/apache-jmeter-4.0 2018-09-01 02:13:03,379 INFO o.a.j.JMeter: user.dir =/ 2018-09-01 02:13:03,380 INFO o.a.j.JMeter: PWD =/ 2018-09-01 02:13:03,380 INFO o.a.j.JMeter: IP: 100.96.1.92 Name: jmeter-master-6b9867cc79-cfp82 FullName: jmeter-master-6b9867cc79-cfp82 2018-09-01 02:13:03,385 INFO o.a.j.s.FileServer: Default base='/' 2018-09-01 02:13:03,392 INFO o.a.j.s.FileServer: Set new base='/' 2018-09-01 02:13:03,562 INFO o.a.j.s.SaveService: Testplan (JMX) version: 2.2. Testlog (JTL) version: 2.2 2018-09-01 02:13:03,578 INFO o.a.j.s.SaveService: Using SaveService properties file encoding UTF-8 2018-09-01 02:13:03,585 INFO o.a.j.s.SaveService: Using SaveService properties version 4.0 2018-09-01 02:13:03,593 INFO o.a.j.s.SaveService: Loading file: EnduranceTest.jmx 2018-09-01 02:13:03,662 WARN e.l.j.w.WebsocketSampler: Detected Maciej Zaleski's WebSocket Sampler plugin is installed too, which is not compatible with this plugin (but both can co-exist). 2018-09-01 02:13:03,687 INFO o.a.j.JMeter: Creating summariser

2018-09-01 02:13:03,693 INFO o.a.j.e.DistributedRunner: Configuring remote engine: 100.68.25.21 2018-09-01 02:13:03,695 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true' 2018-09-01 02:13:03,748 INFO o.a.j.e.DistributedRunner: Starting remote engines 2018-09-01 02:13:03,750 INFO o.a.j.e.DistributedRunner: Starting the test @ Sat Sep 01 02:13:03 UTC 2018 (1523456767325) 2018-09-01 02:13:03,751 INFO o.a.j.e.ClientJMeterEngine: running clientengine run method 2018-09-01 02:13:03,757 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must contain the string: '.functions.' 2018-09-01 02:13:03,757 INFO o.a.j.e.u.CompoundVariable: Note: Function class names must not contain the string: '.gui.' 2018-09-01 02:13:04,267 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true' 2018-09-01 02:13:04,267 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true' 2018-09-01 02:13:04,269 INFO o.a.j.s.SampleEvent: List of sample_variables: [] 2018-09-01 02:13:04,275 INFO o.a.j.s.BatchSampleSender: Using batching (client settings) for this run. Thresholds: num=100, time=60000 2018-09-01 02:13:04,275 INFO o.a.j.s.DataStrippingSampleSender: Using DataStrippingSampleSender for this run 2018-09-01 02:13:04,276 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true' 2018-09-01 02:13:04,277 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true' 2018-09-01 02:13:05,089 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true' 2018-09-01 02:13:05,089 INFO o.a.j.r.RmiUtils: Disabling SSL for RMI as server.rmi.ssl.disable is set to 'true' 2018-09-01 02:13:05,167 INFO o.a.j.e.ClientJMeterEngine: sent test to 100.68.25.21 basedir='.' 2018-09-01 02:13:05,167 INFO o.a.j.e.ClientJMeterEngine: Sending properties {HOST=35.xxx.xxx.112, LOOP=1, RAMP=2, VU=20} 2018-09-01 02:13:05,172 INFO o.a.j.e.ClientJMeterEngine: sent run command to 100.68.25.21 2018-09-01 02:13:05,172 INFO o.a.j.e.DistributedRunner: Remote engines have been started 2018-09-01 02:13:05,177 INFO o.a.j.JMeter: Started remote host: 100.68.25.21 (1523456767325) 2018-09-01 02:13:05,282 INFO o.a.j.s.SampleResult: Note: Sample TimeStamps are START times 2018-09-01 02:13:05,282 INFO o.a.j.s.SampleResult: sampleresult.default.encoding is set to ISO-8859-1 2018-09-01 02:13:05,282 INFO o.a.j.s.SampleResult: sampleresult.useNanoTime=true 2018-09-01 02:13:05,282 INFO o.a.j.s.SampleResult: sampleresult.nanoThreadSleep=5000 2018-09-01 02:13:05,288 INFO o.a.j.r.Summariser: summary + 1 in 00:00:00 = 8.9/s Avg: 13 Min: 13 Max: 13 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0 2018-09-01 02:13:10,324 INFO o.a.j.r.Summariser: summary + 79 in 00:00:05 = 15.7/s Avg: 18 Min: 0 Max: 149 Err: 0 (0.00%) Active: 0 Started: 20 Finished: 20 2018-09-01 02:13:10,324 INFO o.a.j.r.Summariser: summary = 80 in 00:00:05 = 15.5/s Avg: 18 Min: 0 Max: 149 Err: 0 (0.00%) 2018-09-01 02:13:10,325 INFO o.a.j.JMeter: Finished remote host: 100.68.25.21 (1535767990325) 2018-09-01 02:13:15,328 INFO o.a.j.JMeter: Interrupting RMI Reaper

++++++++++++++++

I couldn´t found anything unsual on this log file. Also I have noted that the ClientJMeterEngine Sending properties to my websocket. Please help. My Grafana still nothong showing.

infinitydon commented 6 years ago

I think the issue should be in the influxdb, check the jmeter.log to see if there is an issue writing to the Influxdb backend listener..

Grafana can connect to influxdb datasource successfully?

subinmt commented 6 years ago

@infinitydon I have already provided the jmeter.log file on my previous reply. Also I have recheck with the writing issues, but I couldn´t find anything unusual. Grafana successfully connect to the influxdb without any issue. But I am also suspecting the issue is related on influxdb, because I have exposed the influxdb sevice IP. But I couldn´t access the influxdb on my browser with port 8083 or 8086. Browser showing ¨Unable to connect¨ message. Can you find a solution for this. This is an urgent requirement. Please help. I have one doubt, JMeter need any metrics for showing the results in Grafana using influxdb.? Because Prometheus is fetching log/results based on the metrics. So can you tell me, influxdb need anything like that.?

infinitydon commented 6 years ago

In your jmeter.log you should see the following about the Influxdb backend listener:

2018-09-01 10:30:31,031 INFO o.a.j.v.b.BackendListener: Backend Listener: Starting worker with class: class org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient and queue capacity: 5000 2018-09-01 10:30:31,039 INFO o.a.j.v.b.BackendListener: Backend Listener: Started worker with class: class org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient

FYI -- I have just setup a new cluster following the documentation and everything is working as it should.

You can't access the InlfuxDB service because it is a ClusterIP, it is only reachable within the k8s cluster, you don't have to manually expose the InfluxDB service if you followed the documentation, the scripts will setup the services when you initiate it.

Your problem seems to be with the InfluxDB.. Can you provide the output of the service and pods under the namespace you used to install Jmeter?

subinmt commented 6 years ago

@infinitydon But please note that the BackendListener showing INFO not any ERROR/WARNING,right? I have installed everything based on your docs and I manually exposed the Influxdb services. But still I can´t access the Influxdb on my brower with 8083/8036. FYI, please see the K8s Pods and Servics:

++++++++++++++++++++++++++++++ PODS: NAME READY STATUS RESTARTS AGE influxdb-jmeter-c56fc9c86-8wddq 1/1 Running 0 18h jmeter-grafana-659445996d-gs4vb 1/1 Running 0 18h jmeter-master-6b9867cc79-cfp82 1/1 Running 0 3d jmeter-slaves-7f77759df4-456m4 1/1 Running 0 3d jmeter-slaves-7f77759df4-hnhkp 1/1 Running 0 3d

SERVICS

NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE jmeter-grafana LoadBalancer 100.21.54.100 35.xxx.xxx.132 3000:31843/TCP 18h jmeter-influxdb LoadBalancer 100.22.108.101 104.xxx.xxx.131 8083:31496/TCP,8086:32224/TCP,2003:32578/TCP 18h jmeter-slaves-svc ClusterIP 100.68.25.21 1099/TCP,50000/TCP 2d

++++++++++++++++++++++++++++++

I don know what is wrong with my setup.

infinitydon commented 6 years ago

I could not find any influxdb entry in the jmeter.log that you pasted. The logs I pasted is what you should see in your jmeter.log.

Steps to follow to get a working jmeter cluster:

1.) ./jmeter_cluster.sh

Wait for the all the pods to be running and ready before proceeding.

2.) ./dashboard.sh

3.) ./start_test.sh

If you followed the steps above, I see no reason why it will not work.

Your PODs are having different uptimes (any reason for that?), can you delete the whole namespace you used for the jmeter and start all over?

subinmt commented 6 years ago

@infinitydon I have followed the same steps you have provided. Yes, my pods and services are showing different ages, because I have terminated and recreated the same in several time. I have two more time deleted the all things related to JMeter, but issue still persists.

infinitydon commented 6 years ago

Can you see the influxdb in the jmeter.log when you run a test?

Can't really say what may be wrong, you can join the slack channel kubernauts.slack.com and buzz me @infinitydon, might need to check your grafana dashboard if you don't mind.

subinmt commented 6 years ago

@infinitydon I have already show you the jmeter.log file. I was copied all logs inside the jmeter.log file. I couldn´t find any influxdb details on that.Let me rethink about the grafana dashboard case.

infinitydon commented 6 years ago

You must see the following in the logs:

root@jmeter-master-5477768dd8-pw2pk:/jmeter# grep -i influxdb /jmeter.log 2018-09-01 10:30:31,031 INFO o.a.j.v.b.BackendListener: Backend Listener: Starting worker with class: class org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient and queue capacity: 5000 2018-09-01 10:30:31,039 INFO o.a.j.v.b.BackendListener: Backend Listener: Started worker with class: class org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient 2018-09-01 10:51:40,489 INFO o.a.j.v.b.i.InfluxdbBackendListenerClient: Sending last metrics

If it is not there, it means jmeter master is not sending metrics to the InfluxDB database or jmx script is not configured to use InfluxDB as a backend listener.

Are you getting the same thing when you use the cloudssky.jmx test script?

The sample test script "cloudssky.jmx" has InfluxDB configured as a backend listener:

          **_<elementProp name="influxdbMetricsSender" elementType="Argument">
            <stringProp name="Argument.name">influxdbMetricsSender</stringProp>
            <stringProp name="Argument.value">org.apache.jmeter.visualizers.backend.influxdb.HttpMetricsSender</stringProp>
          <elementProp name="influxdbUrl" elementType="Argument">
            <stringProp name="Argument.name">influxdbUrl</stringProp>
            <stringProp name="Argument.value">http://jmeter-influxdb:8086/write?db=jmeter</stringProp>
      <stringProp name="classname">org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient</stringProp>_**
subinmt commented 6 years ago

@infinitydon Yes. Thats the problem. I found it. My .jmx file doesn´t have any entry to Influxdb backendListerner. Thank you for your big help. Shall I copy and paste the above code to my .jmx file, it would work for me? Please give me the guide line

infinitydon commented 6 years ago

I suggest you use the GUI to edit the jmx file to be on the safe side, sample:

image