kubernauts / jmeter-kubernetes

Load testing as a service (LTaaS) with Apache Jmeter on kubernetes
Apache License 2.0
362 stars 249 forks source link

Unable to Create New Native Thread #19

Closed Japeth closed 5 years ago

Japeth commented 5 years ago

Hi, I'm having some issues with java OOM when trying to run with 60+ jmeter slaves. These errors don't occur when running 40 jmeter slaves. The errors I'm getting seem to be happening on the jmeter master pod. When starting my test, I am seeing a ton of: May 07, 2019 5:21:13 PM sun.rmi.transport.tcp.TCPTransport$AcceptLoop executeAcceptLoop WARNING: RMI TCP Accept-0: accept loop for ServerSocket[addr=0.0.0.0/0.0.0.0,localport=41335] throws java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:717) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:420) at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:377) at java.lang.Thread.run(Thread.java:748)

I've tried playing around with Xms, Xmx, & Xss when starting jmeter, but to no avail. I scaled my jmster-master pod up to 4g of memory for limits, but I'm still having the same issues. Does anyone have any thoughts on what/how resources to scale up?

Japeth commented 5 years ago

I've struggled to get a monitoring console connected to the jmeter master, to try and figure out exactly where I'm running out of memory, but haven't had any luck.

It seems the influxDb listener I have on my tests is the root cause of this. I've been searching around on the internet, and while many people either send the results to a database, or use distributed jmeter, it seems no one does both...

One thing I'm looking at is sending all the results to a csv, and then importing to an InfluxDb, but then I lose any results messages (which I want on failures).

Another thought is to revisit using MSSQL, as importing CSVs seems quite a bit simpler.

Overall, I'm pretty disappointed with the performance of jmeter and sending results to a database. You would think this would be a common scenario using distributed tests.

I also investigated trying to have the slave pods bypass sending data back to the master, and instead sending data directly to the database, but it doesn't seem like there is any way to do this.

Japeth commented 5 years ago

Gave up on this. Ended up removing the listeners entirely, and used a different front-end reporting tool for metrics.