search

apache / jmeter

Apache JMeter open-source load testing tool for analyzing and measuring the performance of a variety of services
https://jmeter.apache.org/
Apache License 2.0
8.31k stars 2.09k forks source link

Jmeter hang when testing javasampler because HashMap.put() is called from multiple threads without sync. #3643

Closed asfimport closed 9 years ago

asfimport commented 9 years ago

Ejay (Bug 58209): Jmeter hang when testing javasampler with num_threads>6000 , because HashMap.put() from multiple threads.

As the following，knowing the thread hang at HashMap.put(); look into the process's jstack info(full content attached)：

2015-08-05 14:48:48 Full thread dump OpenJDK (Taobao) 64-Bit Server VM (20.0-b12-internal mixed mode):

"Attach Listener" daemon prio=10 tid=0x00007f078fd5a000 nid=0x4204 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE

"线??组 1-2788" prio=10 tid=0x00007f07e0111000 nid=0x5fad runnable [0x00007f079d99d000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(HashMap.java:374) at java.util.HashSet.add(HashSet.java:200) at org.apache.jmeter.protocol.java.sampler.JavaSampler.createJavaClient(JavaSampler.java:227) at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:190) at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:431) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:258) at java.lang.Thread.run(Thread.java:662)

"pool-1-thread-1" prio=10 tid=0x00007f0848a90800 nid=0x5c43 waiting on condition [0x00007f07ebca4000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:461) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:360) at java.lang.Thread.run(Thread.java:662)

"线程组 1-1847" prio=10 tid=0x00007f083c96b800 nid=0x5b54 runnable [0x00007f07f9178000 ...

Created attachment jstackinfo: jstack info

jstackinfo ```` 2015-08-05 15:07:34 Full thread dump OpenJDK (Taobao) 64-Bit Server VM (20.0-b12-internal mixed mode): "Attach Listener" daemon prio=10 tid=0x00007f078fd5a000 nid=0x4204 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "线��组 1-2788" prio=10 tid=0x00007f07e0111000 nid=0x5fad runnable [0x00007f079d99d000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(HashMap.java:374) at java.util.HashSet.add(HashSet.java:200) at org.apache.jmeter.protocol.java.sampler.JavaSampler.createJavaClient(JavaSampler.java:227) at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:190) at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:431) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:258) at java.lang.Thread.run(Thread.java:662) "pool-1-thread-1" prio=10 tid=0x00007f0848a90800 nid=0x5c43 waiting on condition [0x00007f07ebca4000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:461) at io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:360) at java.lang.Thread.run(Thread.java:662) "线程组 1-1847" prio=10 tid=0x00007f083c96b800 nid=0x5b54 runnable [0x00007f07f9178000] java.lang.Thread.State: RUNNABLE at java.util.HashMap.put(HashMap.java:374) at java.util.HashSet.add(HashSet.java:200) at org.apache.jmeter.protocol.java.sampler.JavaSampler.createJavaClient(JavaSampler.java:227) at org.apache.jmeter.protocol.java.sampler.JavaSampler.sample(JavaSampler.java:190) at org.apache.jmeter.threads.JMeterThread.process_sampler(JMeterThread.java:431) at org.apache.jmeter.threads.JMeterThread.run(JMeterThread.java:258) at java.lang.Thread.run(Thread.java:662) "nioEventLoopGroup-2-1" prio=10 tid=0x00007f0844652000 nid=0x56d3 runnable [0x000000007efe8000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:210) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:65) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69) - locked <0x0000000217ba7360> (a io.netty.channel.nio.SelectedSelectionKeySet) - locked <0x0000000217bb9e10> (a java.util.Collections$UnmodifiableSet) - locked <0x0000000217bb9c58> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80) at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:622) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:310) at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:116) at io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:137) at java.lang.Thread.run(Thread.java:662) "NanoOffset" daemon prio=10 tid=0x00007f0844159800 nid=0x532c waiting on condition [0x0000000046ea0000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at java.lang.Thread.sleep(Thread.java:302) at java.util.concurrent.TimeUnit.sleep(TimeUnit.java:328) at org.apache.jmeter.samplers.SampleResult$NanoOffset.getOffset(SampleResult.java:1321) at org.apache.jmeter.samplers.SampleResult$NanoOffset.run(SampleResult.java:1314) "DestroyJavaVM" prio=10 tid=0x00007f08444f8000 nid=0x529a waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "UDP Listener" daemon prio=10 tid=0x00007f08444f4800 nid=0x52d9 runnable [0x0000000042557000] java.lang.Thread.State: RUNNABLE at java.net.PlainDatagramSocketImpl.receive0(Native Method) - locked <0x0000000216a00180> (a java.net.PlainDatagramSocketImpl) at java.net.PlainDatagramSocketImpl.receive(PlainDatagramSocketImpl.java:145) - locked <0x0000000216a00180> (a java.net.PlainDatagramSocketImpl) at java.net.DatagramSocket.receive(DatagramSocket.java:725) - locked <0x0000000216a001c0> (a java.net.DatagramPacket) - locked <0x0000000216a001e8> (a java.net.DatagramSocket) at org.apache.jmeter.JMeter.waitForSignals(JMeter.java:1150) at org.apache.jmeter.JMeter.access$200(JMeter.java:99) at org.apache.jmeter.JMeter$2.run(JMeter.java:1133) "StandardJMeterEngine" prio=10 tid=0x00007f08444d3800 nid=0x52d8 in Object.wait() [0x0000000042456000] java.lang.Thread.State: TIMED_WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1194) - locked <0x000000021c386a78> (a java.lang.Thread) at org.apache.jmeter.threads.ThreadGroup.waitThreadStopped(ThreadGroup.java:488) at org.apache.jmeter.threads.ThreadGroup.waitThreadsStopped(ThreadGroup.java:476) at org.apache.jmeter.engine.StandardJMeterEngine.waitThreadsStopped(StandardJMeterEngine.java:514) at org.apache.jmeter.engine.StandardJMeterEngine.run(StandardJMeterEngine.java:424) at java.lang.Thread.run(Thread.java:662) "Low Memory Detector" daemon prio=10 tid=0x00007f0844023800 nid=0x52b4 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread1" daemon prio=10 tid=0x00007f0844021000 nid=0x52b3 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "C2 CompilerThread0" daemon prio=10 tid=0x00007f084401f000 nid=0x52b2 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Signal Dispatcher" daemon prio=10 tid=0x00007f084401c000 nid=0x52b1 runnable [0x0000000000000000] java.lang.Thread.State: RUNNABLE "Finalizer" daemon prio=10 tid=0x00000000401e1800 nid=0x52b0 in Object.wait() [0x00000000404d0000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118) - locked <0x0000000216a08230> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=10 tid=0x00000000401da000 nid=0x52af in Object.wait() [0x00000000417eb000] java.lang.Thread.State: WAITING (on object monitor) at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:485) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) - locked <0x0000000216a10098> (a java.lang.ref.Reference$Lock) "VM Thread" prio=10 tid=0x00000000401d3800 nid=0x52ad runnable "GC task thread#0 (ParallelGC)" prio=10 tid=0x0000000040145800 nid=0x529b runnable "GC task thread#1 (ParallelGC)" prio=10 tid=0x0000000040147800 nid=0x529c runnable "GC task thread#2 (ParallelGC)" prio=10 tid=0x0000000040149800 nid=0x529d runnable "GC task thread#3 (ParallelGC)" prio=10 tid=0x000000004014b000 nid=0x529e runnable "GC task thread#4 (ParallelGC)" prio=10 tid=0x000000004014d000 nid=0x529f runnable "GC task thread#5 (ParallelGC)" prio=10 tid=0x000000004014f000 nid=0x52a0 runnable "GC task thread#6 (ParallelGC)" prio=10 tid=0x0000000040150800 nid=0x52a1 runnable "GC task thread#7 (ParallelGC)" prio=10 tid=0x0000000040152800 nid=0x52a2 runnable "GC task thread#8 (ParallelGC)" prio=10 tid=0x0000000040154800 nid=0x52a3 runnable "GC task thread#9 (ParallelGC)" prio=10 tid=0x0000000040156800 nid=0x52a4 runnable "GC task thread#10 (ParallelGC)" prio=10 tid=0x0000000040158000 nid=0x52a5 runnable "GC task thread#11 (ParallelGC)" prio=10 tid=0x000000004015a000 nid=0x52a6 runnable "GC task thread#12 (ParallelGC)" prio=10 tid=0x000000004015c000 nid=0x52a7 runnable "GC task thread#13 (ParallelGC)" prio=10 tid=0x000000004015e000 nid=0x52a8 runnable "GC task thread#14 (ParallelGC)" prio=10 tid=0x000000004015f800 nid=0x52a9 runnable "GC task thread#15 (ParallelGC)" prio=10 tid=0x0000000040161800 nid=0x52aa runnable "GC task thread#16 (ParallelGC)" prio=10 tid=0x0000000040163800 nid=0x52ab runnable "GC task thread#17 (ParallelGC)" prio=10 tid=0x0000000040165000 nid=0x52ac runnable "VM Periodic Task Thread" prio=10 tid=0x00007f0844036800 nid=0x52b5 waiting on condition JNI global references: 1339 ````

Severity: major OS: All

asfimport commented 9 years ago

Ejay (migrated from Bugzilla): Invoking HashMap.put() from multiple threads casue jmeter hanging

I think the solution is: change java file ：src/protocol/java/org/apache/jmeter/protocol/java/sampler/JavaSampler.java at line 60

//private static final Set<JavaSampler> TEAR_DOWN_SET = new HashSet<JavaSampler>(); private static final Set<JavaSampler> TEAR_DOWN_SET = Collections.newSetFromMap(new ConcurrentHashMap<JavaSampler,Boolean>());

asfimport commented 9 years ago

Sebb (migrated from Bugzilla): Thanks very much for the report. The fix has been applied and will be in the next release of JMeter

URL: http://svn.apache.org/r1694819 Log: JMeter hang when testing javasampler because HashMap.put() is called from multiple threads without sync. https://github.com/apache/jmeter/issues/3643

Modified: jmeter/trunk/src/protocol/java/org/apache/jmeter/protocol/java/sampler/JavaSampler.java jmeter/trunk/xdocs/changes.xml