Pending running backend process after heavy put operations

[GoogleCodeExporter]

We use Voldemort as our KV store to evaluate our middleware performance. 
Intermittently, we observed pending Voldemort java process keep running after 
we performed intensive put operations. For example, data upload or update 
intensive benchmark execution. The pending voldemort process would run for a 
long time (sometimes for days) without ending. And it causes CPU bound and 
Memory bound. Our temp solution was to shutdown Voldemort and restart. Most of 
the time the pending process is gone after restart. Any input on what might be 
the cause is most appreciated.      

What steps will reproduce the problem?
1. setup voldemort cluster (3 nodes in our case)
2. run large data upload or update intensive benchmark
3. observe CPU and Memory usage for Voldemort cluster nodes

What is the expected output? What do you see instead?
expect to see low cpu and memory usage after upload, but instead cpu and memory 
contention were observed. Voldemort java process is the consumer.  

What version of the product are you using? On what operating system?
Voldemort-0.90 running on Scientic Linux release 6.1

Please provide any additional information below.

Here is my store.xml contents:

<stores>
    <store>
    <name>userdata</name>
    <persistence>bdb</persistence>
    <routing>client</routing>
    <replication-factor>2</replication-factor>
    <required-reads>1</required-reads>
    <required-writes>2</required-writes>
    <key-serializer>
      <type>identity</type>
    </key-serializer>
    <value-serializer>
    <type>identity</type>
    </value-serializer>
  </store>
</stores>

Here is my server.properties

# The ID of *this* particular cluster node
node.id=0
#voldemort.home=/data/voldemort-0.81
#metadata.directory=/data/voldemort-0.81/config/yoda_cluster

max.threads=5000
core.threads=2500
max_bootstrap_retries=1000

############### DB options ######################

http.enable=true
socket.enable=true
socket.keepalive=true
routing.timeout.ms=1200000000
socket.timeout.ms =1200000000
jmx.enable=true

# BDB
bdb.write.transactions=false
bdb.flush.transactions=false
bdb.cache.size=12G

# Mysql
mysql.host=localhost
mysql.port=3306
mysql.user=root
#mysql.password=
mysql.database=pouria

#NIO connector settings.
enable.nio.connector=false

#storage.configs=voldemort.store.bdb.BdbStorageConfiguration, 
voldemort.store.readonly.ReadOnlyStorageConfiguration
enable.mysql.engine=true

enable.rebalancing=true
rebalancing.optimization=true
max.rebalancing.attempts=10

Original issue reported on code.google.com by ytp...@gmail.com on 9 Mar 2013 at 12:00

[GoogleCodeExporter]

And below are the JVM settings used:

# Min, max, total JVM size
JVM_SIZE="-server -Xms14g -Xmx14g"

# New Generation Sizes
JVM_SIZE_NEW="-XX:NewSize=2048m -XX:MaxNewSize=2048m"

# Type of Garbage Collector to use
JVM_GC_TYPE="-XX:+UseConcMarkSweepGC -XX:+UseParNewGC"

# Tuning options for the above garbage collector
JVM_GC_OPTS="-XX:CMSInitiatingOccupancyFraction=70 -XX:SurvivorRatio=2 
-XX:ParallelGCThreads=50"

# JVM GC activity logging settings
JVM_GC_LOG="-XX:+PrintTenuringDistribution -XX:+PrintGCDetails 
-XX:+PrintGCDateStamps -Xloggc:$LOG_DIR/gc.log"

Original comment by ytp...@gmail.com on 11 Mar 2013 at 3:09