sqs input with 20 threads drops avg throughput to 1/4th after ~2 hours

[nellicus]

Logstash 1.4.2 pulling events from 2 SQS queues(assume infinite number of events) and feeding into ES with transport using 1 worker, keeps throughput steady to 400 indexing request per second towards ES, then after ~90 minutes indexing requests to ES drop to 100. (No issues whatsoever on ES side).

Reproduced on AWS

Logstash config

input { 
    sqs {
                access_key_id => "secret"
                secret_access_key => "secret"
                queue => "test1"
                region => "xxxxxxxx"
                threads => 20
            }

    sqs {
                access_key_id => "secret"
                secret_access_key => "secret"
                queue => "test2"
                region => "xxxxxxxxx"
                threads => 20
            }
} 

filter { 
}

output { 

        stdout { codec => rubydebug }

    elasticsearch {
                cluster => "xxxxxx"
                protocol => "transport"
                workers => 1
                host => "xxxxxxxxxxx"
                port => "9300"
                index => "xxxxxxx"

        }   
} 

LS debug log and output of

while true;do date >> /tmp/7113-stats.log;jstat  -gccapacity 13682 >> /tmp/7113-stats.log ;jstat -gc 13682 >> /tmp/7113-stats.log; sleep 5;done

at https://s3-eu-west-1.amazonaws.com/users.eu.elasticsearch.org/abonuccelli/7113-reproduced.tar.gz

Attaching es screenshots showing indexing requests going down

[nellicus]

marvel time is GMT+1 (sudden drop plotted between ~18:45-18:47) logs time is GMT