twitter / twemproxy

A fast, light-weight proxy for memcached and redis
Apache License 2.0
12.14k stars 2.06k forks source link

Operation timeout on 2000 concurrent connections #490

Open Aditya-Chowdhry opened 8 years ago

Aditya-Chowdhry commented 8 years ago

Why am I getting operation timeout on 2000 concurrent get requests ? yml file config:

beta:
  listen: 127.0.0.1:22122
  hash: fnv1a_64
  hash_tag: "{}"
  distribution: ketama
  auto_eject_hosts: false
  timeout: 500
  redis: true
  servers:
   - 127.0.0.1:6380:1 server1
   - 127.0.0.1:6380:1 server2
   - 127.0.0.1:6381:1 server3
   - 127.0.0.1:6382:1 server4
   - 127.0.0.1:6383:1 server5
   - 127.0.0.1:6384:1 server6

The request time is increasing with the concurrent requests. What should be the ideal timeout value ? Decreasing mbuff value is not helping either. Some logs:

parsed command 'get'
[2016-08-25 16:39:40.192] nc_redis.c:1661 parsed req 2007 res 0 type 42 state 0 rpos 36 of 36
00000000  2a 32 0d 0a 24 33 0d 0a  67 65 74 0d 0a 24 31 36   |*2..$3..get..$16|
00000010  0d 0a 6d 69 67 72 61 74  65 5f 74 65 73 74 31 39   |..migrate_test19|
00000020  39 39 0d 0a                                        |99..|
[2016-08-25 16:39:40.192] nc_server.c:706 key 'migrate_test1999' on dist 0 maps to server '127.0.0.1:6380:1'
[2016-08-25 16:39:40.192] nc_message.c:172 insert msg 2007 into tmo rbt with expiry of 500 msec
[2016-08-25 16:39:40.192] nc_request.c:611 forward from c 7 to s 13 req 2007 len 36 type 42 with key 'migrate_test1999'
[2016-08-25 16:39:40.192] nc_core.c:300 req 5 on s 8 timedout
[2016-08-25 16:39:40.192] nc_message.c:190 delete msg 5 from tmo rbt
[2016-08-25 16:39:40.192] nc_core.c:237 close s 8 '127.0.0.1:6381' on event 0000 eof 0 done 0 rb 0 sb 0: Operation timed out
[2016-08-25 16:39:40.192] nc_server.c:378 close s 8 swallow req 5 len 32 type 0
[2016-08-25 16:39:40.192] nc_server.c:399 close s 8 schedule error for req 3 len 33 type 42 from c 7: Operation timed out
[2016-08-25 16:39:40.192] nc_server.c:399 close s 8 schedule error for req 4 len 33 type 42 from c 7: Operation timed out
[2016-08-25 16:39:40.192] nc_server.c:399 close s 8 schedule error for req 6 len 33 type 42 from c 7: Operation timed out
[2016-08-25 16:39:40.193] nc_server.c:399 close s 8 schedule error for req 7 len 33 type 42 from c 7: Operation timed out

Code snippet:

for ( var i = 1; i<2000; ++i ) {
    var key = "migrate_test" +  String(i);
    var value = "val " + i;
    a = client.get(key, function(err, reply) {
             if (reply == null){
                count = count + 1
                console.log("----------*****---------",count);
             }else{
                console.log(reply);
             }

        });
}
rockysoar commented 8 years ago

Specify twemproxy backlog: 2048 (default is 512) Be sure of that linux kernel net.core.somaxconn is 2048 or bigger.