search

Netflix / dynomite

A generic dynamo implementation for different k-v storage engines
Apache License 2.0
4.2k stars 532 forks source link

Crush when redis-benchmark ping #712

Open rnz opened 5 years ago

rnz commented 5 years ago

On remote host:

root@test-y:~/memtier/memtier_benchmark-1.2.17# redis-benchmark -h test-b -p 8102 -t ping
Error: Connection reset by peer

On dynomite host:

root@test-b:~/src/dynomite/dynomite# src/dynomite -c ../redis-node.yml
[2019-10-02 11:50:27.159] dn_print_run:206 dynomite-v0.6.15-6-g3b153a0 built for Linux 4.15.18-20-pve x86_64 started on pid 1083
[2019-10-02 11:50:27.159] dn_print_run:212 run, rabbit run / dig that hole, forget the sun / and when at last the work is done / don't sit down / it's time to dig another one
...

[2019-10-02 11:51:38.694] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2c6240 19 from '10.100.223.206:57668'>
[2019-10-02 11:51:38.695] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2d9d90 20 from '10.100.223.206:57670'>
[2019-10-02 11:51:38.697] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2da0a0 21 from '10.100.223.206:57672'>
[2019-10-02 11:51:38.698] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2da3d0 22 from '10.100.223.206:57674'>
[2019-10-02 11:51:38.699] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2da700 23 from '10.100.223.206:57676'>
[2019-10-02 11:51:38.700] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2daa30 24 from '10.100.223.206:57678'>
[2019-10-02 11:51:38.701] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dad60 25 from '10.100.223.206:57680'>
[2019-10-02 11:51:38.702] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2db090 26 from '10.100.223.206:57682'>
[2019-10-02 11:51:38.703] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2db3c0 27 from '10.100.223.206:57684'>
[2019-10-02 11:51:38.704] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2db6f0 28 from '10.100.223.206:57686'>
[2019-10-02 11:51:38.706] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dba20 29 from '10.100.223.206:57688'>
[2019-10-02 11:51:38.708] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dbd50 30 from '10.100.223.206:57690'>
[2019-10-02 11:51:38.710] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dc080 31 from '10.100.223.206:57692'>
[2019-10-02 11:51:38.711] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dc3b0 32 from '10.100.223.206:57694'>
[2019-10-02 11:51:38.712] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dc6e0 33 from '10.100.223.206:57696'>
[2019-10-02 11:51:38.712] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dca10 34 from '10.100.223.206:57698'>
[2019-10-02 11:51:38.713] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dcd40 35 from '10.100.223.206:57700'>
[2019-10-02 11:51:38.714] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dd070 36 from '10.100.223.206:57702'>
[2019-10-02 11:51:38.714] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dd3a0 37 from '10.100.223.206:57704'>
[2019-10-02 11:51:38.716] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dd6d0 38 from '10.100.223.206:57706'>
[2019-10-02 11:51:38.717] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dda00 39 from '10.100.223.206:57708'>
[2019-10-02 11:51:38.717] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2ddd30 40 from '10.100.223.206:57710'>
[2019-10-02 11:51:38.718] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2de060 41 from '10.100.223.206:57712'>
[2019-10-02 11:51:38.719] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2de390 42 from '10.100.223.206:57714'>
[2019-10-02 11:51:38.720] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2de6c0 43 from '10.100.223.206:57716'>
[2019-10-02 11:51:38.722] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2de9f0 44 from '10.100.223.206:57718'>
[2019-10-02 11:51:38.723] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2ded20 45 from '10.100.223.206:57720'>
[2019-10-02 11:51:38.724] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2df050 46 from '10.100.223.206:57722'>
[2019-10-02 11:51:38.725] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2df380 47 from '10.100.223.206:57724'>
[2019-10-02 11:51:38.727] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2df6b0 48 from '10.100.223.206:57726'>
[2019-10-02 11:51:38.728] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2df9e0 49 from '10.100.223.206:57728'>
[2019-10-02 11:51:38.728] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2dfd10 50 from '10.100.223.206:57730'>
[2019-10-02 11:51:38.729] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e0040 51 from '10.100.223.206:57732'>
[2019-10-02 11:51:38.730] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e0370 52 from '10.100.223.206:57734'>
[2019-10-02 11:51:38.731] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e06a0 53 from '10.100.223.206:57736'>
[2019-10-02 11:51:38.732] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e09d0 54 from '10.100.223.206:57738'>
[2019-10-02 11:51:38.733] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e0d00 55 from '10.100.223.206:57740'>
[2019-10-02 11:51:38.733] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e1030 56 from '10.100.223.206:57742'>
[2019-10-02 11:51:38.734] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e1360 57 from '10.100.223.206:57744'>
[2019-10-02 11:51:38.735] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e1690 58 from '10.100.223.206:57746'>
[2019-10-02 11:51:38.735] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e19c0 59 from '10.100.223.206:57748'>
[2019-10-02 11:51:38.736] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e1cf0 60 from '10.100.223.206:57750'>
[2019-10-02 11:51:38.736] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e2020 61 from '10.100.223.206:57752'>
[2019-10-02 11:51:38.737] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e2350 62 from '10.100.223.206:57754'>
[2019-10-02 11:51:38.737] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e2680 63 from '10.100.223.206:57756'>
[2019-10-02 11:51:38.738] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e29b0 64 from '10.100.223.206:57758'>
[2019-10-02 11:51:38.739] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e2ce0 65 from '10.100.223.206:57760'>
[2019-10-02 11:51:38.740] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e3010 66 from '10.100.223.206:57762'>
[2019-10-02 11:51:38.741] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e3340 67 from '10.100.223.206:57764'>
[2019-10-02 11:51:38.742] proxy_accept:203 <CONN_PROXY 0x555a6f2c2190 13 listening on '0.0.0.0:8102'> accepted <CONN_CLIENT 0x555a6f2e3670 68 from '10.100.223.206:57766'>
[2019-10-02 11:51:38.742] req_server_enqueue_imsgq:913 Bad tailq NEXT(0x555a6f2b5dd8->tqh_last) != NULL
Aborted (core dumped)

Config:

root@test-b:~# cat src/dynomite/redis-node.yml 
dyn_o_mite:
  datacenter: dc
  rack: rack1
  dyn_listen: 0.0.0.0:8101
  dyn_seeds:
  - test-c:8101:rack2:dc:1383429731
  - test-d:8101:rack3:dc:1383429731
  listen: 0.0.0.0:8102
  servers:
  - 127.0.0.1:6379:1
  tokens: '1383429731'
  # secure_server_option: datacenter
  # pem_key_file: conf/dynomite.pem
  data_store: 0
  read_consistency : DC_SAFE_QUORUM
  write_consistency : DC_SAFE_QUORUM
  stats_listen: 0.0.0.0:22221