search

Netflix / dynomite

A generic dynamo implementation for different k-v storage engines
Apache License 2.0
4.2k stars 532 forks source link

dynomite crash #696

Open bylijinnan opened 5 years ago

bylijinnan commented 5 years ago

connect to dynomite and execute simple redis command "set a a", then dynomite crash:

[2019-08-21 09:48:07.058] proxy_accept:213 <PROXY 0x137fd40 7 listening on '0.0.0.0:8379'> accepted <CLIENT 0x1384ae0 16 from '127.0.0.1:56662'>
[2019-08-21 09:48:07.058] redis_parse_req:1232 parsed unsupported command 'COMMAND'
[2019-08-21 09:48:07.058] redis_parse_req:1836 parsed bad req 1 res 1 type 0 state 5
00000000  2a 31 0d 0a 24 37 0d 0a  43 4f 4d 4d 41 4e 44 0d   |*1..$7..COMMAND.|
00000010  0a                                                 |.|
[2019-08-21 09:48:07.058] core_close_log:406 close <CLIENT 0x1384ae0 16 from '127.0.0.1:56662'> on event FFFF eof 0 done 0 rb 17 sb 0: Invalid argument
[2019-08-21 09:48:07.058] client_unref_internal_try_put:95 <CLIENT 0x1384ae0 -1 from '127.0.0.1:56662'> unref owner <POOL 0x1374200 'dyn_o_mite'>
[2019-08-21 09:48:24.705] proxy_accept:213 <PROXY 0x137fd40 7 listening on '0.0.0.0:8379'> accepted <CLIENT 0x1384ae0 16 from '127.0.0.1:56664'>
[2019-08-21 09:48:34.524] dn_stacktrace:326 [0] /lib64/libpthread.so.0(+0xf5d0) [0x7f2c35e445d0]
??:0
[2019-08-21 09:48:34.569] dn_stacktrace:326 [1] ../src/dynomite(dmsg_get+0x25) [0x419b45]
/app/dynomite/dynomite2/dynomite/src/dyn_dnode_msg.c:672
[2019-08-21 09:48:34.579] dn_stacktrace:326 [2] ../src/dynomite() [0x41a847]
/app/dynomite/dynomite2/dynomite/src/dyn_dnode_msg.c:47
[2019-08-21 09:48:34.585] dn_stacktrace:326 [3] ../src/dynomite(dyn_parse_rsp+0x40) [0x41b9a0]
/app/dynomite/dynomite2/dynomite/src/dyn_dnode_msg.c:521 (discriminator 3)
[2019-08-21 09:48:34.592] dn_stacktrace:326 [4] ../src/dynomite(msg_recv+0x21e) [0x42341e]
/app/dynomite/dynomite2/dynomite/src/dyn_message.c:948
[2019-08-21 09:48:34.597] dn_stacktrace:326 [5] ../src/dynomite(core_core+0xcf) [0x4162bf]
/app/dynomite/dynomite2/dynomite/src/dyn_core.c:372
[2019-08-21 09:48:34.602] dn_stacktrace:326 [6] ../src/dynomite(event_wait+0x164) [0x43d914]
/app/dynomite/dynomite2/dynomite/src/event/dyn_epoll.c:279
[2019-08-21 09:48:34.607] dn_stacktrace:326 [7] ../src/dynomite(core_loop+0x16a) [0x4171ea]
/app/dynomite/dynomite2/dynomite/src/dyn_core.c:650 (discriminator 3)
[2019-08-21 09:48:34.612] dn_stacktrace:326 [8] ../src/dynomite(main+0x678) [0x40c3a8]
/app/dynomite/dynomite2/dynomite/src/dynomite.c:590
[2019-08-21 09:48:34.617] dn_stacktrace:326 [9] /lib64/libc.so.6(__libc_start_main+0xf5) [0x7f2c357883d5]
??:0
[2019-08-21 09:48:34.621] dn_stacktrace:326 [10] ../src/dynomite() [0x40c7d1]
??:?
[2019-08-21 09:48:34.625] signal_handler:132 signal 11 (SIGSEGV) received, core dumping
Segmentation fault (core dumped)

The core dump file is core-2.zip

The dynomite nodes and redis nodes are in the same machine whose os is CentOS Linux release 7.6.1810.

redis ports: 6xxx dynomite ports 7xxx dynomite listening port 8xxx

Here is my config files:

dyn_o_mite:
  datacenter: dc-a
  rack: rack1
  dyn_listen: 127.0.0.1:7379
  dyn_seeds:
  - 127.0.0.1:7479:rack1:dc-a:2147483647
  - 127.0.0.1:7579:rack1:dc-b:0
  - 127.0.0.1:7679:rack1:dc-b:2147483647
  listen: 0.0.0.0:8379
  servers:
  - 127.0.0.1:6379:1
  tokens: '0'
  secure_server_option: datacenter
  pem_key_file: dynomite.pem
  data_store: 0
  stats_listen: 127.0.0.1:22221
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

dyn_o_mite:
  datacenter: dc-a
  rack: rack1
  dyn_listen: 127.0.0.1:7479
  dyn_seeds:
  - 127.0.0.1:7379:rack1:dc-a:0
  - 127.0.0.1:7579:rack1:dc-b:0
  - 127.0.0.1:7679:rack1:dc-b:2147483647
  listen: 0.0.0.0:8479
  servers:
  - 127.0.0.1:6479:1
  tokens: '2147483647'
  secure_server_option: datacenter
  pem_key_file: dynomite.pem
  data_store: 0
  stats_listen: 127.0.0.1:22222
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

dyn_o_mite:
  datacenter: dc-b
  rack: rack1
  dyn_listen: 127.0.0.1:7579
  dyn_seeds:
  - 127.0.0.1:7379:rack1:dc-a:0
  - 127.0.0.1:7479:rack1:dc-a:2147483647
  - 127.0.0.1:7679:rack1:dc-b:2147483647
  listen: 0.0.0.0:8579
  servers:
  - 127.0.0.1:6579:1
  tokens: '0'
  secure_server_option: datacenter
  pem_key_file: dynomite.pem
  data_store: 0
  stats_listen: 127.0.0.1:22223
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

dyn_o_mite:
  datacenter: dc-b
  rack: rack1
  dyn_listen: 127.0.0.1:7679
  dyn_seeds:
  - 127.0.0.1:7379:rack1:dc-a:0
  - 127.0.0.1:7479:rack1:dc-a:2147483647
  - 127.0.0.1:7579:rack1:dc-b:0
  listen: 0.0.0.0:8679
  servers:
  - 127.0.0.1:6679:1
  tokens: '2147483647'
  secure_server_option: datacenter
  pem_key_file: dynomite.pem
  data_store: 0
  stats_listen: 127.0.0.1:22224
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

Besides, when 4 dynomite nodes are started, they all print "NORMAL" :

[2019-08-21 09:46:39.918] core_print_peer_status:62 1)0xc1e680 dc-a rack1 127.0.0.1:7379:rack1:dc-a:0 NORMAL
[2019-08-21 09:46:39.919] core_print_peer_status:62 0)0xc1e4c0 dc-a rack1 127.0.0.1:7479 NORMAL
[2019-08-21 09:46:39.919] core_print_peer_status:62 2)0xc1e840 dc-b rack1 127.0.0.1:7579:rack1:dc-b:0 NORMAL
[2019-08-21 09:46:39.919] core_print_peer_status:62 3)0xc1ea00 dc-b rack1 127.0.0.1:7679:rack1:dc-b:2147483647 NORMAL