search

Netflix / dynomite

A generic dynamo implementation for different k-v storage engines
Apache License 2.0
4.2k stars 532 forks source link

Dynomite log meaning #798

Open abhinav0101 opened 3 years ago

abhinav0101 commented 3 years ago

Can someone explain what do these logs mean? Specially the "on event 00FF eof 1 done 1 rb 3536 sb 1221" part ?

proxy_accept:220 accepted CLIENT 14 on PROXY 8 from '10.150.114.162:54142' [2021-06-24 13:22:14.032] core_close_log:343 close CLIENT 166 '10.150.114.144:41368' on event 00FF eof 1 done 1 rb 3536 sb 1221

abhinav0101 commented 3 years ago

@ipapapa @diegopacheco

shailesh33 commented 3 years ago

proxy_accept : function name CLIENT: type of connection PROXY: (if my memory serves me well) type of listener in dynomite (there is PROXY for listening on 8102 and peer proxy for listerning on 8101 for talking to other dynomite nodes) 166: every connection is given an internal identifier for debugging purpose in the logs 00FF: I think this is the events received by select call. eof 1 : received EOF done 1: I think it means that its done sending data. rb: read buffer size sb: send buffer size

I hope I got it all correct. @smukil can confirm.

abhinav0101 commented 3 years ago

Hey @shailesh33 Thanks a lot for the quick reply. Support from you guys would ensure that this community keeps growing.

I would really like some help with the below questions.

1.) What do the codes "220" & "343" in the above logs mean? 2.) Recently we experienced an issue with our dynomite setup. We are running 3 racks 1 dc dynomite cluster with RF = 3. One of the 3 dynomite.yml files looks like:-

dyn_o_mite:
  datacenter: dc1
  rack: dc1-rack1
  dyn_listen: 0.0.0.0:8101
  dyn_seeds:
  - 10.150.114.157:8101:dc1-rack2:dc1:0
  - 10.150.114.158:8101:dc1-rack3:dc1:0
  listen: 0.0.0.0:8102
  client_connections: 1500
  servers:
  - 127.0.0.1:6379:1
  tokens: '0'
  pem_key_file: /havail/dynomite/conf/dynomite.pem
  data_store: 0
  read_consistency : DC_QUORUM
  write_consistency : DC_QUORUM

We had some issue with our system recently due to which a client started bombarding dynomite. As you can see above, max client connections we had set is 1500 and the client requests exceeded this threshold. Dynomite was no longer accepting any redis Commands after this happened. For example on running redis-cli -p 8102 on dynomite box we were getting, "Could not connect to redis at 127.0.0.1:8102. Connection refused". I am attaching a log which helped us debug this. What does the code proxy_accept: 173 mean ?

dynomite logs

To resolve for this we are going ahead and increasing client_connections to 10000.

Question :- Is it possible to configure dynomite such that on exceeding max client requests, it stops accepting new connections but keeps working, instead of completely stopping to work?