search

Netflix / dyno

Java client for Dynomite
https://github.com/Netflix/dynomite
Apache License 2.0
186 stars 94 forks source link

Support for Dynomite running over memcached #191

Closed ADenisUA closed 6 years ago

ADenisUA commented 6 years ago

Hi, Could someone please explain why do we need a specific dyno client for Dynomite running over memcached? I thought Dynomite has it's own api, so Dyno client talks directly to Dynomite (so it's should be back-end specific). I would appreciate any advices on how to use DYno client with Dynomite running over memcached. Thanks

ipapapa commented 6 years ago

Dyno does not have its own API. Dyno is a wrapper around other clients, like Jedis, Redisson, and Memcache. We mainly exercise the Jedis at Netflix, so we have not been paying much attention to the Memcache. However, the structure is adaptable and if somebody wants to use it, then we would love to assist.

ADenisUA commented 6 years ago

OK. Several other questions related to memcached support: 1) When I start dynomite (I'm using code from v0.6 branch) it often fails (50% of the time) with: [2017-11-20 12:57:52.619] conf_push_scalar:541 push 'memcached-4:8101:memcached-rack2:dc2:4294967294' [2017-11-20 12:57:52.620] conf_handler:1291 conf handler on dyn_seeds: memcached-4:8101:memcached-rack2:dc2:4294967294 [2017-11-20 12:57:52.622] _dn_alloc:259 malloc(24) at 0x1c53e30 @ ../dyn_array.c:64 [2017-11-20 12:57:52.623] conf_server_init:134 init conf server 0x1c538c0 [2017-11-20 12:57:52.660] dn_resolve_inet:520 address resolution of node 'memcached-4' service '8101' failed: Name or service not known [2017-11-20 12:57:52.660] _dn_free:305 free(0x1c53ed0) @ dyn_string.c:59 [2017-11-20 12:57:52.660] conf_handler:1302 conf: directive "dyn_seeds" has an invalid value [2017-11-20 12:57:52.660] conf_pop_scalar:564 pop 'memcached-4:8101:memcached-rack2:dc2:4294967294' [2017-11-20 12:57:52.660] _dn_free:305 free(0x1c53e50) @ dyn_string.c:59 dynomite: configuration file '/etc/dynomite/conf/memcached-3.yml' syntax is invalid [2017-11-20 12:57:52.660] conf_pop_scalar:564 pop 'dyn_seeds' [2017-11-20 12:57:52.660] _dn_free:305 free(0x1c53810) @ dyn_string.c:59 [2017-11-20 12:57:52.660] conf_pop_scalar:564 pop 'dyn_o_mite' [2017-11-20 12:57:52.660] _dn_free:305 free(0x1c53790) @ dyn_string.c:59 [2017-11-20 12:57:52.660] _dn_free:305 free(0x1c41b70) @ ../dyn_array.c:82 [2017-11-20 12:57:52.660] _dn_free:305 free(0x1c53710) @ dyn_string.c:59 [2017-11-20 12:57:52.660] dn_stacktrace:326 [0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7fa2eaa2c390] ??:0 [2017-11-20 12:57:52.664] dn_stacktrace:326 [1] dynomite(string_deinit+0x3) [0x429953] /dynomite/src/dyn_string.c:58 [2017-11-20 12:57:52.669] dn_stacktrace:326 [2] dynomite(conf_destroy+0x55) [0x410385] /dynomite/src/dyn_conf.c:146 [2017-11-20 12:57:52.674] dn_stacktrace:326 [3] dynomite(conf_create+0x170) [0x410680] /dynomite/src/dyn_conf.c:2233 [2017-11-20 12:57:52.681] dn_stacktrace:326 [4] dynomite(core_start+0x80) [0x415350] /dynomite/src/dyn_core.c:281 [2017-11-20 12:57:52.685] dn_stacktrace:326 [5] dynomite(main+0x64f) [0x40cdff] /dynomite/src/dynomite.c:574 [2017-11-20 12:57:52.690] dn_stacktrace:326 [6] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7fa2ea368830] ??:0 [2017-11-20 12:57:52.693] dn_stacktrace:326 [7] dynomite(_start+0x29) [0x40d2a9] ??:? [2017-11-20 12:57:52.698] signal_handler:132 signal 11 (SIGSEGV) received, core dumping

meaning it can't discover another dynomite node. Also, if one node goes down, and other node tries to replicate - it also goes down. I'm running several dynomite nodes in docker, so there is definitely some race condition between nodes, but guess dynomite should some how gracefully handle such cases (try to reconnect several times after some timeout?). What would you suggest to do to avoid this problem?

Here is my config (for reference):

` dyn_o_mite: datacenter: dc2 dyn_listen: 0.0.0.0:8101 dyn_port: 8101 dyn_seed_provider: simple_provider dyn_seeds:

` dyn_o_mite: datacenter: dc2 dyn_listen: 0.0.0.0:8101 dyn_port: 8101 dyn_seed_provider: simple_provider dyn_seeds:

2)When I'm trying to use Spy Memcached client and save data - it works fine. But when I'm trying to read I receive:

[2017-11-20 12:44:49.561] server_rsp_forward:834 <CONN_CLIENT 0x528b660 11 from '172.18.0.1:54290'> <REQ 0x556fc00 4124:0::1 REQ_MC_GET, len:14> RECEIVED <RSP 0x5b1ce00 4126:0 RSP_MC_END len:5> [2017-11-20 12:44:49.561] stats_server_to_metric:1800 metric 'read_responses' for server [2017-11-20 12:44:49.561] _stats_server_incr:1816 incr field 'read_responses' to 1 [2017-11-20 12:44:49.561] stats_server_to_metric:1800 metric 'read_response_bytes' for server [2017-11-20 12:44:49.561] _stats_server_incr_by:1846 incr by field 'read_response_bytes' to 5 [2017-11-20 12:44:49.561] msg_local_one_rsp_handler:1005 <REQ 0x556fc00 4124:0::1 REQ_MC_GET, len:14> SELECTED <RSP 0x5b1ce00 4126:0 RSP_MC_END len:5> [2017-11-20 12:44:49.564] dn_stacktrace:326 [0] /lib/x86_64-linux-gnu/libpthread.so.0(+0x11390) [0x7f07351b4390] ??:0 [2017-11-20 12:44:49.578] dn_stacktrace:326 [1] dynomite(memcache_post_coalesce+0x114) [0x4331a4] /dynomite/src/proto/dyn_memcache.c:1556 [2017-11-20 12:44:49.596] dn_stacktrace:326 [2] dynomite(req_done+0x174) [0x420994] /dynomite/src/dyn_request.c:148 [2017-11-20 12:44:49.612] dn_stacktrace:326 [3] dynomite() [0x40db52] /dynomite/src/dyn_client.c:282 [2017-11-20 12:44:49.622] dn_stacktrace:326 [4] dynomite(msg_recv+0x575) [0x420385] /dynomite/src/dyn_message.c:890 [2017-11-20 12:44:49.632] dn_stacktrace:326 [5] dynomite(core_core+0xff) [0x414aff] /dynomite/src/dyn_core.c:386 [2017-11-20 12:44:49.641] dn_stacktrace:326 [6] dynomite(event_wait+0xeb) [0x43a90b] /dynomite/src/event/dyn_epoll.c:259 (discriminator 2) [2017-11-20 12:44:49.713] dn_stacktrace:326 [7] dynomite(core_loop+0x166) [0x415996] /dynomite/src/dyn_core.c:650 (discriminator 4) [2017-11-20 12:44:49.732] dn_stacktrace:326 [8] dynomite(main+0x671) [0x40ce21] /dynomite/src/dynomite.c:585 [2017-11-20 12:44:49.743] dn_stacktrace:326 [9] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf0) [0x7f0734af0830] ??:0 [2017-11-20 12:44:49.755] dn_stacktrace:326 [10] dynomite(_start+0x29) [0x40d2a9] ??:? [2017-11-20 12:44:49.774] signal_handler:132 signal 11 (SIGSEGV) received, core dumping /dynomite/startup.sh: line 6: 9 Segmentation fault (core dumped) dynomite --conf-file=/etc/dynomite/conf/${DYN_CONFIG_FILE} --verbosity=11

is this some known problem? I saw an old branch (memcachedFix) - can you merge the fix to the main branch? what was the fix?

Thanks in advance!

ipapapa commented 6 years ago

I suggest you open an issue on Dynomite as this is not a Dyno issue. The fixes can be found here: https://github.com/Netflix/dynomite/commit/5f6cef310b495f48baaa179bae77ad3a74f141ce They are mostly taken out of the corresponding code in Twemproxy and slightly adapted for Dynomite: https://github.com/twitter/twemproxy/blob/master/src/proto/nc_memcache.c Generally speaking adding a protocol is a relatively easy effort, so feel free to fix the memcache portion and file a PR.

ADenisUA commented 6 years ago

Got it. Thanks for quick feedback