search

etolabo / kumofs

kumofs is a scalable and highly available distributed key-value store.
http://kumofs.sourceforge.net/
Other
301 stars 16 forks source link

kumofs-gateway: potential memory leaks #4

Closed astro closed 14 years ago

astro commented 14 years ago

We need to guard our kumofs-gateway processes with ulimit -v, they just keep growing openendedly.

Here's some trailing output from valgrind:

==9187== 16,232 bytes in 1 blocks are possibly lost in loss record 108 of 158
==9187==    at 0x4C2263F: operator new[](unsigned long) (vg_replace_malloc.c:264)
==9187==    by 0x43F61C: mp::wavy::core::impl::impl() (wavy_core.cc:46)
==9187==    by 0x43F965: mp::wavy::core::core() (wavy_core.cc:34)
==9187==    by 0x424F62: T.765 (singleton.h:153)
==9187==    by 0x4250A8: kumo::wavy_server::wavy_server() (wavy_server.cc:26)
==9187==    by 0x40DF7B: kumo::gateway::framework::framework(arg_t const&) (rpc_server.h:50)
==9187==    by 0x40BE58: main (init.h:67)
==9187== 
==9187== 40,720 bytes in 5,049 blocks are possibly lost in loss record 126 of 158
==9187==    at 0x4C229C7: operator new(unsigned long) (vg_replace_malloc.c:220)
==9187==    by 0x42C040: std::vector >::_M_insert_aux(__gnu_cxx::__normal_iterator > >, void* const&) (new_allocator.h:89)
==9187==    by 0x42A1A0: kumo::(anonymous namespace)::handler::request_getx(memproto_header_*, char const*, unsigned short) (stl_vector.h:741)
==9187==    by 0x43285E: memproto_dispatch (memproto.c:148)
==9187==    by 0x427D9C: kumo::(anonymous namespace)::handler::read_event() (memcache_binary.cc:502)
==9187==    by 0x440D1E: mp::wavy::core::impl::operator()() (wavy_core.cc:197)
==9187==    by 0x4421A2: void* mp::pthread_thread::trampoline(void*) (pthread_impl.h:79)
==9187==    by 0x5A7D739: start_thread (pthread_create.c:300)
==9187==    by 0x650669C: clone (clone.S:112)
==9187== 
==9187== 65,536 bytes in 2 blocks are possibly lost in loss record 133 of 158
==9187==    at 0x4C221A7: malloc (vg_replace_malloc.c:195)
==9187==    by 0x42BA1F: kumo::(anonymous namespace)::accepted(int, int) (stream_buffer_impl.h:105)
==9187==    by 0x440F8C: mp::wavy::core::impl::operator()() (functional:2024)
==9187==    by 0x4421A2: void* mp::pthread_thread::trampoline(void*) (pthread_impl.h:79)
==9187==    by 0x5A7D739: start_thread (pthread_create.c:300)
==9187==    by 0x650669C: clone (clone.S:112)
==9187== 
==9187== 89,240 bytes in 1 blocks are possibly lost in loss record 135 of 158
==9187==    at 0x4C2263F: operator new[](unsigned long) (vg_replace_malloc.c:264)
==9187==    by 0x445A38: mp::wavy::output::impl::impl() (wavy_output.cc:152)
==9187==    by 0x445CD5: mp::wavy::output::output() (wavy_output.cc:143)
==9187==    by 0x424F7E: T.765 (singleton.h:154)
==9187==    by 0x4250A8: kumo::wavy_server::wavy_server() (wavy_server.cc:26)
==9187==    by 0x40DF7B: kumo::gateway::framework::framework(arg_t const&) (rpc_server.h:50)
==9187==    by 0x40BE58: main (init.h:67)
==9187== 
==9187== 114,800 bytes in 14 blocks are possibly lost in loss record 137 of 158
==9187==    at 0x4C221A7: malloc (vg_replace_malloc.c:195)
==9187==    by 0x50B9660: msgpack_zone_init (in /home/kumo/kumofs/lib/libmsgpackc.so.2.0.0)
==9187==    by 0x429E83: kumo::(anonymous namespace)::handler::request_getx(memproto_header_*, char const*, unsigned short) (zone.hpp:112)
==9187==    by 0x43285E: memproto_dispatch (memproto.c:148)
==9187==    by 0x427D9C: kumo::(anonymous namespace)::handler::read_event() (memcache_binary.cc:502)
==9187==    by 0x440D1E: mp::wavy::core::impl::operator()() (wavy_core.cc:197)
==9187==    by 0x4421A2: void* mp::pthread_thread::trampoline(void*) (pthread_impl.h:79)
==9187==    by 0x5A7D739: start_thread (pthread_create.c:300)
==9187==    by 0x650669C: clone (clone.S:112)
==9187== 
==9187== 147,456 bytes in 18 blocks are possibly lost in loss record 140 of 158
==9187==    at 0x4C221A7: malloc (vg_replace_malloc.c:195)
==9187==    by 0x427C57: kumo::(anonymous namespace)::handler::read_event() (stream_buffer_impl.h:198)
==9187==    by 0x440D1E: mp::wavy::core::impl::operator()() (wavy_core.cc:197)
==9187==    by 0x4421A2: void* mp::pthread_thread::trampoline(void*) (pthread_impl.h:79)
==9187==    by 0x5A7D739: start_thread (pthread_create.c:300)
==9187==    by 0x650669C: clone (clone.S:112)
==9187== 
==9187== 161,760 bytes in 337 blocks are possibly lost in loss record 141 of 158
==9187==    at 0x4C229C7: operator new(unsigned long) (vg_replace_malloc.c:220)
==9187==    by 0x42856F: kumo::(anonymous namespace)::handler::response_queue::push_entry(kumo::(anonymous namespace)::handler::entry*) (new_allocator.h:89)
==9187==    by 0x42A122: kumo::(anonymous namespace)::handler::request_getx(memproto_header_*, char const*, unsigned short) (memcache_binary.cc:559)
==9187==    by 0x43285E: memproto_dispatch (memproto.c:148)
==9187==    by 0x427D9C: kumo::(anonymous namespace)::handler::read_event() (memcache_binary.cc:502)
==9187==    by 0x440D1E: mp::wavy::core::impl::operator()() (wavy_core.cc:197)
==9187==    by 0x4421A2: void* mp::pthread_thread::trampoline(void*) (pthread_impl.h:79)
==9187==    by 0x5A7D739: start_thread (pthread_create.c:300)
==9187==    by 0x650669C: clone (clone.S:112)
==9187== 
==9187== 237,800 bytes in 29 blocks are possibly lost in loss record 146 of 158
==9187==    at 0x4C221A7: malloc (vg_replace_malloc.c:195)
==9187==    by 0x50B9660: msgpack_zone_init (in /home/kumo/kumofs/lib/libmsgpackc.so.2.0.0)
==9187==    by 0x43A241: rpc::(anonymous namespace)::callback_entry::callback_submit_real(std::tr1::function, msgpack::object, msgpack::object, std::auto_ptr)>, std::tr1::shared_ptr, msgpack::object, msgpack::object, std::tr1::shared_ptr) (zone.hpp:112)
==9187==    by 0x43DCED: std::tr1::_Function_handler, msgpack::object, msgpack::object, std::auto_ptr)>, std::tr1::shared_ptr, msgpack::object, msgpack::object, std::tr1::shared_ptr))(std::tr1::function, msgpack::object, msgpack::object, std::auto_ptr)>, std::tr1::shared_ptr, msgpack::object, msgpack::object, std::tr1::shared_ptr)> >::_M_invoke(std::tr1::_Any_data const&) (functional:1137)
==9187==    by 0x440F8C: mp::wavy::core::impl::operator()() (functional:2024)
==9187==    by 0x4421A2: void* mp::pthread_thread::trampoline(void*) (pthread_impl.h:79)
==9187==    by 0x5A7D739: start_thread (pthread_create.c:300)
==9187==    by 0x650669C: clone (clone.S:112)
==9187== 
==9187== 282,744 bytes in 5,049 blocks are possibly lost in loss record 147 of 158
==9187==    at 0x4C229C7: operator new(unsigned long) (vg_replace_malloc.c:220)
==9187==    by 0x41DB0C: std::tr1::_Function_base::_Base_manager, msgpack::object, msgpack::object, std::auto_ptr, rpc::retry*, void (*)(void*, kumo::gate::res_get&, std::auto_ptr), void*)> ()(kumo::gateway::mod_store_t*, std::tr1::_Placeholder<1>, std::tr1::_Placeholder<2>, std::tr1::_Placeholder<3>, std::tr1::_Placeholder<4>, rpc::retry*, void (*)(void*, kumo::gate::res_get&, std::auto_ptr), void*)> >::_M_manager(std::tr1::_Any_data&, std::tr1::_Any_data const&, std::tr1::_Manager_operation) (functional:1507)
==9187==    by 0x41D861: std::tr1::function, msgpack::object, msgpack::object, std::auto_ptr)>::function(std::tr1::function, msgpack::object, msgpack::object, std::auto_ptr)> const&) (functional:1989)
==9187==    by 0x41D1B7: kumo::gateway::mod_store_t::Get(kumo::gate::req_get&) (functional:1841)
==9187==    by 0x42A12D: kumo::(anonymous namespace)::handler::request_getx(memproto_header_*, char const*, unsigned short) (memcache_binary.cc:560)
==9187==    by 0x43285E: memproto_dispatch (memproto.c:148)
==9187==    by 0x427D9C: kumo::(anonymous namespace)::handler::read_event() (memcache_binary.cc:502)
==9187==    by 0x440D1E: mp::wavy::core::impl::operator()() (wavy_core.cc:197)
==9187==    by 0x4421A2: void* mp::pthread_thread::trampoline(void*) (pthread_impl.h:79)
==9187==    by 0x5A7D739: start_thread (pthread_create.c:300)
==9187==    by 0x650669C: clone (clone.S:112)
==9187== 
==9187== LEAK SUMMARY:
==9187==    definitely lost: 0 bytes in 0 blocks
==9187==    indirectly lost: 0 bytes in 0 blocks
==9187==      possibly lost: 1,207,272 bytes in 10,748 blocks
==9187==    still reachable: 225,711,931 bytes in 123,928 blocks
==9187==         suppressed: 0 bytes in 0 blocks
==9187== Reachable blocks (those to which a pointer was found) are not shown.
==9187== To see them, rerun with: --leak-check=full --show-reachable=yes
==9187== 
==9187== For counts of detected and suppressed errors, rerun with: -v
==9187== ERROR SUMMARY: 39 errors from 39 contexts (suppressed: 4 from 4)
frsyuki commented 14 years ago

I've fixed a memory leak problem on kumo-gateway and released kumofs-0.4.3. It was a circular reference reference problem. It had occurred when a connection is closed before a response is returned. This release may fix the problem. Please check it out.

astro commented 14 years ago

Thank you.

Unfortunately, the process still grows with 0.4.3. It might be caused by other circumstances (which I'll report separately).

frsyuki commented 14 years ago

Short story: Please try to use jemalloc. I've released kumofs-0.4.4 and it accepts --with-jemalloc option on ./configure to link libjemalloc. The detailed information about jemalloc is available at http://www.canonware.com/jemalloc/. Its source package is available at http://www.canonware.com/download/jemalloc/ (I tested with jemalloc-1.0.2.tar.bz2).

Long story: I found that memory usage of kumo-gateway grows when gateway retries Get requests. It occurs when kumo-server crashes or some network errors happen. And I also found that sometimes the usage decreases after the retrying. I guessed that the reason why memory usage grows is that retrying accumulates request command on memory, and it becomes large amount on high-load situation (there are many clients and many retrying commands on memory), but it is freed after retrying. The problem is probably that the memory allocator won't return the freed memory space to the operating system. I found that the memory usage decreases dramatically when I use jemalloc (LD_PRELOAD=/usr/local/lib/jemalloc.so kumo-gateway). Now kumofs-0.4.4 has --with-jemalloc option on configure. It links libjemalloc on compile-time. I hope it solves the problem.

astro commented 14 years ago

I'm now running 0.4.4 built with jemalloc. I also modified my branch of erlmc to accept responses ordered arbitrarily, and connected it to kumo-gateway's cloudy port.

Unfortunately, kumo-gateway still grows over time. Do you need another valgrind run? Maybe I can collect other statistics than just VmRSS?

Our load is fairly low, about 100..200 req/s and the kumo-servers are not failing.

Is there a recommended value size for TokyoCabinet? We're having about 200 KB per key.

frsyuki commented 14 years ago

Now I'm sure it is resolved! Please check out kumofs-0.4.5. I've confirmed the memory leak problem on my machine and fixed the problem. It had occurred when the value size is bigger than 32KB.

I also modified my branch of erlmc to accept responses ordered arbitrarily, and connected it to kumo-gateway's cloudy port. Great! The original binary protocol is too bad...

Is there a recommended value size for TokyoCabinet? It is a difficult problem. The author of Tokyo Cabinet (Mikio Hirabayashi, @mikio1978 on Twitter) says that it is just a wrapper library that wraps mmap or read/write systemcall, you can use any size as you like. For example, on "mixi" (mixi is one of the most popular SNSs in Japan. It's mikio's place of work), the size of value stored on TC is 1KB (see http://1978th.net/tokyocabinet/tokyoproducts.pdf). I've ever stored 10MB values into TC. It has also worked well. On "Ficia" (a photo sharing service in Japan), size of a value stored on kumofs is only 100 or 200 bytes. Total amount of values stored on one server is more important. Because kumofs uses Tokyo Cabinet's Hash database, it will cause many HDD seeks when the size exceeds memory size. There are some hints: 1) 'set(k_combined,v1+v2) + get(k_combined)' is faster and sclable than 'set(k1,v1) + set(k1,v2) + get_multi(k1,k2)'. As two different keys are likely to be stored on different servers, 'get_multi(k1,k2)' will be separated into two queries. 2) 'set(k_combined,v1+v2) + get(k_combined)' generates larger network trafic than 'set(k1) + get(k1)'. Combining two values into one big value is sometime not efficient depending on the usage.

astro commented 14 years ago

Thank you, I think that did it for us.

Again, I think this info should be on the KumoFS homepage. Tell me if I can help out with that.