search

nicolasff / webdis

A Redis HTTP interface with JSON output
https://webd.is
BSD 2-Clause "Simplified" License
2.82k stars 307 forks source link

Ever increasing memory consumption #143

Closed adamdougal closed 2 years ago

adamdougal commented 6 years ago

Hey,

I'm currently running Webdis in a Docker container and deploying it into a Kubernetes cluster however I'm finding that Webdis's memory consumption is ever increasing. I've got no c experience but have run it with Valgrind and I think it's possibly indicating a memory leak:

==1== Invalid read of size 1
==1==    at 0x4C2CBD1: strncasecmp (vg_replace_strmem.c:650)
==1==    by 0x402E76: cmd_is_subscribe (cmd.c:375)
==1==    by 0x4020F2: cmd_free (cmd.c:55)
==1==    by 0x40B7D9: format_send_reply (common.c:123)
==1==    by 0x40A2FC: json_reply (json.c:40)
==1==    by 0x4125BD: __redisRunCallback (async.c:269)
==1==    by 0x412D13: redisProcessCallbacks (async.c:468)
==1==    by 0x412F04: redisAsyncHandleRead (async.c:532)
==1==    by 0x408DDF: redisLibeventReadEvent (libevent.h:45)
==1==    by 0x4E453DB: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9)
==1==    by 0x403352: worker_main (worker.c:158)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1==  Address 0x9825aa0 is 0 bytes inside a block of size 5 free'd
==1==    at 0x4C29E90: free (vg_replace_malloc.c:473)
==1==    by 0x40205D: cmd_free (cmd.c:45)
==1==    by 0x40B7D9: format_send_reply (common.c:123)
==1==    by 0x40A2FC: json_reply (json.c:40)
==1==    by 0x4125BD: __redisRunCallback (async.c:269)
==1==    by 0x412D13: redisProcessCallbacks (async.c:468)
==1==    by 0x412F04: redisAsyncHandleRead (async.c:532)
==1==    by 0x408DDF: redisLibeventReadEvent (libevent.h:45)
==1==    by 0x4E453DB: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9)
==1==    by 0x403352: worker_main (worker.c:158)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1== HEAP SUMMARY:
==1==     in use at exit: 9,015,103 bytes in 1,739,476 blocks
==1==   total heap usage: 181,491,983 allocs, 179,752,507 frees, 16,658,018,432 bytes allocated
==1==
==1== Thread 1:
==1== 50 bytes in 10 blocks are possibly lost in loss record 18 of 52
==1==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==1==    by 0x531B989: strdup (strdup.c:42)
==1==    by 0x402DC5: cmd_select_format (cmd.c:358)
==1==    by 0x40267D: cmd_run (cmd.c:171)
==1==    by 0x4034C9: worker_process_client (worker.c:197)
==1==    by 0x40773E: http_client_on_message_complete (client.c:203)
==1==    by 0x41F0BB: http_parser_execute (http_parser.c:1404)
==1==    by 0x407D1C: http_client_execute (client.c:341)
==1==    by 0x402FB2: worker_can_read (worker.c:61)
==1==    by 0x4E453DB: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9)
==1==    by 0x403352: worker_main (worker.c:158)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1==
==1== 234 bytes in 2 blocks are possibly lost in loss record 28 of 52
==1==    at 0x4C2AF2E: realloc (vg_replace_malloc.c:692)
==1==    by 0x40E425: sdsMakeRoomFor (sds.c:142)
==1==    by 0x40E67E: sdscatlen (sds.c:241)
==1==    by 0x40DD6D: __redisAppendCommand (hiredis.c:910)
==1==    by 0x413384: __redisAsyncCommand (async.c:641)
==1==    by 0x41354A: redisAsyncCommandArgv (async.c:678)
==1==    by 0x402BD6: cmd_send (cmd.c:284)
==1==    by 0x402B60: cmd_run (cmd.c:273)
==1==    by 0x4034C9: worker_process_client (worker.c:197)
==1==    by 0x40773E: http_client_on_message_complete (client.c:203)
==1==    by 0x41F0BB: http_parser_execute (http_parser.c:1404)
==1==    by 0x407D1C: http_client_execute (client.c:341)
==1==
==1== 432 bytes in 48 blocks are possibly lost in loss record 31 of 52
==1==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==1==    by 0x40E1E2: sdsnewlen (sds.c:55)
==1==    by 0x40E27B: sdsempty (sds.c:71)
==1==    by 0x40DB9D: redisBufferWrite (hiredis.c:850)
==1==    by 0x412F70: redisAsyncHandleWrite (async.c:549)
==1==    by 0x408E0D: redisLibeventWriteEvent (libevent.h:51)
==1==    by 0x4E453DB: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9)
==1==    by 0x403352: worker_main (worker.c:158)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1==
==1== 450 bytes in 50 blocks are possibly lost in loss record 32 of 52
==1==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==1==    by 0x40E1E2: sdsnewlen (sds.c:55)
==1==    by 0x40E27B: sdsempty (sds.c:71)
==1==    by 0x40D259: redisContextInit (hiredis.c:601)
==1==    by 0x40D659: redisConnectNonBlock (hiredis.c:701)
==1==    by 0x41224E: redisAsyncConnect (async.c:157)
==1==    by 0x409361: pool_connect (pool.c:129)
==1==    by 0x40329E: worker_pool_connect (worker.c:135)
==1==    by 0x403342: worker_main (worker.c:155)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1==
==1== 450 bytes in 50 blocks are possibly lost in loss record 33 of 52
==1==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==1==    by 0x40E1E2: sdsnewlen (sds.c:55)
==1==    by 0x40E27B: sdsempty (sds.c:71)
==1==    by 0x4145DE: redisReaderCreateWithFunctions (read.c:426)
==1==    by 0x40D214: redisReaderCreate (hiredis.c:589)
==1==    by 0x40D26C: redisContextInit (hiredis.c:602)
==1==    by 0x40D659: redisConnectNonBlock (hiredis.c:701)
==1==    by 0x41224E: redisAsyncConnect (async.c:157)
==1==    by 0x409361: pool_connect (pool.c:129)
==1==    by 0x40329E: worker_pool_connect (worker.c:135)
==1==    by 0x403342: worker_main (worker.c:155)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1==
==1== 1,360 bytes in 5 blocks are possibly lost in loss record 39 of 52
==1==    at 0x4C2AD10: calloc (vg_replace_malloc.c:623)
==1==    by 0x40111C1: allocate_dtv (dl-tls.c:296)
==1==    by 0x40118CD: _dl_allocate_tls (dl-tls.c:460)
==1==    by 0x5085BE7: allocate_stack (allocatestack.c:589)
==1==    by 0x5085BE7: pthread_create@@GLIBC_2.2.5 (pthread_create.c:495)
==1==    by 0x403382: worker_start (worker.c:166)
==1==    by 0x403EF8: server_start (server.c:217)
==1==    by 0x401FAD: main (webdis.c:16)
==1==
==1== 66,096 bytes in 162 blocks are possibly lost in loss record 50 of 52
==1==    at 0x4C2AD10: calloc (vg_replace_malloc.c:623)
==1==    by 0x407772: http_client_new (client.c:212)
==1==    by 0x403C53: server_can_accept (server.c:120)
==1==    by 0x4E453DB: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9)
==1==    by 0x40403A: server_start (server.c:241)
==1==    by 0x401FAD: main (webdis.c:16)
==1==
==1== 87,550 bytes in 50 blocks are possibly lost in loss record 51 of 52
==1==    at 0x4C2AF2E: realloc (vg_replace_malloc.c:692)
==1==    by 0x40E425: sdsMakeRoomFor (sds.c:142)
==1==    by 0x40E67E: sdscatlen (sds.c:241)
==1==    by 0x4147CA: redisReaderFeed (read.c:464)
==1==    by 0x40DA51: redisBufferRead (hiredis.c:814)
==1==    by 0x412EBA: redisAsyncHandleRead (async.c:527)
==1==    by 0x408DDF: redisLibeventReadEvent (libevent.h:45)
==1==    by 0x4E453DB: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9)
==1==    by 0x403352: worker_main (worker.c:158)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1==
==1== 8,690,300 bytes in 1,738,060 blocks are definitely lost in loss record 52 of 52
==1==    at 0x4C28C20: malloc (vg_replace_malloc.c:296)
==1==    by 0x531B989: strdup (strdup.c:42)
==1==    by 0x402DC5: cmd_select_format (cmd.c:358)
==1==    by 0x40267D: cmd_run (cmd.c:171)
==1==    by 0x4034C9: worker_process_client (worker.c:197)
==1==    by 0x40773E: http_client_on_message_complete (client.c:203)
==1==    by 0x41F0BB: http_parser_execute (http_parser.c:1404)
==1==    by 0x407D1C: http_client_execute (client.c:341)
==1==    by 0x402FB2: worker_can_read (worker.c:61)
==1==    by 0x4E453DB: event_base_loop (in /usr/lib/x86_64-linux-gnu/libevent-2.0.so.5.1.9)
==1==    by 0x403352: worker_main (worker.c:158)
==1==    by 0x5085063: start_thread (pthread_create.c:309)
==1==
==1== LEAK SUMMARY:
==1==    definitely lost: 8,690,300 bytes in 1,738,060 blocks
==1==    indirectly lost: 0 bytes in 0 blocks
==1==      possibly lost: 156,622 bytes in 377 blocks
==1==    still reachable: 168,181 bytes in 1,039 blocks
==1==         suppressed: 0 bytes in 0 blocks
==1== Reachable blocks (those to which a pointer was found) are not shown.
==1== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==1==
==1== For counts of detected and suppressed errors, rerun with: -v
==1== ERROR SUMMARY: 7739216 errors from 14 contexts (suppressed: 0 from 0)
sgoveia commented 3 years ago

@nicolasff love this project, can you confirm or deny this issue. Thanks..

nicolasff commented 2 years ago

Hey @adamdougal, sorry I never got back to you on this, and thanks for the Valgrind trace. It was very detailed and pointed straight to the source of the leak, which was a redundant call to strdup.

This memory leak was fixed in 66ef564b13bde66cba4d2e37b35ccba4099e4d91 on May 11, 2021, and has been part of Webdis since 0.1.16 (released on July 12, 2021). The commit details the reason for the memory leak, if you're interested. It happened only with requests that used a query string parameter to set the content-type, e.g. with ?type=text/css.

/cc @sgoveia since you were following this issue.