search

envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.71k stars 4.75k forks source link

Detect Memory leaks in Envoy #35741

Open dinesh-murugiah opened 3 weeks ago

dinesh-murugiah commented 3 weeks ago

Hi

Title How to Detect Memory leaks in Envoy

Description i am trying to expand feature of the Envoy redis proxy to support pubsub and blocking and other admin commands thereby it will be closer to features when compared to enterprise solutions During this development i am running in to memory leaks , i am yet to write test cases for all the code intrduced is there a better/simpler way to detect memory leaks .

i tried to use it with valgrind , but valgrid seems to report problems in many other libraries linked and i am not able to make it provide reliable output

i tried tcmalloc by enable --config tcmalloc=gperftools , when running the heap profile generated with pprof i am not able to get symbols properly

I have enabled asan in the build statically using following option

bazel build --config=clang --config=libc++ -c dbg --copt=-O0 --strip=never --copt=-fsanitize=address --copt=-fno-common --copt=-fno-sanitize-recover=all --copt=-g3 --linkopt=-fsanitize=address --copt=-DADDRESS_SANITIZER=1 --copt=-fno-sanitize=undefined --linkopt=-fno-sanitize=undefined --define tcmalloc=disabled

this also does not seems to provide any output for a know leak(running without fix) which i fixed in later builds

request some guidance in better way to detect memory leak and right configs to enable during builld and steps to collect data

adisuissa commented 3 weeks ago

cc @nezdolik @KBaichoo @jmarantz who may have some experience in tracking memory leaks

jmarantz commented 3 weeks ago

Usually we get pretty good memory-leak traces from running unit tests with a default build (fastbuild).

How was it that you discovered you had a memory leak in the first place?

KBaichoo commented 3 weeks ago

+1 on how you found out you had memory leaks.

See also https://www.envoyproxy.io/docs/envoy/latest/operations/admin.html#get--memory for getting some memory stats -- presumably from your leak a graph of these numbers should just be going up, and https://www.envoyproxy.io/docs/envoy/latest/api-v3/config/bootstrap/v3/bootstrap.proto#config-bootstrap-v3-memoryallocatormanager for configuring the memory allocator manager which might be useful for your usecase.

Note that TCMalloc doesn't give memory back to the OS by default if you're using some OS level metric for monitoring the memory leak in which case the memory allocator manager configuration would be useful.