search

networkservicemesh / sdk

Apache License 2.0
35 stars 35 forks source link

Span logger leaks memory in infinite watching `server.Find` case #1085

Open Bolodya1997 opened 2 years ago

Bolodya1997 commented 2 years ago

Current Behavior

In infinite watching server.Find case span logger just collects logs and doesn't push them. It leads to unlimited memory growth in registry server.

Issue in code

In infinite case defer never happens: https://github.com/networkservicemesh/sdk/blob/c7b15bacaae52e931b097ecb16a45eb584124261/pkg/registry/core/trace/nse_registry.go#L87-L88

pprof output

registry-k8s-5b77d84b85-lgxn4.pb.gz

Possible solution

Instead of having single span in such case we can handle such hierarchy:

[root span] - empty
[child span 1] .. [child span n] - with data

So all of [child spans] will refer to the single [root span] and will be pushed with some timeout or with some amount of logs collected. [root span] will always be empty so there will be no memory leak.

denis-tingaikin commented 2 years ago

Could you please attach something from logs, build links, pictures, source code links from your research?

Bolodya1997 commented 2 years ago

Could you please attach something from logs, build links, pictures, source code links from your research?

Added link to code and pprof output.

Mixaster995 commented 2 years ago

Tried to investigate this problem - can't reproduce leaks. Probably requires really long-time running setup or some special tweaks imitating leaks. Since problem don't look critical at this moment, so i postponed investigation Prepared draft PR with possible solution that should fix this problem - https://github.com/networkservicemesh/sdk/pull/1108