search

libp2p / js-libp2p

The JavaScript Implementation of libp2p networking stack.
https://libp2p.io
Other
2.27k stars 435 forks source link

Memory leak issue #2477

Open twoeths opened 3 months ago

twoeths commented 3 months ago

Severity:

Medium

Description:

Refer to the profile taken by a lodestar beacon-node here

There are > 15k of objects retained belonging to these classes:

Steps to reproduce the error:

Run a lodestar mainnet node and leave it for a long time, then monitor heap size of network thread

twoeths commented 3 months ago

I found all Sockets have a timeout and they're all destroyed

Screenshot 2024-04-11 at 14 39 33

all connection statuses are also "closed"

Screenshot 2024-04-11 at 14 40 39

also there are socket events attached during the MultiaddrConnection.close() function, especially the "drain" one

Screenshot 2024-04-11 at 14 42 36

it seems to me the MultiaddrConnection.close() was called, but somehow it still keep some resources especially the socket listeners

SgtPooki commented 3 months ago

@tuyennhv slightly off-topic but is there any chance you could share the grafana dashboard json you use over at lodestar so we can use it in ipfs/helia-http-gateway ?

twoeths commented 3 months ago

@SgtPooki here's the libp2p dashboard in lodestar https://github.com/ChainSafe/lodestar/blob/unstable/dashboards/lodestar_libp2p.json

Screenshot 2024-04-11 at 19 54 28 Screenshot 2024-04-11 at 19 54 46
twoeths commented 3 months ago

@achingbrain need to reopen until it's confirmed this is fixed in lodestar, waiting for a new release of transport-tcp to test

achingbrain commented 3 months ago

@tuyennhv did you manage to test the new release?

twoeths commented 3 months ago

@achingbrain we tested v9.0.22 and found it does not resolve the memory leak issue. It also introduced some performance impact that caused node to forward less gossip messages to mesh peers, I'm not sure if it links to libp2p-tcp specifically or other libs during the upgrade to have to rollback in our release

see https://github.com/ChainSafe/lodestar/pull/6701/files

twoeths commented 3 months ago

I think we need to upgrade all libp2p dependencies to the latest versions and evaluate again cc @wemeetagain need to take a new profile with leaked memory for libp2p-tcp >= v9.0.22 to move this forward

achingbrain commented 3 months ago

I think we need to upgrade all libp2p dependencies to the latest versions

Please do, there was a small leak fixed by @libp2p/utils@5.3.1 so that's a good idea in general.

twoeths commented 3 months ago

in progress PR by @wemeetagain : https://github.com/ChainSafe/lodestar/pull/6711