Heap memory increased in network thread

[twoeths]

Describe the bug

Memory kept increasing and then got stable. So it's similar to a leak but it did not keep increasing forever

This does not happen on any holesky nodes, and does not happen on test mainnet node.

Expected behavior

Heap memory should be stable

Steps to reproduce

No response

Additional context

No response

Operating system

Linux

Lodestar version or commit hash

v1.17.0

[twoeths]

network thread gc time keeps increasing along with heap memory

gc happens randomly and if it runs right at block proposal time (when network thread need to work hard to serialize big block message) then it may cause a delay in #6596

[twoeths]

it looks more like a memory leak with 100 peers in @nflaig node https://github.com/ChainSafe/lodestar/pull/6556#issuecomment-2017692440

[nflaig]

it looks more like a memory leak with 100 peers in @nflaig node #6556 (comment)

yeah, it didn't look like it was stopping to increase on my node, still collecting data with 50 peers but will switch to 100 peers again and take some heap snapshots

[twoeths]

link to the heap snapshot https://github.com/ChainSafe/lodestar/pull/6556#issuecomment-2043250725

[twoeths]

tracked in this libp2p issue

[twoeths]

need to try @libp2p/tcp v9.0.22 which is a candidate for this issue, see https://github.com/libp2p/js-libp2p/pull/2487

right now I get this error updating it

@lodestar/beacon-node: src/network/libp2p/index.ts:76:7 - error TS2322: Type '(components: TCPComponents) => Transport' is not assignable to type '(components: Components) => Transport'.
@lodestar/beacon-node:   Property '[transportSymbol]' is missing in type 'import("/Users/tuyennguyen/Projects/workshop/lodestar/packages/beacon-node/node_modules/@libp2p/interface/dist/src/transport/index").Transport' but required in type 'import("/Users/tuyennguyen/Projects/workshop/lodestar/node_modules/@libp2p/interface/dist/src/transport/index").Transport'.
@lodestar/beacon-node:  76       tcp({
@lodestar/beacon-node:           ~~~~~
@lodestar/beacon-node:  77         maxConnections: networkOpts.maxPeers,
@lodestar/beacon-node:     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
@lodestar/beacon-node: ... 
@lodestar/beacon-node:  85         },
@lodestar/beacon-node:     ~~~~~~~~~~
@lodestar/beacon-node:  86       }),
@lodestar/beacon-node:     ~~~~~~~~
@lodestar/beacon-node:   ../../node_modules/@libp2p/interface/dist/src/transport/index.d.ts:53:5
@lodestar/beacon-node:     53     [transportSymbol]: true;
@lodestar/beacon-node:            ~~~~~~~~~~~~~~~~~
@lodestar/beacon-node:     '[transportSymbol]' is declared here.
@lodestar/beacon-node: Found 1 error in src/network/libp2p/index.ts:76

@wemeetagain could you help with this? 🙏