search

uber / tchannel

network multiplexing and framing protocol for RPC
MIT License
1.15k stars 129 forks source link

Peer leaks and CPU spikes #1328

Closed ShanniLi closed 8 years ago

ShanniLi commented 8 years ago

This is to track the issue of peer leaks and CPU spikes in some Hyperbahn nodes.

During the on-call, I observed a spike of CPU and connection FDs in some Hyperbahn nodes. The memory dumps suggested that we are leaking TChannelPeer (130k objects).

The suspicion is that it is related to the aggressive outgoing peer selection logic. However, the amount of the leaked connections doesn't add up to the leaked peers. One reason may be when a non-ephemeral connection is closed, the peer never gets deleted ...

I would consider this issue open until we have more clarity on the cause.

cc: @jcorbin @Raynos @kriskowal

ShanniLi commented 8 years ago

The CPU spikes are caused by the multi-tenant issue in Hyperbahn. Closing this issue since there is another task tracking it.

Raynos commented 8 years ago

There is no "multi-tenant" issue -.-