Max peers config has no effect

haikoschol commented 3 weeks ago

Describe the bug

The default value for the --max-peers CLI flag is 50. On a long-running bare metal Westend node, Grafana shows 479 under "Network total connected peers" and 957 under "Network total node connections". Either the configuration setting has no effect or the metrics are reported incorrectly.

Expected Behavior

The node should maintain no more than the maximum number of peer connections configured.

Current Behavior

The number of peers/connections keeps rising for a while until it stabilizes on a value much higher than the configured maximum. The number it stabilizes on seems to be correlated to the CPU/memory resources of the node.

To Reproduce

Steps to reproduce the behavior:

run Gossamer without passing --max-peers to use the default of 50
wait a while
check the number of peers with curl -s localhost:9876/metrics | grep peer_count_total

Specification

go version: 1.23.2
gossamer version: development
gossamer commit hash: ce5200c3f19d84e77598ea9b19b5a49053e681de
operating system: Linux

haikoschol commented 2 weeks ago

This might be a symptom of #4293.

haikoschol commented 1 week ago

This might be a symptom of #4293.

It's not:

$ grep peers config.toml
# Minimum number of peers to connect to
min-peers = 5
# Maximum number of peers to connect to
max-peers = 50
# Comma separated list of peers to always keep connected to
persistent-peers = ""

This was on a machine that reports 316 "Network total connected peers" and 735 "Network total node connections".

However, those numbers are apparently wrong as well.

Number of all TCP connections of the gossamer process:

$ lsof -n -P -iTCP  | grep gossamer | wc -l
201

Number of all TCP connections in the ESTABLISHED state:

$ lsof -n -P -iTCP -sTCP:ESTABLISHED | grep gossamer | wc -l
195

So it seems that both the prometheus metrics are wrong and the config value is not honored by the network service.

ChainSafe / gossamer