p2p: mark num_peers in metrics as inbound/outbound

[Zygimantass]

Feature Request

Summary

Add metrics tendermint_p2p_inbound_peers and tendermint_p2p_outbound_peers that would signify which side initiated the p2p connection.

Problem Definition

This metric would help fine-tune the peer limits, considering the ratio of inbound : outbound peers that a node has.

Proposal

Add two metrics that would be incremented / decremented in p2p/switch.go based on whether peer.IsOutbound() returns true or false. An alternative solution could be adding a direction label to the existing tendermint_p2p_peers gauge, but that would break existing monitoring solutions.

[cason]

Observe that this information, number of outbound and inbound peers, is periodically (every 30s) printed to the log (Info level):

func (r *Reactor) ensurePeers() {
        var (
                out, in, dial = r.Switch.NumPeers()
                numToDial     = r.Switch.MaxNumOutboundPeers() - (out + dial)
        )
        r.Logger.Info(
                "Ensure peers",
                "numOutPeers", out,
                "numInPeers", in,
                "numDialing", dial,
                "numToDial", numToDial,
        )

https://github.com/cometbft/cometbft/blob/main/p2p/pex/pex_reactor.go#L457

[cason]

Having said so, I agree with the proposal, I would also consider having a metric for dialing peers as well.