near / nearcore

Reference client for NEAR Protocol
https://near.org
GNU General Public License v3.0
2.31k stars 619 forks source link

Do not record the CONGESTION_RECEIPT_FORWARDING_UNUSED_CAPACITY_GAS metric to the same shard #11699

Open jancionear opened 3 months ago

jancionear commented 3 months ago

The CONGESTION_RECEIPT_FORWARDING_UNUSED_CAPACITY_GAS metric describes how much gas we could've sent to another shard:

static CONGESTION_RECEIPT_FORWARDING_UNUSED_CAPACITY_GAS: Lazy<IntGaugeVec> = Lazy::new(|| {
    try_create_int_gauge_vec(
        "near_congestion_receipt_forwarding_unused_capacity_gas",
        "How much additional gas could have been forwarded in the same chunk from one shard to another. An indicator for congestion backpressure.",
        &["sender_shard_id", "receiver_shard_id"],
    )
    .unwrap()
});

This makes sense for sending between different shards (e.g 0 -> 1), but gas capacity is unlimited when sending between the same shard (e.g 0 -> 0), because of which the metric has extremely large values when sender_shard_id = receiver_shard_id. When one timeseries has large values, Grafana adjusts the visualisation to fit them, which means that the y axis has very large values, which causes the real, non-infinite values to be almost invisible at the bottom.

I couldn't find a way to easily filter out timeseries where sender_shard_id == receiver_shard_id, this the most "pretty" way I found:

avg by (sender_shard_id, receiver_shard_id) (near_congestion_receipt_forwarding_unused_capacity_gas{chain_id=~"$chain_id", node_id=~"$node_id", sender_shard_id=~"$shard_id"} / 1e15 unless {sender_shard_id = "0", receiver_shard_id = "0"} unless {sender_shard_id = "1", receiver_shard_id = "1"} unless {sender_shard_id = "2", receiver_shard_id = "2"} unless {sender_shard_id = "3", receiver_shard_id = "3"} unless {sender_shard_id = "4", receiver_shard_id = "4"} unless {sender_shard_id = "5", receiver_shard_id = "5"})

Let's not set this metric when sender and receiver are the same, it'll save a lot of headache on the Grafana side.

/cc @jakmeier