Implement rate limits for network messages

[Trisfald]

Issue to track the implementation of rate limits for network messages received by nodes.

Motivation

Prevent abuse and attacks carried by malicious actors. Technically, as of today, a bad actor could attempt to establish a connection to a node as a peer and proceed with sending a huge number of network messages; some of those messages might cause the node to be overwhelmed by intensive computations.

Tasks

• [x] Research current traffic received per message
• [x] Analysis of existing network messages to discover which of them are computational intensive
• [x] First draft of possible rate limits
• [x] Implement or import basic utilities to rate limit network messages
• [x] Use the utilities mentioned above in the context of a peer connection to apply the limits
• [x] Implement a way to configure rate limits, considering all ramifications of node operations (updates, many types of deployments, etc)
• [x] Update grafana dashboards
• [x] Test the feature

[Trisfald]

Draft document with tentative rate limits.

[Trisfald]

Snippet of configuration showing how default rate limits can be customized:

    "experimental": {
      "inbound_disabled": false,
      "connect_only_to_boot_nodes": false,
      "skip_sending_tombstones_seconds": 0,
      "tier1_enable_inbound": true,
      "tier1_enable_outbound": true,
      "tier1_connect_interval": {
        "secs": 60,
        "nanos": 0
      },
      "tier1_new_connections_per_attempt": 50,
      "network_config_overrides": {
        "connect_to_reliable_peers_on_startup": null,
        "max_send_peers": null,
        "routed_message_ttl": null,
        "max_routes_to_store": null,
        "highest_peer_horizon": null,
        "push_info_period_millis": null,
        "outbound_disabled": null,
        "accounts_data_broadcast_rate_limit_burst": null,
        "accounts_data_broadcast_rate_limit_qps": null,
        "routing_table_update_rate_limit_burst": null,
        "routing_table_update_rate_limit_qps": null,
        "received_messages_rate_limits": {
          "rate_limits": {
            "Block": {
              "maximum_size": 1,
              "refill_rate": 1.0,
              "initial_size": 1
            }
          }
        }
      }
    }
  },

[Trisfald]

Test is in progress with two custom made nodes deployed on mainnet. Dashboard link.