Add "Best Practices for Distributed Validator Deployment" to Charon Docs

[boulder225]

🎯 Problem to be solved

Charon doc lacks a section on best practices for setting up and running DV clusters. With the upcoming v1.0 release, we should provide the following guidelines and recommendations to ensure DV clusters' performance, security, and stability.

Obol Labs - DV Deployment Best Practices

🛠️ Proposed solution

• [x] Create a new "Best Practices" section in the docs.
• [x] Include the following hardware recommendations:
  • For clusters with 1,000+ validators:
    • 8 core, 16 thread CPU with single thread rating >2,500 on CPU benchmark
    • 32GB RAM
    • NVMe storage (>2TB for 5,000 validators)
    • 25,000 IOPS
    • 25mb/s internet bandwidth
• [x] Emphasize bare metal machines or high-performance cloud instances, prioritizing physical setups over virtualized ones.
• [x] Recommend keeping intra-cluster latency below 235ms, with <80ms being optimal for high-scale deployments.
• [x] Advise provisioning machines within the same continent to minimize latency and ensure high availability.
• [x] Highlight the need for direct connections between Charon clients (port 3610 open) to reduce latency and bandwidth constraints.
• [x] Recommend each instance to have its own independent beacon node, validator client, and Charon.
• [x] To reduce latency, suggest installing Charon clients on the EL/CL machines instead of the VC machine (if they are separate).
• [x] Encourage the use of Obol Splits smart contracts for non-custodial fund
• [x] Recommend cluster sizes that allow for Byzantine Fault Tolerance.
• [x] Advise operators to have an EL/CL pair for each node.
• [x] Suggest thoroughly analysing performant, low-latency MEV-Boost relays before inclusion in DV node configuration.
• [x] Recommend a combination of multiple consensus and execution clients for healthy client diversity within the cluster.
• [x] Encourage operators to store node health and validator performance data independently.
• [x] Reference the Miga Labs and Lido Performance Reports for additional insights.

[boulder225]

Hey team! Please add your planning poker estimate with Zenhub @gsora @KaloyanTanev