brianlong
commented
4 years ago This is interesting with a large IP V6 address space (/48?) per node to use for the VIPs:
- Each peer-pairing uses a different V6 address from its own /48 pool. (2^80) * #-of-cluster-nodes connection-pair possibilities. That is a really big address space!
- The VIP V6 pairings will not be advertised. The V6 space is essentially the 'private' space.
- Reject/block all V6 addresses that are not used for peering.
- Spoofing the V6 pairings will be very difficult (but not totally impossible) if the attacker doesn't know which address pairs are in use.
Then downgrade the importance of the V4 interface:
- V4 is used only to establish private peer-pairings.
- Use QoS & bandwidth limits on the V4 interface -- make it slow by design!
The goal: If someone floods the slow V4 interface, it will have minimal impact on the V6 interface or the validator. A V4 attack will impede the ability to make new peering connections, but the cluster will still run with the existing pairings over V6.
Said differently, if you can't prevent a V4 DDOS attack, at least minimize the impact of the attack. The attacker might not bother if they can't take the cluster down.
ghost
Problem
Validators public ports are open and public and easy to DDOS.
Proposed Solution
There is no easy solution to DDOS, so our strategy is to allow the validator to separate bad traffic from good traffic. Basic idea is to use Virtual IP/Ports better known as VIPs. A VIP is a map of
(Sender IP, Virtual IP, Virtual Port) => (Physical IP, Physical Port)
Packets received at VIP from the sender IP are forwarded to the actual physical IP and physical port. A sample of peers covering the vast majority of the stake in the cluster that the node is connected to will have its own private VIP. Each peer can then be blocked separately, and the public VIP can be run with its own Quality of Service. The node should be able to shutdown the public VIP and rotate the IP address without losing connection to any of its peers who have an established private VIP.
Initialization of the public VIP
Creating Private VIPs
Private VIPs establish a different VIP per peer for the validator. Validator selects the top 99% of the network stake and creates a VIP per peer. The VIPs are encrypted with the validators keys, and the crds value containing all the private VIPs is published in gossip.
Periodically:
Receiving Private VIPs
Validators shouldn't need to override the ContactInfo's inside the crds table.
iptablesrules can automatically remap data sent to the public VIP to the private one.Periodically:
Notes:
VIPs should remain encrypted and randomly sorted to avoid discovery of the private peers. Validators would have to decrypt and verify each VIP to discover their peer information. Although validators can keep their index consistent within the PrivateVIP entries.
VIPs automatically expire if the peer stops updating its PrivateVIP in N seconds. Once expired the peer should fallback to taking to the public interface.
VIP server should call out to user modifiable scripts (python, json in/out) to create and destroy VIPs. This would allow users to define how ACLs and QoS fields are updated. VIPServer should keep track of timing and existence of the mappings, but the actual calls to
iptablesor creating dynamic ACLs on aws should be done in user defined scripts.The interface that solana needs to build is a rest api to the crds. Node needs to read the contact infos and VIPSs and write their own VIPs. The vip server can handle the rest.