search

scionproto / scion

SCION Internet Architecture
https://scion.org
Apache License 2.0
402 stars 161 forks source link

Beacon server high availability #50

Closed kormat closed 9 years ago

kormat commented 9 years ago

In order to remove Beacon Servers as SPOFs, we're moving to a distributed architecture. The current proposal is based on Zookeeper, with the Kazoo python client library.

All Beacon Servers (BS) in an AD connect to a Zookeeper (ZK) cluster. They register as part of a Party; this is used to create multiple DNS entries for the Beacon service. The BSes try to grab the master lock, whichever BS succeeds is the master until failure/shutdown.

When an Edge Router (ER) receives a new PCB, it connects to any BS in the DNS record. If that connection times out, the ER tries the next record, and so on. The receiving BS drops the new PCB into the shared ZK "incoming" directory. All BSes have watches set on this directory, and load any new PCBs into their PathStores.

Once per propagation period, the master BS will select k PCBs from Pathstore (using freshness criteria to make sure nothing stale is selected) and propagate them. It will also move all PCBS from the ZK "incoming" directory to the "recent" directory, clearing the "recent" dir beforehand. Any newly-started BS will load all PCBs from both "incoming" and "recent" directories to prime their PathStores.

kormat commented 9 years ago

@pszalach @shitz @LorenzoBaesso

kormat commented 9 years ago

The current implementation is still... not very good. I'm seeing the PCB processing taking up to 5s sometimes. It might make more sense to switch to polling the shared path for PCBs, rather than using a watcher. That way we could handle bulk processing much easier.

kormat commented 9 years ago

If zookeeper goes down (or can't be contacted), it takes on the order of minutes for Kazoo to reconnect once it's back up :/

E.g., took zookeeper down for 3 minutes, then brought it back up. It took Kazoo a further 3 minutes to notify us that the connection had dropped, and to reconnect.

pszal commented 9 years ago

Have you tried KazooRetry? e.g.: ... retry = KazooRetry(max_tries=-1, max_delay=1) zk = KazooClient(..., connection_retry=retry) zk.start(timeout=None) ... For me works pretty good.

On 17.04.2015 18:13, kormat wrote:

If zookeeper goes down (or can't be contacted), it takes on the order of minutes for Kazoo to reconnect once it's back up :/

— Reply to this email directly or view it on GitHub https://github.com/netsec-ethz/scion/issues/50#issuecomment-94018458.

kormat commented 9 years ago

On 20 April 2015 at 09:44, pszalach notifications@github.com wrote:

Have you tried KazooRetry? e.g.: ... retry = KazooRetry(max_tries=-1, max_delay=1) zk = KazooClient(..., connection_retry=retry) zk.start(timeout=None) ... For me works pretty good.

Ah hah. It might well be the exponential backoff that's biting me. Ok, i'll try using a custom retry object. Thanks!

kormat commented 9 years ago

92 helps a lot with performance, but it's still too cpu-intensive. Investigating the bottlenecks currently.

pszal commented 9 years ago

As decided today to handle PCB propagation during partitions:

if is_master() or len(beacon_servers) == 1 or zk.state.disconnected():
    propagate()
kormat commented 9 years ago

This is now in place.