sync v2

[dshulyak]

this is a placeholder (not a spec) for all known sync improvements:

• [ ] drop custom communication protocol for fetcher and use stream per request

this was implemented way before project switched to libp2p and had other requirements in mind. right now this code has no benefits.

• [ ] efficient sync from scratch

the goal is to have implementation that makes good use of hardware resources:

• limited by network throughput it can get from peers. good metric is how fast state is synced based on total throughput it can get from peers. for example if it can get 10MB/s from peers and total unique data is 1GB, then whole sync should take ~100s

• downloading data should not be blocked by validation and writing to disk. this is straighforward to solve with pipelining, but will require some changes in a way sync interacts with validators

• [ ] dos resistance

current implementation lacks protection against abuse. some queries are expensive and we should rate limit them in a way that allows honest nodes to get data that they need, and prevents abusers from causing harm.

additionally every sync endpoint should be fuzzed. it needs to be taken into account when writing code for it.

• [ ] hash resolution that also download late data (atxs/ballot/malfeasance proofs)

existing implementation may miss out of order data. this is required long term for consensus correctness. the implementation idea was to index data by received timestamp, and send such data to peers according to the timestamp.

---

from that list above it will be good implement 3 first items within 3 months. 4th item in 6 month or so

[pigmej]

We should also make it so that requests are evenly split, because right now they're not.

I understand the need for recent layers (yellow), but all others are DDoSing nodes.

[lrettig]

We should also make it so that requests are evenly split, because right now they're not.

I understand the need for recent layers (yellow), but all others are DDoSing nodes.

I don't understand any of what I'm looking at here. What does this mean? What are the colors? What are the axes?

[fasmat]

@lrettig: Requests for layer data comes in bursts, roughly every 5 minutes. For recent layers (yellow - recent 10 layers) this is OK, but for older layers (other colours, e.g. blue - last 2000 layers) this should be more evenly spread out to avoid spikes of requests / traffic that could be spread across a larger timeframe.

[pigmej]

very good summary from @fasmat that's exactly what it is. X time, Y number, colors, and legend "layers behind CurrentLayer at the time of asking.

[ivan4th]

Current code related to synvc2 effort is here: spacemeshos/go-spacemesh#5769 The generic peer-to-peer set reconciliation algorithm based on the "monoid tree" structure is already implemented. The work is being done to perform "SREP"-style" multi-peer sync that eventually can replace gossip.

There's also another proposal of a potentially simpler sync approach, but it makes an assumption that transferring e.g. a month worth of ATX IDs is and always will be cheap, and while this assumption may (or may not) hold for a while for a while after the ATX merge, it is not guaranteed to hold in the long run