Open alrevuelta opened 1 year ago
Picking a shard at random would be the easiest and a good first step but maybe we could count the # of nodes for each shard we get from Discv5 and use the lowest?
I'm not sure it would be better since the nodes we get from Discv5 are kinda random too.
but maybe we could count the # of nodes for each shard we get from Discv5 and use the lowest?
I see two problems with this:
imho the best approach is some statistical one that ensures coverage without i) relying on the ENR and ii) without having to crawl the whole network.
Problem
With autosharding we need to ensure that every shard in the network has a similar number of nodes relaying traffic in it. For example, if there are 1000 nodes in the network and 10 shards, ideally each shard will have 100 nodes (assuming each node connects to 1 shard).
If we assume that content topics follow an uniformal distribution, then we can consider this problem as solved. But content topics can easily be biased to land in a specific shard. So we need an extra protection, "altruistic sharding TM" (open for names).
Suggested solution
Make each node to be subscribed to 1 (or more?) extra shards, on top of the ones that the node is interested in (because of the content topic). This extra shard can be rotated every x time and picked randomly?.
Unfinished, algorithim and reserch to be defined
Alternatives considered
.
Additional context
.
Acceptance criteria
.