Reconsider how we exchange routing table information

[bowenwang1996]

Currently we always exchange the entire routing table with peers when new connections are established and broadcast newly received routing table information. This is not efficient and leads to an increase of network traffic. We should consider optimizing this process with better strategies to synchronize routing table between nodes in the network.

• [x] propose new algorithm - Details of the new algorithm are in #3934
• [x] Investigate if we verify edges multiple times Should be fixed by: https://github.com/near/nearcore/pull/4098
• [x] Investigate how edges are stored
• [x] Make a prototype https://github.com/near/nearcore/pull/4112
• [ ] Finish optimizing routing table computation
• [ ] Finish optimizing Validation of edges

[bowenwang1996]

@pmnoxx should we close this one in favor of #3934 ?

[pmnoxx]

@bowenwang1996 I decided to split issues into two to make it more cleaner. #3934 focuses on theoretical work This one on the actual implementation.

[pmnoxx]

I wrote a prototype: https://github.com/near/nearcore/pull/4112 I'm going to test it and see if it works.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[pmnoxx]

There are still a few optimizations we can do to exchange routing tables:

• Improve routing table exchange computation by factor of 4x. It used to take 1s to compute routing table computation for 100_100 test, now it takes 1ms, we can finish optimizing it, and make it 0.25ms.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity in the last 2 months. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.