Improve bitswap performance

ipfs-inactive / package-managers

[ARCHIVED] 📦 IPFS Package Managers Task Force

MIT License

97 stars 11 forks source link

Improve bitswap performance #85

Open dirkmc opened 5 years ago

dirkmc commented 5 years ago

In particular we'd like to focus on improving bitswap performance for

one node retrieving a large amount of data from several nodes who already have it
one node retrieving a segment of data from several nodes who have different parts of it
a group of nodes retrieving data from several far-away nodes who already have it

dirkmc commented 5 years ago

I created an issue in go-bitswap to explain the current implementation of bitswap sessions, as a first step towards understanding where improvements can be made.

dirkmc commented 5 years ago

I created an issue in go-bitswap to highlight some issues with latency measurement, and proposals to address them

dirkmc commented 5 years ago

I created an issue in go-bitswap to highlight some issues with session request sharding, and explore parameters for a solution.

dirkmc commented 5 years ago

I created an issue in go-bitswap to discuss randomizing want list requests.

dirkmc commented 5 years ago

I created a PR in go-bitswap to improve handling of response messages: it processes messages as a batch instead of one-by-one, which improves session request splitting.

dirkmc commented 5 years ago

This week

merged PR to improve handling of response messages
merged refactoring PR which will simplify bigger changes we'd like to make once we have worked out how to tackle request sharding (Expect to have this worked out early next week)
opened PR to recognize when we've received a block that we want but haven't had a chance to ask for yet: https://github.com/ipfs/go-bitswap/pull/174

dirkmc commented 5 years ago

Last week

merged PR to recognize when we've received a block that we want but haven't had a chance to ask for yet: https://github.com/ipfs/go-bitswap/pull/174
opened issue to discuss RFC for Bitswap extensions: https://github.com/ipfs/go-bitswap/issues/186

dirkmc commented 5 years ago

Last week

Implemented PoC of Bitswap Protocol extensions This week
Improving PoC performance against Bitswap benchmarks

dirkmc commented 5 years ago

FYI I was working on improving the Bitswap benchmarks by making small modifications to the PoC, but it was too haphazard and buggy, so I'm going to do a proper refactor first.

dirkmc commented 5 years ago

With the latest changes the PoC seems stable, so I'm looking into improving performance.

I created an issue to discuss how we can improve the performance of bitswap's engine (ie the part of bitswap that receives wants and sends out blocks). I've implemented Proposals 1 - 4, which address ordering wants and grouping them by block size. Next I will implement proposal 5, which adds parallelism.