Proposal: Dynamical rate limiting based on bandwidth of p2p-network

lowzj commented 4 years ago

Why you need it?

related issue: #1341

Dragonfly has a good feature to provide rate limiting not only for individual downloading task, also for entire host. The value of rate limiting in Dragonfly is manually configured based on users' experience. It works well in most scenarios. But this static configuration way may cause 2 problems:

The network bandwidth in the host cannot be fully utilized to speed up downloading when the bandwidth is idle.
The downloading task will increase the pressure on the host network and affect the business application deployed on the host when its network is busy.

So this idea is to proposed to dynamically adjust the rate limiting of downloading and uploading data.

How it could be?

Other related information

zaunist commented 4 years ago

I am a junior student majoring in network engineering, and I intend to participate in this summer activity. I would like to know what technical requirements are required to participate in this project, can the mentor give some pointers？

lowzj commented 4 years ago

@beyondexp Welcome to this summer of code, I think that you should:

good at GoLang, basic data structure and algorithm
have a continuous passion for the open source community and willing to share

zaunist commented 4 years ago

@beyondexp Welcome to this summer of code, I think that you should:

good at GoLang, basic data structure and algorithm

have a continuous passion for the open source community and willing to share

I don’t know if I can meet your requirements, but I will do my best

qhb1001 commented 4 years ago

Hello guys! Thanks for this great project.

I would like to list the ideas I got for this proposal, to make sure that I'm on the right track. If you find something wrong or inapproprite to you, please comment! Also, I would really appreciate it if you can provide me with additional recourses for me to dive into for the better implementation regarding the proposal. Okay, here we goes.

The goal of this proposal is to optimize the bandwidth usage in DragonFly, which could be seperated into two parts:

Optimization to the bandwidth of supernode
- Requirement: Currently different clients by default have the same static bandwidth, which is not good for the variable situation of the network. It would be better for the supernode to dynamically assign the suitable bandwidth to the clients, so that the hosts with a lot of available bandwitdth could get more bandwidth, while the hosts with a busy network should drop its bandwidth to ensure the other app in local host not effected by DragonFly.
  
  Idea: All the P2P networks with the supernodes and clients architecture encounter this kinds of problem. And I would like to dive into the papers regarding the classic P2P network to have a better understanding. If you have some expected implementation, please let me know.
- Requirement: There is one scheduling problem when the supernode makes the decision to distribute the image/file blocks to the clients: which client should the data be transmitted to?
  
  Idea: Thanks for @zeusro in #1314 sharing the EIP dynamic programming algorithm, which gives me the idea to abstract this scheduling problem to the backpack problem. The algorithm could run very fast, given the fact that the data scale (# of clients) is small.
Optimization to the bandwidth of client
- Requirement: Given the fact that the supernodes have different available bandwidth, it would be better for the client to choose the one with the most available bandwidth to upload or download the data.
  
  Idea: Simply traverse the supernodes list and find the one with the most available bandiwdth.

Thanks advance for your time on this comment!

dragonflyoss / Dragonfly