Joystream / joystream

Joystream Monorepo
http://www.joystream.org
GNU General Public License v3.0
1.42k stars 115 forks source link

Improve concurrent downloads mechanism #5052

Open mnaamani opened 8 months ago

mnaamani commented 8 months ago

Colossus has an implementation for queuing and processing tasks in a work stack. There are several problems with the implementation especially the way it is used in the syncing worker. To get the best concurrency the sync worker should add all tasks to the work queue.

We should improve it to get better concurrency without the limitation mentioned above and to work better when we improve on how we fetch tasks in batches (see: https://github.com/Joystream/joystream/issues/5051)