fair-research / bdbag

Big Data Bag Utilities
https://fair-research.org
Apache License 2.0
49 stars 23 forks source link

Parallel/concurrent fetch support. #44

Open mikedarcy opened 3 years ago

mikedarcy commented 3 years ago

Add support for multiple concurrent fetch operations. This can significantly speed up transfers when enough bandwidth is available. At a minimum, support parallelism in the HTTP/FTP fetch handlers. Investigate support for other transports as well.