search

freebsd / pkg

Package management tool for FreeBSD. Help at #pkg on Libera Chat or pkg@FreeBSD.org
Other
744 stars 279 forks source link

Add Concurrent Downloads of Multiple Packages #1628

Open robmathers opened 6 years ago

robmathers commented 6 years ago

I'd like to see pkg have the ability to use parallel downloads to better take advantage of fast connections. With a connection capable of sustaining 5+ MB/s, it's rarely used to its full capacity when upgrading packages, which are typically many small downloads, rather than large ones (especially if set to a quarterly upgrade cycle).

By offering an option for multiple concurrent downloads, bandwidth could be used fully and upgrades completed sooner.

I'd suggest a default value of 2 concurrent downloads, but my 150Mbit connection could easily sustain 5 or 10 or more.

v1ne commented 5 years ago

Please do! Debian added this years ago and it helps a lot with small packages. (tuning FreeBSD so that the TCP ramp-up is faster would help, too. ;) )

grahamperrin commented 1 year ago

Also:

suchipi commented 1 year ago

+1 on this; installing large metapackages like xorg can take a long time since downloads are all serial. Many web servers can send data faster with multiple connections open.

Crest commented 1 year ago

Does pkg (1.20+) try to take advantage of HTTP pipelining when fetching multiple packages to avoid having to establish and ramp up a connection per packet? How would using multiple concurrent downloads (by default without trickery like running pkg fetch from xargs) impact the official package mirrors (e.g. more concurrent connections)?

michael-o commented 9 months ago

Does pkg (1.20+) try to take advantage of HTTP pipelining when fetching multiple packages to avoid having to establish and ramp up a connection per packet? How would using multiple concurrent downloads (by default without trickery like running pkg fetch from xargs) impact the official package mirrors (e.g. more concurrent connections)?

HTTP pipelining is dead with http/2.

Crest commented 2 months ago

I "solved" the problem for my usecase with a Varnish caching server, because it's flexible enough to override caching hostile official package mirror HTTP headers and still validate each cache hit using an HTTP HEAD requests to compare via ETag. Finally pkg fetch saturates a 10Gb/s LAN connection and I don't have to feel guilty for hammering the CDN with lots of repeated downloads while testing my automations.