[feature request] Prefetch

[adrfantini]

A similar project to this, pacoloco, has the capability to periodically prefetch packages, based on a CRON schedule, so that when you have to fetch them, they are already stored in the cache.
I think this is brilliant, it helps with reducing download times when packages are needed. It would be a great feature addition!

[nroi]

I thought about this feature in the past, I admit it would be a nice feature from the user's perspective, but I've decided not to implement this for different reasons.

Most importantly, because it's wasteful in terms of resources: With this feature, you can't make sure that prefetched packages are actually needed by the client. It could be the case that the client first downloads version 1 of a given package, then flexo does a prefetch of version 2, but before the client downloads version 2, the package is already updated to version 3, and the client then directly upgrades from version 1 to version 3. So, you've just wasted the mirror's bandwidth by having downloaded version 2. Also, how do you determine which packages to prefetch? The client might have downloaded a given package and then immediately uninstalled it. If you then also prefetch those packages, this becomes even more wasteful than it already is. So you would somehow have to tell flexo which packages are actually required by the client, which would increase the complexity quite a bit.

Given that Arch Linux mirrors are hosted by volunteers who have to cover the hosting costs with their own money, I don't feel good about implementing this kind of feature.

[adrfantini]

I understand and respect this decision.

[FixedWing99]

Perhaps core components could be treated differently? Everyone downloads core components. This would also afford an opportunity to test it out.

Stephen

[nroi]

Perhaps core components could be treated differently?

But core components also tend to be rather small, which makes this feature not as useful. I just checked the packages sizes from core.db:

• Average size is around 5 MiB.
• Median size is just 304 KiB.

At first glance, it seems to me like this would introduce significant complexity in the code base, without making flexo that much more useful.