search

appveyor / ci

AppVeyor community support repository
https://www.appveyor.com
344 stars 66 forks source link

Caching proxy server? #643

Open krlmlr opened 8 years ago

krlmlr commented 8 years ago

Would it be possible to provide a caching proxy server for all downloads? This might reduce the "in" traffic a lot, which is probably free to you but at the same time is somebody else's "out" traffic. Thanks for considering.

FeodorFitsner commented 8 years ago

Nice idea. Are there any existing solutions for that?

krlmlr commented 8 years ago

I'm not familiar with the technology, but squid and apache came up on a very coarse search. Need to think about HTTPS, too.

dlemstra commented 8 years ago

I would also like it if our downloads are cached. We need to download some packages from chocolatey and this occasionally fails. We have 14 environments that need to download the same packages. If one of them fails we will need to restart the whole build. This build takes 4 hours at the moment so that means it could take a while before the restarted build is complete. I was unable to set up caching or download retries for these installers so having a caching proxy would also help us.

dlemstra commented 8 years ago

It turns out this was already added :+1: (https://www.appveyor.com/docs/how-to/http-proxy). Might be time to close this issue @FeodorFitsner.

FeodorFitsner commented 8 years ago

That proxy is not caching. Regarding failing Chocolatey packages - the solution depends on errors you get and package specifics. Sometimes re-implementing package installation in your own script with retries/caching could help.

dlemstra commented 8 years ago

Would it be an option to add caching to those proxies?

FeodorFitsner commented 8 years ago

Yeah, we might add this in the future. It's going to be usable mostly for nuget and Chocolatey.

dlemstra commented 8 years ago

Might be easier/cheaper to use the private CDN of Chocolatey? (https://chocolatey.org/docs/features-private-cdn)

FeodorFitsner commented 8 years ago

Chocolatey with private CDN sounds interesting, thanks! Will contact them for the pricing.

roji commented 6 years ago

Just wanted to add my +1 here, I think this would be a great addition to Appveyor, and much better than the current build cache feature (where HTTP-downloaded files are in question).

Note that there could be two options here:

  1. A classic caching proxy that would have to be pointed to at the client side. This is relatively easy to set up - just a server (or several) with tinyproxy or squid, and a lot of storage. Your images could be set up to automatically use it (e.g. via NuGetDefaults.Config for nuget)
  2. A transparent caching proxy, which automatically intercepts HTTP/HTTPS traffic. This is more complicated to set up, but has the advantage of being completely transparent and for working for all web traffic without requiring any client-side configuration or setup.

The advantages of either of the above include:

  1. It's implicitly shared between all your users, so a big space and bandwidth save. Your current build cache feature is per-user.
  2. It wouldn't require any sort of manual setup on the part of users - everything happens transparently. I imagine there are quite a few builds out there which don't set up their own caching of the nuget folder, these would get an immediate and considerable speed boost

Hope you consider this!

cmorty commented 5 years ago

Seems like traffic is just too cheep. Having a transparent proxy would reduce traffic (Good for AppVeyor) and really speed things up dramatically (Good for the customers).