search

JuliaLang / julia

The Julia Programming Language
https://julialang.org/
MIT License
45.64k stars 5.48k forks source link

Optimize downloads for CI #27895

Closed ViralBShah closed 4 years ago

ViralBShah commented 6 years ago

We surpassed our comped download on Fastly this month (> 9TB).

There are probably opportunities to save bandwidth by doing things like optimizing on CI downloads (by caching binaries directly into the CI systems). We should see if there are other things we can do.

staticfloat commented 6 years ago

Some quick stats for the last month of June:

Domain GB/day
julialangnightlies-s3 156.2
julialang-s3 129.7
julialangmirror-s3 39.1

Everything else is peanuts compared to these three. I'm really surprised by the nightlies numbers; I'm guessing this is predominantly CI, as I just don't see us pushing 150 GB of nightlies per day to typical users.

If our CI (Travis, Appveyor, etc...) can be served by a cheaper source, (e.g. interposing an AWS-local source for AWS-hosted CI services to make use of the free within-availability zone bandwidth) we can likely cut this down by a lot.

ViralBShah commented 6 years ago

Appveyor does not run on AWS (they are on rackspace). I am not sure where Travis and Circle run. I suspect we need to push local binaries to all the places. Maybe cut down circle, if we no longer need it with Travis being stable (although that can change any time)?

staticfloat commented 4 years ago

Oh hey, old issue! We have successfully implemented our own cloud-native caching system for Julia downloads, and we're well on our way to doing the same for packages. Issue, consider thyself closed!