Mirrors for repositories

[GoogleCodeExporter]

Sorry to write it here (i am unable to post at the google groups). 

I think it would be better if it would be possible to have some kind of mirror 
system. 1.npacked.example.com
2.npacked.example.com
3.npacked.example.com

The same would go for the own hosted downloads. (The xml should get some kind 
of signature so it can't be changed). The downloads have hash checksum. This 
would decentralize the system around one mirror url. 

Original issue reported on code.google.com by Pimmetje on 19 Jun 2014 at 10:05

[GoogleCodeExporter]

it's actually even better here

Original comment by tim.lebe...@gmail.com on 21 Jun 2014 at 6:56

• Changed state: Thinking

[GoogleCodeExporter]

Original comment by tim.lebe...@gmail.com on 22 Jun 2014 at 2:15

• Changed title: Mirrors for repositories

[GoogleCodeExporter]

There are several ways to make it load balance/mirror. I am willing to help 
implement it. And if needed i would be able to run a mirror. Just let me know. 

Original comment by Pimmetje on 22 Jun 2014 at 5:16

[GoogleCodeExporter]

could you please describe these ways here?

Original comment by tim.lebe...@gmail.com on 22 Jun 2014 at 6:56

[GoogleCodeExporter]

One would u using one primary mirror that does a header redirect for 
distributing the load over several mirrors. Based on load, location of the 
user, other parameters a mirror could be chosen. The main site is that 
responsible to check the availability of it's mirrors. In case the primary 
mirror fails a second DNS record is tried. Until one that works of the end of 
the list is reached. 

The secondary method could be a direct mirror or another load balancer. Round 
robin DNS could be used to make the request distributed over several servers if 
needed. In this case u need more then one DNS record in case the chosen mirror 
is down this one is kept in cache for some time. 

The second option would be to only use a DNS system. The first few records 
would then be a Round Robin record. Further down the list every mirror should 
have it's own record. To make sure all mirrors are checked. 

So basically u can have a load balancer or a Round Robin DNS pool or a 
combination. 

A DNS pool is the easiest to implement but is herder to maintain (u need to 
change the DNS records). A load balancer script is harder to build / implement 
but easier to maintain. 

Just my vision. I think there are maybe better ways but this where the options 
i could write down right away. 

If u could share our requirements e.d. I maybe could think of a method that 
fits the situation even better :D. 

Original comment by Pimmetje on 22 Jun 2014 at 8:46

[GoogleCodeExporter]

thank you for the explanations.

The new repository locations at npackd.appspot.com are on the Google App Engine 
infrastructure 
(http://en.wikipedia.org/wiki/Google_App_Engine#Reliability_and_Support). 
Attached is the graph for the number of requests per second for the last 30 
days. The only limitation currently is the amount of money I can pay per day. 
I'd say a 10x as many requests could be served without problems. Thank you for 
the suggestion to host a mirror though. Maybe this will be necessary some time.

Original comment by tim.lebe...@gmail.com on 23 Jun 2014 at 5:57

Attachments:

• chart.png

[GoogleCodeExporter]

It's always hard to weight cost vs potential uptime/speed. I have no idea how 
much data storage / traffic it takes at the moment but Google is in that sense 
not cheap. A small load balancer should of course need to run on such system 
but that would not take much traffic & space so the cost of only that should be 
quite low.

Original comment by Pimmetje on 24 Jun 2014 at 7:05

[GoogleCodeExporter]

I will close this issue. The default repositories are OK. All the methods you 
mentioned already work with the current version of Npackd.

Original comment by tim.lebe...@gmail.com on 5 Oct 2014 at 10:39

• Changed state: WontFix