search

HaxeFoundation / haxelib

The Haxe library manager
https://lib.haxe.org/
MIT License
173 stars 77 forks source link

Add haxelib website to cloudflare #33

Open waneck opened 11 years ago

waneck commented 11 years ago

The current haxelib repo server has a very slow internet connection; downloading NME's 60MB zip takes more than 30min!

I can offer to sponsor a server that has better speed to solve this problem; but an easier solution would be to try to use CloudFlare on top of the sever.

CloudFlare is a free CDN (see https://www.cloudflare.com/features-cdn ), and it should speed up the server's upload rate considerably.

Simn commented 11 years ago

It seems to run at normal speed for nme. NME 3.5.6 installed within about 20 seconds.

waneck commented 11 years ago

really? It almost took an hour here. And my internet connection isn't too bad (50mbps)

back2dos commented 11 years ago

I can imagine that the haxe.org server is not highly prioritized in transatlantic internet traffic. So for all our non-european users CloudFlare is really worth consideration. However, if we do use CloudFlare, I think we need to set the cache headers right and we probably must forbid uploading the same version again, since it will be cached by the CDN - at least that's how I understand things.

waneck commented 11 years ago

Makes sense for me, back2dos! It should be as simple as setting a .htaccess configuration for it.

MattTuttle commented 11 years ago

:+1: NME takes a good 10 minutes for me in the central US. Most other libraries are not a problem since they usually fall under 1MB.

back2dos commented 10 years ago

@waneck Can you take charge of this, even if only by mere delegation? ;)

jasononeil commented 10 years ago

Sad to say I haven't done anything with either Cloudfront or AWS yet, but I'm keen to learn and contribute to that if someone can point me in the right direction (just a suggested strategy to follow and where to view the various APIs would probably be enough).

On Wed, May 28, 2014 at 9:05 PM, Juraj Kirchheim notifications@github.comwrote:

@waneck https://github.com/waneck Can you take charge of this, even if only by mere delegation? ;)

— Reply to this email directly or view it on GitHubhttps://github.com/HaxeFoundation/haxelib/issues/33#issuecomment-44403382 .

waneck commented 10 years ago

I can handle it. But I'll need some help to get access to the Haxe.org DNS

waneck commented 10 years ago

Anyway, if you want to do it, the only thing you need is a cloudflare account, and repeat all our current DNS records into CF

jasononeil commented 10 years ago

Ping @ncannasse (I assume?)

ncannasse commented 10 years ago

@waneck reach me on Skype so we can setup it together

MatiasArriola commented 10 years ago

+1 I couldn't get installed any package today. Download speeds are really slow, and it ends up throwing Http connection timeout after a while. I tried using the -notimeout parameter, but I kept getting interrupted by the same timeout errors. Observed the same behavior from both linux and windows. This is reproducible only with big packages such as lime. Note that everything worked fine weeks ago using the same connection. I'm located in Argentina, and looks like it happened to other people as well. I guess for now the best workaround is to install everything from git.

back2dos commented 9 years ago

What's the progress on this?

waneck commented 9 years ago

Oh sorry, I completely missed @ncannasse's post. I'll get in contact with him to set this up

andyli commented 9 years ago

Hey, I would also like to see a faster haxelib download soon. Right now TravisCI builds a helloworld example in ~7 min, in which > 4 min is spent on haxelib install hxcpp. Interestingly, it takes only a few second to haxelib git hxcpp ....

nadako commented 8 years ago

Any progress?

andyli commented 8 years ago

I've just enabled AWS CloudFront for lib.haxe.org. Please report any problem regarding to haxelib install/submit/whatever.

I used CloudFront instead of CloudFlare because:

  1. CloudFront allows proxying to different origins based on URL. i.e. Currently the origin of http://lib.haxe.org/files/* is AWS S3 while the origin of all other request is AWS Elastic Beanstalk (EC2).

  2. CloudFlare does not allow proxying too much non-webpage content, which is our main use-case. According to its terms of use, "SECTION 10: LIMITATION ON NON-HTML CACHING":

    Using an account primarily as an online storage space, including the storage or caching of a disproportionate percentage of pictures, movies, audio files, or other non-HTML content, is prohibited.

andyli commented 8 years ago

Sadly, CloudFront suddenly doesn't play well with our current haxelib clients, so I've disabled it. I don't think I've changed anything from our side that could cause the issue, so may be it is CloudFront.

The issue is, when haxelib install ufront-mvc using the haxelib bundled with haxe 3.2.1 or 3.1.3, it will error with Invalid string length. The reason is probably this: https://groups.google.com/forum/#!topic/haxelang/F2wRMafvVrI. I checked that the issue goes away if we set http.noShutdown = true in the haxelib client. But we cannot really fix the haxelib clients that are already released... :( It is funny that it happens only for some libs (e.g. ufront-mvc) but not others. Also, the haxelib client bundled with haxe dev is somehow not affected...

My plan is to fix the haxelib client to support http redirect (301/302), and then let our server to route the new clients to download libraries from a CloudFront-backed url. Old clients will still be directly served by the web server, which will be a bit slower, but at least works.

buckle2000 commented 7 years ago

Downloading hx-cpp takes about 10 min in Hong Kong with manual download. CLI download is far worse.