implement automatic fallback for cdn urls

[juliangruber]

fixes https://github.com/Medium/phantomjs/issues/595

Here's example output when you provide a faulty cdn:

$ PHANTOMJS_CDNURL=http://foobar.com node install.js 
PhantomJS not found on PATH
Downloading http://foobar.com/phantomjs-2.1.1-macosx.zip
Saving to /var/folders/lb/v0m255j93wl31f312glkrkqw0000gn/T/phantomjs/phantomjs-2.1.1-macosx.zip
Receiving...

Error requesting archive.
Status: 404
Request options: {
  "uri": "http://foobar.com/phantomjs-2.1.1-macosx.zip",
  "encoding": null,
  "followRedirect": true,
  "headers": {},
  "strictSSL": false
}
Response headers: {
  "server": "nginx/1.10.1",
  "date": "Mon, 08 Aug 2016 16:21:52 GMT",
  "content-type": "text/html",
  "transfer-encoding": "chunked",
  "connection": "close",
  "accept-ranges": "bytes",
  "vary": "Accept-Encoding"
}
Make sure your network and proxy settings are correct.

If you continue to have issues, please report this full log at https://github.com/Medium/phantomjs
Something unexpected happened, please report this full log at https://github.com/Medium/phantomjs
Downloading https://github.com/Medium/phantomjs/releases/download/v2.1.1/phantomjs-2.1.1-macosx.zip
Saving to /var/folders/lb/v0m255j93wl31f312glkrkqw0000gn/T/phantomjs/phantomjs-2.1.1-macosx.zip
Receiving...
  [==============================----------] 74%
Received 16746K total.
Extracting zip contents
Removing /Users/julian/dev/medium/phantomjs/lib/phantom
Copying extracted folder /var/folders/lb/v0m255j93wl31f312glkrkqw0000gn/T/phantomjs/phantomjs-2.1.1-macosx.zip-extract-1470673319245/phantomjs-2.1.1-macosx -> /Users/julian/dev/medium/phantomjs/lib/phantom
Writing location.js file
Done. Phantomjs binary available at /Users/julian/dev/medium/phantomjs/lib/phantom/bin/phantomjs

[nicks]

this seems like it will lead to cascading failure, so that one CDN goes down, and the other CDNs get DDOSed. I think we need to find tools that strongly encourage people to avoid hitting the CDNs in their continuous builds, not hit them harder.

[juliangruber]

the purpose of a cdn is to deliver content to the masses, isn't it? if each employ their own rate limiting feature (which i'm sure they do) this shouldn't be a problem.

[mekdev]

Hi guys,

Any idea when this fix will go in. We have currently removed phantomjs-prebuilt from our dependency so it does not fail CI builds

[nicks]

If you are downloading phantomjs as part of a CI process, please stop. Install phantomjs once on PATH. This package will automatically use a pre-installed binary it finds on PATH, and you will be much happier.

This PR will never be merged. This is a tragedy-of-the-common problem, and gobbling up more resources will make the problem worse.

[juliangruber]

@nicks I'm a bit confused, isn't the purpose of phantomjs-prebuilt to have a module that doesn't require you to have phantomjs(1) installed in your $PATH?

[juliangruber]

for anyone else who's having issues with this, I just published https://www.npmjs.com/package/phantomjs-prebuilt-that-works with the CDN fallback logic in place.

[jdalton]

@nicks

If you are downloading phantomjs as part of a CI process, please stop.

Would a readme note on how to tackle Phantom v2 on Travis-CI help? I use Travis caches to avoid repeat installs:

.travis.yml

cache:
  directories:
    - travis_phantomjs

before_install:
  # Upgrade PhantomJS to v2.1.1.
  - "export PHANTOMJS_VERSION=2.1.1"
  - "export PATH=$PWD/travis_phantomjs/phantomjs-$PHANTOMJS_VERSION-linux-x86_64/bin:$PATH"
  - "if [ $(phantomjs --version) != $PHANTOMJS_VERSION ]; then rm -rf $PWD/travis_phantomjs; mkdir -p $PWD/travis_phantomjs; fi"
  - "if [ $(phantomjs --version) != $PHANTOMJS_VERSION ]; then wget https://github.com/Medium/phantomjs/releases/download/v$PHANTOMJS_VERSION/phantomjs-$PHANTOMJS_VERSION-linux-x86_64.tar.bz2 -O $PWD/travis_phantomjs/phantomjs-$PHANTOMJS_VERSION-linux-x86_64.tar.bz2; fi"
  - "if [ $(phantomjs --version) != $PHANTOMJS_VERSION ]; then tar -xvf $PWD/travis_phantomjs/phantomjs-$PHANTOMJS_VERSION-linux-x86_64.tar.bz2 -C $PWD/travis_phantomjs; fi"
  - "phantomjs --version"

[nicks]

@jdalton cool! I don't know much about travis but that seems reasonable to me. would you like to submit a PR or want me to add it?

[jdalton]

I'll submit a PR :D