search

nchammas / flintrock

A command-line tool for launching Apache Spark clusters.
Apache License 2.0
638 stars 116 forks source link

Add option to download Spark from a custom URL #101

Closed rmessner closed 8 years ago

rmessner commented 8 years ago

We are trying to setup a cluster with Spark 1.6.1, without HDFS, but it fails. The reason is that the file downloaded is corrupted, so we can't untar it.

Would be a nice feature to allow user to specify an alternate mirror to download the pre-build spark. This would add a new key to configuration, like : 

services:
  spark:
    version: 1.6.0
    preferred-mirror: http://apache.crihan.fr/dist/spark/spark-${spark_version}/${file}

The default value would be https://s3.amazonaws.com/spark-related-packages/${file} and the variables available file ( as the file name of pre-build spark is always the same whatever the mirror is ), spark_version ( e.g. : 1.6.1) , distribution ( e.g. : hadoop2.6 )

Related to #71.

nchammas commented 8 years ago

Thanks for reporting this. I was just thinking about it earlier this week.

The problem you are seeing with Spark 1.6.1 is a known issue upstream that I've been bugging the core Spark maintainers about for a few weeks now. :disappointed:

The Spark 1.6.1 package on S3 that Flintrock uses is corrupt, and unfortunately we don't have the ability to control it. That resource is controlled by the Spark maintainers.

For the record, your proposal here is the Spark analogue to #71, which is for Hadoop.

rmessner commented 8 years ago

My co-workers just told me they have the same issue, so i will make the same fix that i submit on #104 if it's okay for you @nchammas .

nchammas commented 8 years ago

You mean you'll open a PR for #71 that's similar to #104?

rmessner commented 8 years ago

Yes, or make it in the same PR, as you want

nchammas commented 8 years ago

Oh sure, it can be the same PR.

JoshRosen commented 8 years ago

AFAIK the corrupt packages should be fixed now; let me know if they're still a problem.

nchammas commented 8 years ago

Yes, I believe they're fixed now. Thanks for taking care of that @JoshRosen!

Retitling issue accordingly.