Open arokem opened 7 years ago
Is this from branch-2.0
? I think the problem is we didn't backport the change that added 1.6.1 and 1.6.2 to branch-2.0 as seen from [1]. Can you check if adding 1.6.2 there fixes the problem ?
[1] https://github.com/amplab/spark-ec2/blob/06f5d2bc7c222aecb56e2f7bb8b8e160bc501104/spark_ec2.py#L78
This is from branch-1.6
Hmm that means SPARK_VERSION
isn't being correctly parsed somehow. Because as in [2] the default should be sbin
and not bin
As far as I can tell, there's neither a bin
nor a sbin
directory under
/root/spark
. The only thing under /root/spark
is:
/root/spark/conf/spark-env.sh
On Mon, Oct 3, 2016 at 9:13 PM, Shivaram Venkataraman < notifications@github.com> wrote:
Hmm that means SPARK_VERSION isn't being correctly parsed somehow. Because as in [2] the default should be sbin and not bin
[2] https://github.com/amplab/spark-ec2/blob/ 4b57900a24e25accd9c3f14c867920730813bf11/spark-standalone/setup.sh#L5
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/amplab/spark-ec2/issues/57#issuecomment-251291593, or mute the thread https://github.com/notifications/unsubscribe-auth/AAHPNoh-XHzaa-sDSe9b5615YDmnKWFyks5qwdJfgaJpZM4KNRnZ .
That means that Spark wasn't downloaded properly - My guess is this has to do with the tar.gz files for hadoop1 not being found on S3. You could try --hadoop-version=yarn
as a workaround
Thanks! For now, I have resorted to set spark_version
to 1.6.0
, which seems to work, but I'll try that too. Feel free to close this issue, unless you want to keep track of this. And thanks again.
I ran into something similar recently.
It appears that for spark 1.6.2 only a subset of the binaries were uploaded to s3:
$ s3cmd ls s3://spark-related-packages/spark-1.6.2*
2016-06-27 23:47 241425242 s3://spark-related-packages/spark-1.6.2-bin-cdh4.tgz
2016-06-27 23:47 230444067 s3://spark-related-packages/spark-1.6.2-bin-hadoop1-scala2.11.tgz
2016-06-27 23:48 271799224 s3://spark-related-packages/spark-1.6.2-bin-hadoop2.3.tgz
2016-06-27 23:49 273797124 s3://spark-related-packages/spark-1.6.2-bin-hadoop2.4.tgz
2016-06-27 23:50 278057117 s3://spark-related-packages/spark-1.6.2-bin-hadoop2.6.tgz
2016-06-27 23:50 196142809 s3://spark-related-packages/spark-1.6.2-bin-without-hadoop.tgz
2016-06-27 23:51 12276956 s3://spark-related-packages/spark-1.6.2.tgz
While
$ s3cmd ls s3://spark-related-packages/spark-1.6.0*
2015-12-27 23:07 252549861 s3://spark-related-packages/spark-1.6.0-bin-cdh4.tgz
2015-12-27 23:15 241526957 s3://spark-related-packages/spark-1.6.0-bin-hadoop1-scala2.11.tgz
2015-12-27 23:23 243448482 s3://spark-related-packages/spark-1.6.0-bin-hadoop1.tgz
2015-12-27 23:31 282904569 s3://spark-related-packages/spark-1.6.0-bin-hadoop2.3.tgz
2015-12-27 23:41 244381359 s3://spark-related-packages/spark-1.6.0-bin-hadoop2.4-without-hive.tgz
2015-12-27 23:48 284903527 s3://spark-related-packages/spark-1.6.0-bin-hadoop2.4.tgz
2015-12-28 00:00 289160984 s3://spark-related-packages/spark-1.6.0-bin-hadoop2.6.tgz
2015-12-28 00:08 201549664 s3://spark-related-packages/spark-1.6.0-bin-without-hadoop.tgz
2015-12-28 00:16 12204380 s3://spark-related-packages/spark-1.6.0.tgz
I think the problem here is that the artifacts are missing from the release not just from s3. i.e. http://www-us.apache.org/dist/spark/spark-1.6.2/spark-1.6.2-bin-hadoop1.tgz gives me a 404
Seems the same issue as #43.
Specifically, I get these messages during launch of the cluster, and these files are indeed not in place once the cluster starts up:
Indeed, no spark web interface on port 8080 either