Open DSLituiev opened 8 years ago
Seemingly there is no archive for spark-1.6.2-bin-hadoop1.tgz
there.
Using --hadoop-major-version 2
leads to the following:
2016-07-28 04:53:42 (3.83 MB/s) - ‘spark-1.6.2-bin-hadoop2.4.tgz’ saved [273797124/273797124]
Unpacking Spark
[timing] spark init: 00h 01m 12s
Initializing ephemeral-hdfs
ERROR: Unknown Hadoop version
[timing] ephemeral-hdfs init: 00h 00m 00s
Initializing persistent-hdfs
ERROR: Unknown Hadoop version
[timing] persistent-hdfs init: 00h 00m 00s
Initializing mapreduce
ERROR: Unknown Hadoop version
mapreduce/init.sh: line 20: return: -1: invalid option
return: usage: return [n]
File or directory /root/mapreduce doesn't exist!
[timing] mapreduce init: 00h 00m 00s
Initializing spark-standalone
Yeah it looks like that file is missing -- @JoshRosen can you help in uploading spark-1.6.2-bin-hadoop1.tgz
to the S3 bucket spark-related-packages
?
@DSLituiev try --hadoop-major-version=2
, it worked fine for me.
I don't know if file have been uploaded, but i noticed that the name changed to spark-1.6.2-bin-hadoop1-scala2.11.tgz
on https://www.apache.org/dist/spark/spark-1.6.2/
@omdv: As I mentioned above (assuming nothing changed in code or repositories):
Using --hadoop-major-version 2
leads to the following:
2016-07-28 04:53:42 (3.83 MB/s) - ‘spark-1.6.2-bin-hadoop2.4.tgz’ saved [273797124/273797124]
Unpacking Spark
[timing] spark init: 00h 01m 12s
Initializing ephemeral-hdfs
ERROR: Unknown Hadoop version
[timing] ephemeral-hdfs init: 00h 00m 00s
Initializing persistent-hdfs
ERROR: Unknown Hadoop version
[timing] persistent-hdfs init: 00h 00m 00s
Initializing mapreduce
ERROR: Unknown Hadoop version
mapreduce/init.sh: line 20: return: -1: invalid option
return: usage: return [n]
File or directory /root/mapreduce doesn't exist!
[timing] mapreduce init: 00h 00m 00s
Initializing spark-standalone
@DSLituiev could you try it with =
sign?
@omdv: it works. Thank you!
I faced following issue while running
./spark-ec2 --key-pair=<> --identity-file=<> --region=us-west --instance-type=t2.micro -s 2 launch test-cluster
: