Right now it seems the latest hadoop version in spark-ec2 is 2.4, but actually in spark download page, it can be up to 2.7, also it is available in aws s3 http://s3.amazonaws.com/spark-related-packages. So the question is - how to launch such Spark cluster and if it does not support is there any workaround?
--hadoop-major-version=HADOOP_MAJOR_VERSION
Major version of Hadoop. Valid options are 1 (Hadoop
1.0.4), 2 (CDH 4.2.0), yarn (Hadoop 2.4.0) (default:
yarn)
Right now it seems the latest hadoop version in spark-ec2 is 2.4, but actually in spark download page, it can be up to 2.7, also it is available in aws s3 http://s3.amazonaws.com/spark-related-packages. So the question is - how to launch such Spark cluster and if it does not support is there any workaround?
--hadoop-major-version=HADOOP_MAJOR_VERSION Major version of Hadoop. Valid options are 1 (Hadoop 1.0.4), 2 (CDH 4.2.0), yarn (Hadoop 2.4.0) (default: yarn)