Open matthewadams opened 7 years ago
I dont think spark-ec2 is designed to run with the client on your machine and a cluster on EC2. Its designed to run with the client on the master machine and the cluster on EC2. FWIW I think the version of Spark running on cluster needs to exactly match the version on the client for things to work -- that is anyway a question for the user mailing list / stack overflow.
@shivaram said:
the version of Spark running on cluster needs to exactly match the version on the client for things to work
@shivaram, that is exactly what I did. I ensured that my local scala, spark & hadoop versions exactly matched what was on the server.
Since I've burned so much time on spark-ec2
, I'm now trying to use Amazon EMR and having good success. Pointers to comparisons between spark-ec2
and Spark on Amazon EMR would be appreciated.
I'm getting the error below when attempting to
bin/run-example --master spark://ec2-000-000-000-000.compute-1.amazonaws.com:7077 SparkPi 10
from my local machine (ips changed to protect the innocent). This was run with a custom build/distribution of spark 2.0.2 using scala 2.10 with hadoop 2.4 (via build commands./dev/change-scala-version.sh 2.10 && ./dev/make-distribution.sh --name custom-spark --tgz -Phadoop-2.4 -Dscala-2.10 -DskipTests
).Note that this also happens using vanilla binary distributions of spark 2.0.2 & 2.1.0 (as mentioned in https://github.com/amplab/spark-ec2/issues/91#issue-212624665).
It is worth noting that if I
ssh
into the master and run the example from there, everything works fine: