search

BD2KGenomics / cgcloud

Image and VM management for Jenkins, Spark and Mesos clusters in EC2
Other
22 stars 17 forks source link

Add port forwardings for Spark and HDFS web apps, e.g. 8080, 4040, ... #11

Open hannes-ucsc opened 9 years ago

hannes-ucsc commented 9 years ago

From @hannes-ucsc on April 18, 2015 7:24

Copied from original issue: BD2KGenomics/cgcloud-spark#2

hyviquel commented 8 years ago

The modification I did by hand to allow this: Opening Inbound in the security group Custom TCP 8080-8081 for ALL Custom Inbound TCP 4040-4045 for ALL Add export SPARK_PUBLIC_DNS="ec2-xxxxxxxxx.compute.amazonaws.com" to /opt/sparkbox/spark/conf/spark-env.sh

hannes-ucsc commented 8 years ago

@hyviquel, Did you have to do the spark-env.sh modification on every node or just the master. And what wasn't working when you didn't have it?

hyviquel commented 8 years ago

Sorry, I missed your reply. No, I do not need to modify all spark-env.sh but the Spark cluster has to be restart from the master node using stop-all.sh and start-all.sh scripts.

hyviquel commented 8 years ago

A more generic way to do it is by adding the following code to spark-env.sh :

public_name=$(curl -s http://169.254.169.254/latest/meta-data/public-hostname)
export SPARK_PUBLIC_DNS="$public_name"
hannes-ucsc commented 8 years ago

spark-tools should make the spark-env.sh modification that @hyviquel suggests.