Problem with Spark slaves connection

BD2KGenomics / cgcloud

Image and VM management for Jenkins, Spark and Mesos clusters in EC2

Other

22 stars 17 forks source link

Problem with Spark slaves connection #235

Closed hyviquel closed 8 years ago

hyviquel commented 8 years ago

Spark slaves do not connect alone to the master after creation of the cluster. I need to restart spark using the following command to make them connect

/opt/sparkbox/spark/sbin/stop-all.sh
/opt/sparkbox/spark/sbin/start-all.sh

I don't know if this is the problem but I update spark to version 2.0.1 and hadoop to version 2.7.3

hannes-ucsc commented 8 years ago

I'm inclined to think that this has to do with your changes to the code. The tests would break if this was a problem on a vanilla master.

hannes-ucsc commented 8 years ago

If you want help with this, I propose that you file a PR with the changes you mentioned (upgrading Spark and Hadoop). We can then see if this replicates on our continuous integration server.

hyviquel commented 7 years ago

fixed by 1641103cdb7348a2c1183a16ad29091c947a2c47