search

amplab / docker-scripts

Dockerfiles and scripts for Spark and Shark Docker images
261 stars 102 forks source link

spark-worker default_cmd Script not Working as Expected #43

Closed LuqmanSahaf closed 10 years ago

LuqmanSahaf commented 10 years ago

I am using spark 1.0.0 docker images. It appears to me that the script default_cmd in spark-worker is not working as it should be working. This script calls prepare_spark $1 of /root/spark_files/configure_spark.sh. I have debugged it a lot. Even, I have called configure_spark.sh from spark-base image by using docker run -it. The problem is that these script do not replace the __MASTER__ tag in core_site.xml in /root/hadoop_files/ with the argument provided. Instead, the worker expects the master to be master. That is, it is static. Please, can someone help me out with this as I need it to create clusters on different machines. If I am not able to specify master like this, then I cannot create a cluster on different machines as the worker nodes will not know about the master. It works on single machine though, but that is because I have installed the docker-dns service.

LuqmanSahaf commented 10 years ago

I am sorry it is replacing the tag as it is supposed to. I was looking at /root/hadoop/core-site.xml instead of /etc/hadoop/core-site.xml. Once again, I am sorry. But the problem remains that why is the worker not contacting the master using its IP instead of the name 'master'.

LuqmanSahaf commented 10 years ago

I have created another issue and closing this one for the problem. https://github.com/amplab/docker-scripts/issues/44

marcinjurek commented 9 years ago

Hey I have the same problem. Were you able to figure it out?

LuqmanSahaf commented 9 years ago

@marcinjurek I had to edit the configure_spark.sh myself to let me use custom master. Further, if you want to deploy on multiple machines you may have to use some kind of SD or packet forwarding. Look at coreos/flannel (I'm using it) and weave. Moreover, to use names you might have to edit the hosts file and not able to use docker-dns as is. You may want to use DNS as it's clean. You might want to my project LuqmanSahaf/etcd-spark to have an idea. I am not working on this project as now I'm setting Spark in completely different way. Hope this helps.

LuqmanSahaf commented 9 years ago

SDN*