search

amplab / spark-ec2

Scripts used to setup a Spark cluster on EC2
Apache License 2.0
393 stars 299 forks source link

Cannot run tasks on two differnet nodes #86

Open mmistroni opened 7 years ago

mmistroni commented 7 years ago

HI all i am creating an ec2 cluster using 2.0 branch. The cluster is created with 4 cores When created, i am connecting to each slave , kicking off exactly the same application with the following command:

oot@ip-172-31-4-154 bin]$ ./spark-submit --master spark://ec2-54-186-158-159.us-west-2.compute.amazonaws.com:7077 --executor-cores 1 /root/pyscripts/dataprocessing_Sample.py file:///root/pyscripts/tree_addhealth.csv

But the second app is being kept in waiting, even though only 2/4 cores are in use. I am getting this in to the logs

17/02/18 21:00:57 INFO TaskSchedulerImpl: Adding task set 0.0 with 1 tasks 17/02/18 21:01:12 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

Could you please advise why? I can provide as much information as you need .....

kr marco

shivaram commented 7 years ago

This question is not related to the EC2 scripts -- I'd suggest asking this on the Spark users mailing list / stack overflow as described in http://spark.apache.org/community.html

mmistroni commented 7 years ago

Will do. Apologies Kr

On 19 Feb 2017 1:14 am, "Shivaram Venkataraman" notifications@github.com wrote:

This question is not related to the EC2 scripts -- I'd suggest asking this on the Spark users mailing list / stack overflow as described in http://spark.apache.org/community.html

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/amplab/spark-ec2/issues/86#issuecomment-280887643, or mute the thread https://github.com/notifications/unsubscribe-auth/AJ7RDuEYmqEFVq5_2-ZrjO4723d8j2eEks5rd5eGgaJpZM4MFQY0 .

mmistroni commented 7 years ago

HI indeed,before i ask this, could youu please advise how can i disable this setting this in the generated /spark/conf/spark-env.sh ?

export SPARK_WORKER_CORES=2

i have tried to comment out that line in spark-ec2/templates/root/spark/conf/spark-env.sh but when the cluster gets generated and i go to the /root/spark/conf/spark-env.sh that line is not commented out.

Could you kinldy advise how i can do that? i have tried to stop the cluster, edit the file in the master, copy-dir that directory but when i start the cluster it generates master and slaves at brand new addresses.....

kind regards marco