search

hashicorp / nomad-spark

DEPRECATED: Apache Spark with native support for Nomad as a scheduler
44 stars 16 forks source link

Problem with dynamic allocation and external shuffle service #30

Open amitthk opened 5 years ago

amitthk commented 5 years ago

If we enable dynamic allocation and external shuffle service we get following error:

Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

We are able to run the below job only with dynamic allocation and external shuffle service disabled:

spark-submit --class com.company.app.runners.SingleContainerRunner --master nomad --deploy-mode cluster --conf spark.nomad.sparkDistribution=http://internal.repo/spark-2.3.2-bin-nomad-0.8.6-20181220.tgz --files http://internal.repo/app.yaml,http://internal.repo/app.properties,http://internal.repo/app-interactive-session.yaml http://internal.repo/app-executors-1.0.4-runtime.jar -container interactive -properties rel:app.properties -config rel:app.yaml

OneCricketeer commented 4 years ago

Related to #25 & #27 ?