search

ywilkof / spark-jobs-rest-client

Fluent client for interacting with Spark Standalone Mode's Rest API for submitting, killing and monitoring the state of jobs.
Apache License 2.0
109 stars 58 forks source link

High availability with Zookeeper #11

Open ptriboll opened 7 years ago

ptriboll commented 7 years ago

Is there a solution to handle spark master url like spark://host1:port1,host2:port2. Cluster configuration is based on settings spark.deploy.recoveryMode and spark.deploy.zookeeper.url What is the best approach for rest client api?

ywilkof commented 7 years ago

Hi @ptriboll, I am not so familiar with the Zookeeper setup. In the spark properties map of the request, all keys must be valid properties acknowledged by Spark, as described in the configuration documentation. According to this documentation, the spark.master param does not seem accept a list of host:ports in the Standalone cluster setup.

If it turns out it is possible to give a list of hosts in the Standalone mode then of course we should support it.