search

ooyala / spark-jobserver

REST job server for Spark. Note that this is *not* the mainline open source version. For that, go to https://github.com/spark-jobserver/spark-jobserver. This fork now serves as a semi-private repo for Ooyala.
Other
344 stars 135 forks source link

what is the best practice #82

Closed hamroune closed 9 years ago

hamroune commented 9 years ago

Hi every one, I have an EC2 Spark cluster, And I need to know what is the best practice using job-server, for example it is a good idea to put it on the master? and point to set the master as "localhost:7077"? I need to have a clear idea about Job-server and scale it whith the cluster, thx in advance

yixu34 commented 9 years ago

At ooyala we have the job server on a different host from the master. It so happens that there's only support for a single job server host right now, which might make it seem convenient to colocate the master and the job server. But we're currently working on making the job server highly available, which involves having multiple job server hosts. In that scenario you wouldn't want to colocate the master(s) with the job server node(s), since it means you wouldn't be able to scale them up/down independently of each other.

hamroune commented 9 years ago

Thx for reply, I think it will be inetersting to scale up independently, to prevent single point of failure for our "spark as RESTful". thank you again