search

ooyala / spark-jobserver

REST job server for Spark. Note that this is *not* the mainline open source version. For that, go to https://github.com/spark-jobserver/spark-jobserver. This fork now serves as a semi-private repo for Ooyala.
Other
344 stars 135 forks source link

Mesos Master URL can not get parsed #93

Closed MichaelHirn closed 9 years ago

MichaelHirn commented 9 years ago

I finally got spark and spark-jobserver up and running on mesos. So far in the config i provided spark.master with local. Although it worked, i am not sure if this is the right way to go. Does it take care of the mesos actually and distribute nicely?

I am not sure, so i tried it with this spark.master url mesos://zk://... but got this error.

Caused by: java.lang.RuntimeException: Could not parse Master URL: 'mesos://zk:/.../mesos'
    at spark.jobserver.SparkWebUiActor.getSparkHostName(SparkWebUiActor.scala:106)
    at spark.jobserver.SparkWebUiActor.<init>(SparkWebUiActor.scala:37)

The fix for this error is trivial, i am actually more concerned if it's actually the right way to run it in production on a mesos cluster with local or if it would make more sense to run it with the mesos url.

velvia commented 9 years ago

Hi Michael,

In production you will definitely want to run with the spark.master pointed at mesos. Also, this was a bug but on master it should be fixed.

-Evan

On Sat, Nov 29, 2014 at 3:08 PM, Michael Hirn notifications@github.com wrote:

I finally got spark and spark-jobserver up and running on mesos. So far in the config i provided spark.master with local. Although it worked, i am not sure if this is the right way to go. Does it take care of the mesos actually and distribute nicely?

I am not sure, so i tried it with this spark.master url mesos://zk://... but got this error.

Caused by: java.lang.RuntimeException: Could not parse Master URL: 'mesos://zk:/.../mesos' at spark.jobserver.SparkWebUiActor.getSparkHostName(SparkWebUiActor.scala:106) at spark.jobserver.SparkWebUiActor.(SparkWebUiActor.scala:37)

The fix for this error is trivial, i am actually more concerned if it's actually the right way to run it in production on a mesos cluster with local or if it would make more sense to run it with the mesos url.

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/93.

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

MichaelHirn commented 9 years ago

Hi Velvia,

Thanks for your fast respond. But i do not think it is fixed. Maybe I am wrong and i didn't get something, but last commit to spark.jobserver.SparkWebUiActor was 26 days ago, there is still the same code that i am using. The code is pretty clear. Match for local and spark via the regex else it throws the Could not parse Master URL error.

Am I missing something?

velvia commented 9 years ago

Yes, you are looking in the wrong repo. The exact commit that fixes it is here:

https://github.com/spark-jobserver/spark-jobserver/commit/0db8b41aeb829e8300bb74f572a2b4bb318d0d7a

On Sun, Nov 30, 2014 at 2:00 AM, Michael Hirn notifications@github.com wrote:

Hi Velvia,

Thanks for your fast respond. But i do not think it is fixed. Maybe I am wrong and i didn't get something, but last commit to spark.jobserver.SparkWebUiActor was 26 days ago, there is still the same code that i am using. The code is pretty clear. Regex for local and spark if the master url doesn't match, then it throws the Could not parse Master URL error.

Am I missing something?

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/93#issuecomment-64980787 .

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

MichaelHirn commented 9 years ago

Awesome. Thanks for pointing that out to me. I guess this can be closed now.

Thanks