Add support for spark-shell

[tribbloid]

Adding support for spark-shell will drastically reduce time of prototyping and experimentation. (since you don't have to package and upload every time - only library changes) spark-shell is like other spark jobs (its also submitted by spark-submit) except that it takes command line input and interpret them on the fly. So it's not hard to implement it on top of existing editor UI like IPython-notebook.

I have spent a few days on its integration with IPython and once I get some results, I'll be able to commit that into your repo

[velvia]

Hi there,

Are you envisioning spark-shell as a separate job for spark job server? Not sure what you mean exactly. I had an idea earlier where you can open a spark shell into an existing job, allowing one to debug, which would be extremely cool.

On Thu, Jul 17, 2014 at 3:40 PM, Peng Cheng notifications@github.com wrote:

Adding support for spark-shell will drastically reduce time of prototyping and experimentation. (since you don't have to package and upload every time

• only library changes) spark-shell is like other spark jobs (its also submitted by spark-submit) except that it takes command line input and interpret them on the fly. So it's not hard to implement it on top of existing editor UI like IPython-notebook.

I have spent a few days on its integration with IPython and once I get some results, I'll be able to commit that into your repo

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/41.

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

[tribbloid]

I think it's cool as well but probably takes a long time to train people to use it (I personally never learn Julia debugger despite how I tried). local debugging let you capture almost all errors

My intention is to merely create a reply in browser so people don't have to ssh into server to use it

Yours Peng

Sent from Blue Mail

On Jul 17, 2014, 19:33, at 19:33, Evan Chan notifications@github.com wrote:

Hi there,

Are you envisioning spark-shell as a separate job for spark job server? Not sure what you mean exactly. I had an idea earlier where you can open a spark shell into an existing job, allowing one to debug, which would be extremely cool.

On Thu, Jul 17, 2014 at 3:40 PM, Peng Cheng notifications@github.com wrote:

Adding support for spark-shell will drastically reduce time of prototyping and experimentation. (since you don't have to package and upload every time

• only library changes) spark-shell is like other spark jobs (its also submitted by spark-submit) except that it takes command line input and interpret them on the fly. So it's not hard to implement it on top of existing editor UI like IPython-notebook.

I have spent a few days on its integration with IPython and once I get some results, I'll be able to commit that into your repo

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/41.

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

---

Reply to this email directly or view it on GitHub: https://github.com/ooyala/spark-jobserver/issues/41#issuecomment-49379842

[velvia]

Awesome, thanks, looking forward to it.

On Thu, Jul 17, 2014 at 8:46 PM, Peng Cheng notifications@github.com wrote:

I think it's cool as well but probably takes a long time to train people to use it (I personally never learn Julia debugger despite how I tried). local debugging let you capture almost all errors

My intention is to merely create a reply in browser so people don't have to ssh into server to use it

Yours Peng

Sent from Blue Mail

On Jul 17, 2014, 19:33, at 19:33, Evan Chan notifications@github.com wrote:

Hi there,

Are you envisioning spark-shell as a separate job for spark job server? Not sure what you mean exactly. I had an idea earlier where you can open a spark shell into an existing job, allowing one to debug, which would be extremely cool.

On Thu, Jul 17, 2014 at 3:40 PM, Peng Cheng notifications@github.com wrote:

Adding support for spark-shell will drastically reduce time of prototyping and experimentation. (since you don't have to package and upload every time

• only library changes) spark-shell is like other spark jobs (its also submitted by spark-submit) except that it takes command line input and interpret them on the fly. So it's not hard to implement it on top of existing editor UI like IPython-notebook.

I have spent a few days on its integration with IPython and once I get some results, I'll be able to commit that into your repo

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/41.

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

---

Reply to this email directly or view it on GitHub: https://github.com/ooyala/spark-jobserver/issues/41#issuecomment-49379842

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/41#issuecomment-49392817 .

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

[tribbloid]

Hi Evan, its working now (with some constraints in SparkContext usage, I'll elaborate later) This is a demo page with 10 cores: http://ec2-54-88-40-125.compute-1.amazonaws.com:8888/tree#notebooks This is sourcecode: https://github.com/tribbloid/ISpark Request to commit into jobserver

[velvia]

Hey Peng,

This is a really cool project!

Have you heard about Databricks Cloud? Looks like they are building something similar.

As far as integration into Job Server, how would you expect to integrate this into job server? Maybe tell me more? I see that your project is standalone right now, so I'm not sure what you have in mind.

cheers, Evan

On Fri, Jul 25, 2014 at 3:16 PM, Peng Cheng notifications@github.com wrote:

Hi Evan, its working now (with some constraints in SparkContext usage, I'll elaborate later) This is a demo page with 10 cores: http://ec2-54-88-40-125.compute-1.amazonaws.com:8888/tree#notebooks This is sourcecode: https://github.com/tribbloid/ISpark Request to commit into jobserver

— Reply to this email directly or view it on GitHub https://github.com/ooyala/spark-jobserver/issues/41#issuecomment-50210903 .

The fruit of silence is prayer; the fruit of prayer is faith; the fruit of faith is love; the fruit of love is service; the fruit of service is peace. -- Mother Teresa

[tribbloid]

Hi Evan,

Thanks a lot for liking it! Yeah I've heard of it and seen it :) Matei demo it himself in Toronto. Unfortunately my application for beta tester has been pending for 3 weeks and there is no response. Guess those Databricks guys definitely want to prioritize big customers with many old-school data analysts, not peanuts like me who have all types of weird feature request (like linking with Mahout REPL etc.) Also their service is purely managed on EC2, so if I use it for free they lose money. Eventually I have to do it myself.

The project was derived from IScala, it's a standard Spark application that can be spark-submit-ed into any cluster while its driver also act like a zeroMQ backend of IPython notebook. If you think its handy, you can make it a module, compile it into a jar, and create a shortcut to submit it to master in either scala/Java or bash script.

Did you get the beta tester badge from Databricks? How long have you being waiting?

All the best,

Yours Peng

[kelvinchu]

Hi Peng,

It's very intersting! BTW, did you see this beore: http://blog.cloudera.com/blog/2014/08/how-to-use-ipython-notebook-with-apache-spark/

Could you let me know what is the difference between your approach and their approach? I am very interested in using IPython Notebook like UI for Spark too. Thanks.

Kelvin