Open tribbloid opened 10 years ago
I think you only need to add it to the kernel dependencies https://github.com/Bridgewater/scala-notebook/blob/98eecbcf03a87c44df64b589bb455289235e9c71/project/build.scala#L101
Tribbloid, I'm trying to get it working with spark as well. Adding it to the kernel allows you to import it, but I'm still running into issues because scala notebook uses akka 2.1.4, while spark 1.0 uses akka 2.2.3. If you get it working, I'd love to hear about it, and I'll do the same.
Hi Tyler,
Thanks a lot! I'm not a proficient sbt user but I've encounter another jar hell problem in maven: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-throws-NoSuchFieldError-when-testing-on-cluster-mode-td8064.html#a8083
The solution in maven is to include maven-shade plugin and rename the akka package in the notebook (since spark-shell will always load himself first). I'll fork this repo and migrate to maven in a few days.
I'm not sure if sbt-assembly has a similar switch that can do this. Perhaps it worths a try :)
Yours Peng
I tried upgrading to akka 2.2 but had trouble with the remote forking. Maybe I'll try it again.
Ken
On Jun 29, 2014, at 9:10 PM, Tyler Prete notifications@github.com wrote:
Tribbloid, I'm trying to get it working with spark as well. Adding it to the kernel allows you to input it, but I'm still running it issues because scala notebook uses akka 2.1.4, while spark 1.0 uses akka 2.2.3. If you get it working, I'd love to hear about it, and I'll do the same.
— Reply to this email directly or view it on GitHub.
Hi all, I'm going to give this a try. However, I would prefer to catch up where you guyz are. Is there any sample repo from where I can go ahead? Thanks in advance
So I started an integration... the hard way, but works quite fine: https://github.com/andypetrella/scala-notebook/tree/spark
Any doc to get it working with Spark?
I'll try to put some doc in the README, and most probably a blog, however the changes are quite deep and will be hard (non trivial) to migrate back :-(
I gave this one a try https://github.com/andypetrella/scala-notebook/tree/spark but it couldn't compile the code. See the image below
To not pollute this entry, could you fill an issue on the repo? Although this is not a compilation error but a (AFAICS) a misusage of sbt. There are several manner to use it, the longer but safer:
sbt project server run
The fastest:
sbt server/run
I'm trying to combine it with Apache Spark (one of the hottest framework for data analysis) but I don't know how to do it. Can someone give me a hint so I can possibly implement it myself? Of course, if someone is still maintaining this project.