search

jupyter / declarativewidgets

[RETIRED] Jupyter Declarative Widget Extension
http://jupyter.org/declarativewidgets/docs.html
Other
120 stars 38 forks source link

(feature request) upgrade to latest Toree version (with support for Spark 2.0) #514

Open tmoerman opened 7 years ago

tmoerman commented 7 years ago

As you are probably aware, Toree now supports spark 2.0, would be super-nice if DW could upgrade as well.

Thanks in advance.

lbustelo commented 7 years ago

@tmoerman have you tested this already. I thought that the Spark guys retained the DataFrame type around.

lbustelo commented 7 years ago

Also... there is no packaged version of Toree on top of Spark 2.0 yet. We are working on it.

DarinJ commented 7 years ago

I can confirm there's an issue I've got toree kernels for 1.6 (0.1.0.dev5) and 2.0 (0.2.0.dev1) running and the updating textbox fails to fall a function with the spark 2.0 kernel but is fine under spark 1.6. There is one conspicuous but uninformative ERROR in the jupyper's stderr.

I then downloaded the source for declarative widgets and toree 0.2.0 to compile, this fails due to a missing class: SparkIMain, messing around I found toree switched to using IMain and eventually got a hacky version of declarative widgets to compile with a dependency of toree 0.2.0 and spark 2.0.1. I'm still testing but as I need it for some dashboards I'll probably get something going.

If there's interest I can provide my detailed notes and a patch likely next week. I suspect the way it should be is that jars are published for different spark versions to Maven Central/clojars/Bintray and PyPi so one can %addjar the one needed.

kalvinnchau commented 7 years ago

@DarinJ Any chance you still have the notes and patch for getting this working with Spark 2.0, Toree 0.2?